Information processing system, information terminal, information processing method and program

Abstract

To provide an information processing system (information terminal) that can quickly output the results of inference from a learned model.SOLUTION: An information processing system includes an information terminal 110 that acquires medical image data from a medical imaging device 100, and an information processing device 130 that makes an inference on the medical image data with a learned model. The information terminal 110 and the information processing device 130 are connected through a network 120. The information terminal 110 reduces the data size of the medical image data and sends it to the information processing device 130. The information processing device 130 makes an inference on the medical image data with the data size reduced and sends a first inference result to the information terminal 110.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, an information processing method, and a program that perform inference using a trained model. [Background technology]

[0002] There is known a Computer Aided Diagnosis (CAD) system that analyzes medical data acquired by a medical imaging device and presents diagnostic support information to a doctor. The CAD system applies machine learning technology to medical data, such as medical image data, and outputs diagnostic support information.

[0003] Patent Document 1 discloses that in order to improve the reliability of analysis based on machine learning, multiple processed medical signals related to a subject are obtained by implementing at least one of different imaging methods and different signal processing, and inference is performed for each of the multiple processed medical signals using multiple trained models. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-42810 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, inference is performed using multiple trained models for each of multiple processed medical signals, but it is desirable to reduce the data size (data volume) of the processed medical signals when transmitting and receiving them, and to output the inference results quickly.

[0006] Therefore, an object of the present invention is to provide an information processing system (information terminal) that can quickly output inference results obtained using a trained model. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, an information processing system includes an information terminal that acquires medical image data from a medical imaging device, and an information processing device that performs inference on the medical image data using a learned model, the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical image data and transmits it to the information processing device, and the information processing device performs inference on the medical image data with the reduced data size and transmits a first inference result to the information terminal. Effect of the Invention

[0008] According to the present invention, inference results obtained using a trained model can be quickly output. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing a configuration of an information processing system according to the present invention. [Diagram 2] FIG. 1 is a diagram showing a configuration of an information processing device according to the present invention. [Diagram 3] FIG. 2 is a diagram showing the configuration of a storage unit in the information processing device of the present invention. [Figure 4] 3A and 3B are diagrams showing an example of a screen display form in a display unit of the present invention. [Diagram 5] FIG. 1 is a diagram showing a configuration of an information terminal according to the present invention. [Figure 6] FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. [Figure 8] 4 is a flowchart showing the operation of a learning phase of the present invention. [Figure 9] 4 is a flow chart showing the operation of the inference phase of the present invention. [Figure 10] FIG. 11 is a diagram showing a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. EXAMPLES

[0011] FIG. 1 shows the configuration of an information processing system according to the present invention.

[0012] The information processing system of the present invention comprises an information terminal 110, a network 120, and an information processing device 130. The information terminal 110 may be a single information terminal or may be a plurality of information terminals. The medical imaging device 100 images a subject and generates medical image data.

[0013] An operation unit 112 and a display unit 114 are connected to the information terminal 110. The operation unit 112 receives various instructions from an operator and transmits the various instructions to the information terminal 110 and the medical imaging device 100. The operation unit 112 is composed of, for example, a mouse, a keyboard, a button, a panel switch, a foot switch, a trackball, a joystick, etc. The display unit 114 displays a GUI for inputting various instructions in the operation unit 112, and displays medical image data acquired from the medical imaging device 100.

[0014] In FIG. 1, operation unit 112 and display unit 114 are provided separately from information terminal 110, but information terminal 110 may have the functions of operation unit 112 and display unit 114 internally.

[0015] The medical imaging apparatus 100 is an apparatus that generates medical image data of a subject, such as an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, or an ultrasonic diagnostic apparatus.

[0016] X-ray CT scanners are equipped with an X-ray source and an X-ray detector. While rotating the X-ray source and the X-ray detector around the subject, the subject is irradiated with X-rays from the X-ray source and the data detected by the X-ray detector is projected to generate CT image data.

[0017] The MRI apparatus generates a predetermined magnetic field for a subject placed within a static magnetic field, and generates MRI image data by performing a Fourier transform on the acquired data.

[0018] An ultrasound diagnostic device transmits ultrasound waves to a subject, receives ultrasound waves reflected from the subject, and generates ultrasound image data.

[0019] The medical image data (CT image data, MRI image data, ultrasound image data, etc.) generated by the medical imaging device 100 is three-dimensional image data (volume data) or two-dimensional image data. The medical image data also includes raw data. The medical image data may be video image data consisting of multiple frame data. The medical image data may also include measurement data obtained by performing various measurements using the medical image data.

[0020] The medical imaging apparatus 100 is connected to an information terminal 110. The information terminal 110 is a PC terminal, a mobile phone such as a smartphone, a notebook terminal, a tablet terminal, or the like. The information terminal 110 can also set subject information and can associate medical data acquired from the medical imaging apparatus 100 with the subject information. The information terminal 110 can also display various data such as medical image data and measurement data acquired from the medical imaging apparatus 100.

[0021] The information terminal 110 and the information processing device 130 are connected to a network 120. The network 120 includes a communication network outside the hospital, such as wireless communication (Wi-Fi), the Internet, a wireless base station, a provider, a communication line, etc. The network 120 may also include an intranet, which is a communication network within the hospital. The information terminal 110 can communicate with the information processing device 130 via the network 120. The information terminal 110 can transmit medical image data to the information processing device 130. The information processing device 130 can transmit an inference result obtained by performing inference on the medical image data to the information terminal 110.

[0022] 2 shows the configuration of an information processing device 130 of the present invention. The information processing device 130 includes a teacher data generating unit 200 that generates teacher data using medical image data, a learning unit 202 that performs learning on the medical image data using the teacher data generated by the teacher data generating unit 200, a storage unit 204 that stores a trained model generated by the learning unit 202, and an inference unit 206 that performs inference on the medical image data using the trained model.

[0023] The components (functions) of the information processing device 130 are realized by, for example, a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program (software) stored in a memory. The information processing device 130 has a processor and a memory inside. The processor can execute each process of the information processing device 130 according to the program stored in the memory, and can cause the information processing device 130 to function as a teacher data generating unit 200, a learning unit 202, a memory unit 204, an inference unit 206, etc.

[0024] The teacher data generating unit 200 is connected to the network 120 and can acquire medical image data. The teacher data generating unit 200 generates teacher data using the medical image data. The teacher data generated by the teacher data generating unit 200 is determined according to the inference task performed by the neural network and the target of classification.

[0025] Examples of inference tasks performed by neural networks include classification tasks to classify the classes of medical image data, detection tasks to detect what is captured in which position of the medical image data, and segmentation tasks to extract target areas from medical image data.

[0026] When training a neural network to perform a classification task, the training data generation unit 200 generates training data that pairs medical image data with a correct answer label, which is a label indicating an object that appears in the medical image data.

[0027] On the other hand, when performing a detection task using a neural network, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct image to which an ROI (Region Of Interest) indicating the position of an object captured in the medical image data and a correct label that is a label indicating the object have been assigned.

[0028] When performing a segmentation task using a neural network, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct image to which position information of pixels of a target area reflected in the medical image data and a correct label that is a label indicating the target area have been assigned.

[0029] For example, a neural network is trained to perform a task of segmenting the presence or absence, type, and area of ​​a lesion from medical image data acquired from the information terminal 110. In this case, the training data generating unit 200 generates training data that pairs medical image data having a lesion area with information on a correct answer label indicating the type of lesion for the medical image data and a correct answer image with position information of the pixels of the lesion assigned.

[0030] The teacher data generating unit 200 may also perform preprocessing of medical image data depending on the neural network that is learning in the learning unit 202. For example, when the target of inference by the neural network is medical image data, the acquired medical image data is subjected to processes such as noise removal, filtering, image cropping, and resolution change. When the target of inference is natural language such as text, the teacher data generating unit 200 performs preprocessing of the data to be processed, such as performing morphological analysis and applying vector conversion technology, depending on the target of inference of the neural network and the task.

[0031] 2 shows a form in which the teacher data generating unit 200 is included inside the information processing device 130, the information terminal 110 may include the teacher data generating unit 200 therein. That is, the teacher data generating unit 200 may be included as a component of the information terminal 110. For example, after the teacher data as described above is generated by the information terminal 110, the inference device may be trained by the learning unit 202 of the information processing device 130 via the network 120.

[0032] The learning unit 202 is connected to the teacher data generating unit 200. The learning unit 202 generates a trained model by using a neural network to learn medical image data in association with teacher data. Here, the trained model indicates parameters determined by performing a learning process up to a predetermined standard and information on a model corresponding to the parameters. Note that the trained model may be used to train another model as transfer learning, or further learning processing may be performed on the trained model.

[0033] A neural network includes multiple layers. Among neural networks, CNN (Convolutional Neural Network), which is a type of deep learning technology, has multiple intermediate layers between the input layer and the output layer, although not shown. The multiple intermediate layers include a convolution layer, a pooling layer, an upsampling layer, and a synthesis layer. The convolution layer is a layer that performs convolution processing on a group of input values. In the convolution layer, the input medical image data is convolved to extract features of the medical image data.

[0034] The pooling layer is a layer that performs processing to make the number of output value groups smaller than the number of input value groups by thinning out or synthesizing the input value groups. The upsampling layer is a layer that performs processing to make the number of output value groups larger than the number of input value groups by duplicating the input value groups or adding values ​​interpolated from the input value groups. The synthesis layer is a layer that inputs value groups such as the output value groups of a certain layer or pixel values ​​constituting medical image data from multiple sources and synthesizes them by concatenating or adding them. The number of intermediate layers can be changed at any time depending on the learning content.

[0035] The storage unit 204 is connected to the learning unit 202. The storage unit 204 stores a trained model that has been trained to extract the type and area of ​​a lesion in medical image data, for example. The trained model is generated using a neural network, for example. However, in addition to CNN or RNN (Recurrent Neural Network), which is one of deep learning technologies among neural network technologies, or models derived from CNN or RNN, other machine learning technologies such as support vector machines, logistic regression, and random forests may be used, or a method based on a rule base may be used.

[0036] The inference unit 206 is connected to the network 120 and can acquire medical image data and measurement data. The inference unit 206 is connected to the storage unit 204 and can perform inference using a trained model stored in the storage unit 204.

[0037] The inference unit 206 performs inference on the newly generated medical image data, for example, using a trained model trained to extract the type and area of ​​a lesion in the medical image data. If the newly generated medical image data contains a lesion, the inference unit 206 can output the type and area of ​​the lesion.

[0038] 3 shows a configuration of the storage unit 204 of the information processing device 130 of the present invention. The storage unit 204 associates a trained model trained in the learning unit 202 with incidental information that identifies the type of the trained model and stores them. Specifically, the storage unit 204 assigns first incidental information 310 to a first trained model 300 trained in the learning unit 202, and stores the first trained model 300. The storage unit 204 assigns second incidental information 312 to a second trained model 302 trained in the learning unit 202, and stores the second trained model 302. The storage unit 204 assigns third incidental information 314 to a third trained model 304 trained in the learning unit 202, and stores the third trained model 304.

[0039] In FIG. 3, three trained models are stored in the storage unit 204 together with the accompanying information, but the storage unit 204 can also store four or more trained models together with the accompanying information. In addition, each of the multiple trained models is different in at least one of the characteristics of trained medical image data, the characteristics of teacher data, the diagnostic purpose, the class indicating the classification target, the model structure, and the like. Each trained model can be specified or identified based on the accompanying information assigned to each trained model. In addition, the inference unit 206 in the information processing device 130 can select a trained model according to the characteristics of the medical image data acquired from the information terminal 110, and use the selected trained model to perform inference on the medical image data acquired from the information terminal 110.

[0040] The additional information includes the type of trained model. The inference unit 206 can identify the type of trained model by referring to the additional information. The additional information includes information that identifies at least one of the characteristics of trained medical image data, the characteristics of teacher data, the diagnostic purpose, the class indicating the classification target, and the model structure.

[0041] FIG. 4 shows an example of a screen display form in the display unit 114 of the present invention. The display unit 114 displays a selection screen 400 for selecting the type of trained model used by the inference unit 206. In other words, the selection screen 400 is a selection unit for selecting the trained model used by the inference unit 206. Icons suitable for diagnostic purposes are displayed on the selection screen. By selecting each icon, the operator can select a trained model for detecting pneumonia, a trained model for detecting pulmonary nodules, a trained model for detecting lung cancer, a trained model for detecting emphysema, a trained model for detecting pleural effusion, a trained model for detecting pneumothorax, and the like. In addition, the operator can select all trained models by selecting the ALL icon.

[0042] Here, icons corresponding to pneumonia, pulmonary nodules, and pneumothorax are selected. The inference unit 206 can identify a trained model corresponding to pneumonia, pulmonary nodules, and pneumothorax by referring to the additional information (diagnosis purpose). In this manner, a trained model for detecting pneumonia, pulmonary nodules, and pneumothorax is selected.

[0043] The display unit 114 also displays a medical image 402 based on the medical image data. The medical image 402 is a plurality of slice images of a CT image (three-dimensional volume data). The inference unit 206 performs inference using a trained model corresponding to pneumonia, pulmonary nodule, and pneumothorax. As shown in FIG. 4, for example, when the inference unit 206 detects a pneumothorax in a CT image, the information processing device 130 transmits detection information and a region in which the pneumothorax is detected to the information terminal 110. The display unit 114 displays detection information 406 in which the pneumothorax is detected. The display unit 114 also displays a region 404 in which the pneumothorax is detected. That is, when the inference unit 206 in the information processing device 130 detects a lesion in the medical image data, the information processing device 130 transmits detection information in which the lesion is detected to the information terminal 110. The display unit 114 displays detection information 406 in which the lesion is detected.

[0044] 5 shows the configuration of an information terminal 110 of the present invention. The information terminal 110 has a function of thinning out medical image data, a function of performing pre-processing for inference, a function of transmitting and receiving information, and the like.

[0045] The information terminal 110 has a function of reducing the data size of medical image data. The information terminal 110 includes an image acquisition unit 502 that acquires medical image data from the medical imaging device 100, a processing unit 504 that performs thinning processing on the medical image data, a transmission unit 506 that transmits the medical image data processed by the processing unit 504 to an external device such as the information processing device 130 via the network 120, and a reception unit 508 that receives information transmitted from the external device such as the information processing device 130 via the network 120. The display unit 114 displays a medical image based on the medical image data and information transmitted from the external device such as the information processing device 130.

[0046] The processing unit 504 performs thinning processing on the first medical image data acquired by the image acquisition unit 502, and generates second medical image data having a data size smaller than the data size (data amount) of the first medical image data. The processing unit 504 generates third medical image data, which is the remaining medical image data obtained by subtracting the second medical image data from the first medical image data. Here, the processing unit 504 generates multiple medical image data (second medical image data, third medical image data) having a data size smaller than the first medical image data from the first medical image data.

[0047] The processing unit 504 outputs second medical image data generated by performing a thinning process on the first medical image data to the transmitting unit 506. Thereafter, the processing unit 504 outputs third medical image data, which is the remaining medical image data obtained by subtracting the second medical image data from the first medical image data, to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data to an external device such as the information processing device 130, and then transmits the third medical image data to the external device such as the information processing device 130. Note that the transmitting unit 506 may transmit the second medical image data to an external device such as the information processing device 130, and then transmit the first medical image data to an external device such as the information processing device 130.

[0048] 2, the information processing device 130 (the inference unit 206) receives the second medical image data transmitted from the information terminal 110 (the transmission unit 506) via the network 120, and performs inference on the second medical image data. The inference unit 206 performs inference on the second medical image data using the trained model stored in the storage unit 204.

[0049] The inference unit 206 can specify the type of the trained model by referring to the above-mentioned accompanying information. For example, the inference unit 206 specifies the type of the trained model by using the characteristics of the trained medical image data. The inference unit 206 can specify a trained model suitable for the thinned medical image data. The inference unit 206 performs inference on the second medical image data by using the trained model specified by the accompanying information.

[0050] The inference unit 206 extracts lesion information (lesion type and area) from the second medical image data. The inference unit 206 transmits the inference result obtained by performing inference on the second medical image data to the information terminal 110 via the network 120. If the second medical image data contains a lesion, the inference unit 206 transmits the lesion information (lesion type and area). If the second medical image data does not contain a lesion, the inference unit 206 transmits information indicating that the second medical image data does not contain a lesion. The information terminal 110 (receiving unit 508) receives the inference result (lesion information) obtained by performing inference on the second medical image data.

[0051] The display unit 114 displays the inference result obtained by performing inference on the thinned second medical image data together with the medical image based on the unthinned first medical image data. Here, the display unit 114 does not need to display the medical image based on the thinned second medical image data.

[0052] For example, assume that the first medical image data is a 1024×1024 matrix with 1000 slice images, and the thinned second medical image data is a 1024×1024 matrix with 500 slice images. Since the second medical image data has about half the data size (data amount) of the first medical image data, the transfer time of the second medical image data is shorter than the transfer time of the first medical image data. By transmitting the second medical image data having a data size smaller than the data size (data amount) of the first medical image data to the information processing device 130 (inference unit 206) and performing inference, the inference result can be output quickly.

[0053] Furthermore, the inference unit 206 identifies a trained model suitable for medical image data obtained by integrating the second medical image data and the third medical image data. The inference unit 206 performs inference on the medical image data obtained by integrating the second medical image data and the third medical image data using a trained model suitable for unthinned medical image data. The display unit 114 displays a first inference result obtained by inference on the second medical image data, and then displays a second inference result obtained by inference on the medical image data obtained by integrating the second medical image data and the third medical image data. Therefore, an accurate inference result can be output.

[0054] 6 and 7 show the processing form of the processing unit 504 of the present invention. Here, it is assumed that the medical image data is three-dimensional volume data. The three-dimensional volume data is made up of a plurality of slice images (frame images). Here, for simplicity of illustration, the explanation will be given assuming eight slice images.

[0055] As shown in FIG. 6(a), the processing unit 504 performs thinning processing on the first medical image data 600 to generate second medical image data 602 having a data size smaller than that of the first medical image data 600. The processing unit 504 selects odd-numbered slice images to generate the second medical image data 602. In FIG. 6(a), the selected slice images are shown in white, and the unselected slice images are shown in gray. The processing unit 504 generates third medical image data 604, which is the remaining medical image data obtained by subtracting the second medical image data 602 from the first medical image data 600. The processing unit 504 generates the third medical image data 604 from the even-numbered slice images.

[0056] The processing unit 504 outputs second medical image data 602 consisting of odd-numbered slice images to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data 602 to the information processing device .

[0057] The inference unit 206 performs inference on the second medical image data 602 using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for the thinned medical image data.

[0058] Here, the auxiliary information of the trained model stored in the memory unit 204 includes characteristics of the trained medical image data, and the characteristics of the medical image data include at least one of data reduction information, thinning information, data size, matrix size, number of slice images, region information, etc.

[0059] The data size reduction information is information indicating how much the data size has been reduced compared to the original medical image data. For example, if the original medical image data has been reduced to 1 / 2 size, the data size reduction information is 1 / 2. If the original medical image data has been reduced to 1 / 3 size, the data size reduction information is 1 / 3. The thinning information is information indicating how much the original medical image data has been thinned out. For example, if frames of the original medical image data are selected alternately, the thinning information is 1 / 2.

[0060] For example, the learning unit 202 generates a trained model by previously learning the thinned medical image data in association with teacher data. The memory unit 204 stores the trained model together with the accompanying information. The accompanying information is information indicating that the training was performed using the thinned medical image data. Since the second medical image data 602 is thinned medical image data, the inference unit 206 performs inference on the second medical image data 602 using the trained model identified by the accompanying information.

[0061] The inference unit 206 transmits the inference result obtained by performing inference on the second medical image data 602 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the thinned second medical image data 602 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference result of the medical image data.

[0062] Then, the transmitting unit 506 transmits the thinned-out third medical image data 604 to the information processing device 130. The inference unit 206 performs inference on the first medical image data 600 obtained by integrating the second medical image data 602 and the third medical image data 604. In the information processing device 130, a process is performed to restore the thinned-out second medical image data 602 and the third medical image data 604 to the original first medical image data 600.

[0063] The inference unit 206 identifies a trained model suitable for unthinned medical image data. The inference unit 206 performs inference on the first medical image data 600 using the trained model suitable for unthinned medical image data.

[0064] The inference unit 206 transmits the inference result obtained by performing inference on the first medical image data 600 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the first medical image data 600 together with a medical image based on the first medical image data 600. Since inference is performed on the first medical image data 600 that has not been thinned out, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

[0065] In this way, a first inference result obtained by performing inference on the thinned second medical image data 602 and a second inference result obtained by performing inference on the non-thinned first medical image data 600 are displayed in two stages on the display unit 114. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

[0066] As shown in FIG. 6(b), the processing unit 504 performs thinning processing on the first medical image data 600 to generate second medical image data 612 having a data size smaller than that of the first medical image data 600. The processing unit 504 selects slice images in a predetermined range (near the center) to generate second medical image data 612. The area near the center of the medical image data is an area where lesions tend to be concentrated in the medical image data. In FIG. 6(b), the selected slice images are shown in white, and the unselected slice images are shown in gray. The processing unit 504 generates third medical image data 614, which is the remaining medical image data obtained by subtracting the second medical image data 612 from the first medical image data 600. The processing unit 504 generates the third medical image data 614 from slice images near the edge.

[0067] The processing unit 504 outputs second medical image data 612 consisting of slice images in a predetermined range (near the center) to the transmitting unit 506. Thereafter, the processing unit 504 outputs third medical image data 614 consisting of slice images outside the predetermined range (near the end) to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data 612 to the information processing device 130.

[0068] The inference unit 206 performs inference on the second medical image data 612 by using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information.

[0069] The second medical image data 612 is thinned medical image data, but since the slice interval is the same as that of the first medical image data 600, the inference unit 206 performs inference on the second medical image data 612 using a trained model suitable for unthinned medical image data.

[0070] The inference unit 206 transmits the inference result obtained by performing the inference on the second medical image data 612 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing the inference on the second medical image data 612 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference result.

[0071] Then, the transmission unit 506 transmits the third medical image data 614 to the information processing device 130. The inference unit 206 performs inference on the first medical image data 600 obtained by integrating the second medical image data 612 and the third medical image data 614.

[0072] The inference unit 206 identifies a trained model suitable for unthinned medical image data. The inference unit 206 performs inference on the first medical image data 600 using the trained model suitable for unthinned medical image data.

[0073] The inference unit 206 transmits the inference result obtained by performing inference on the first medical image data 600 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the first medical image data 600 together with a medical image based on the first medical image data 600. Since inference is performed on the first medical image data 600 that has not been thinned out, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

[0074] In this way, the first inference result obtained by performing inference on the second medical image data 612 in a predetermined range (near the center) and the second inference result obtained by performing inference on the first medical image data 600 are displayed in two stages on the display unit 114. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

[0075] Furthermore, as shown in FIG. 7(a), the processing unit 504 performs a matrix size reduction process on the first medical image data 600. The processing unit 504 generates second medical image data 622 having slice images with a matrix size smaller than the matrix size of the slice images constituting the first medical image data 600. The processing unit 504 generates third medical image data 624 having slice images with a matrix size larger than the matrix size of the slice images constituting the second medical image data 622. For example, if the slice images constituting the first medical image data 600 have a size of 1024×1024 matrix, the slice images constituting the second medical image data 622 have a size of 512×512 matrix. The slice images constituting the third medical image data 624 have a size of 768×768 matrix.

[0076] FIG. 7(a) shows second medical image data 622 and third medical image data 624, which are composed of slice images that have been subjected to a matrix size reduction process.

[0077] The processing unit 504 outputs the second medical image data 622 to the transmitting unit 506. Thereafter, the processing unit 504 outputs the third medical image data 624 to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data 622 to the information processing device 130.

[0078] The inference unit 206 performs inference on the second medical image data 622 using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for the reduced matrix size.

[0079] The storage unit 204 stores the trained model together with the supplementary information (matrix size). The supplementary information is information indicating that the trained model has been trained using medical image data composed of slice images of a predetermined matrix size. The inference unit 206 performs inference on the second medical image data 622 using the trained model specified by the supplementary information.

[0080] The inference unit 206 transmits the inference result obtained by performing the inference on the second medical image data 622 consisting of slice images subjected to the matrix size reduction process to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing the inference on the second medical image data 622 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference result.

[0081] Then, the transmitting unit 506 transmits third medical image data 624 having a slice image with a matrix size different from the matrix size of the slice image of the second medical image data 622 to the information processing device 130. The matrix size of the slice image of the third medical image data 624 is larger than the matrix size of the slice image of the second medical image data 622. The inference unit 206 performs inference on the third medical image data 624. The inference unit 206 performs inference on the third medical image data 624 using a trained model identified by the supplementary information (matrix size).

[0082] The inference unit 206 transmits the inference result obtained by performing the inference on the third medical image data 624 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing the inference on the third medical image data 624 together with the medical image based on the first medical image data 600. Since the inference is performed on the third medical image data 624 having a slice image with a matrix size larger than the matrix size of the slice images constituting the second medical image data 622, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

[0083] In this way, a first inference result obtained by performing inference on the second medical image data 622 having slice images with a second matrix size (512×512) and a second inference result obtained by performing inference on the third medical image data 624 having slice images with a third matrix size (768×768) are displayed in two stages on the display unit 114. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

[0084] Also, as shown in FIG. 7(b), the processing unit 504 performs region extraction processing on the first medical image data 600 to generate second medical image data 632 having a data size smaller than that of the first medical image data 600. The processing unit 504 selects a predetermined region (central region) of each slice image to generate the second medical image data 632. In FIG. 7(b), the selected image region is shown in white, and the unselected image region is shown in gray. The processing unit 504 generates third medical image data 634, which is the remaining medical image data obtained by subtracting the second medical image data 632 from the first medical image data 600. The processing unit 504 generates the third medical image data 634 consisting of slice images excluding the predetermined region (central region).

[0085] The processing unit 504 outputs second medical image data 632 consisting of slice images of the predetermined region (central region) to the transmitting unit 506. Thereafter, the processing unit 504 outputs third medical image data 634 consisting of slice images excluding the predetermined region (central region) to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data 632 to the information processing device 130.

[0086] The inference unit 206 performs inference on the second medical image data 632 using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for a predetermined area.

[0087] The storage unit 204 stores the trained model together with the supplementary information (region information). The supplementary information is information indicating that training has been performed using medical image data consisting of slice images of a predetermined region. Since the second medical image data 632 is medical image data consisting of slice images of a predetermined region (central region), the inference unit 206 performs inference on the second medical image data 632 using the trained model specified by the supplementary information.

[0088] The inference unit 206 transmits the inference result obtained by performing the inference on the second medical image data 632 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing the inference on the second medical image data 632 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference result.

[0089] Then, the transmitting unit 506 transmits the third medical image data 634 consisting of slice images excluding the predetermined region (central region) to the information processing device 130. The inference unit 206 performs inference on the first medical image data 600 obtained by integrating the second medical image data 632 and the third medical image data 634.

[0090] The inference unit 206 identifies a trained model suitable for unthinned medical image data. The inference unit 206 performs inference on the first medical image data 600 using the trained model suitable for unthinned medical image data.

[0091] The inference unit 206 transmits the inference result obtained by performing inference on the first medical image data 600 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the first medical image data 600 together with a medical image based on the first medical image data 600. Since inference is performed on the first medical image data 600 that has not been thinned out, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

[0092] In this way, the first inference result obtained by performing inference on the second medical image data 632 consisting of slice images of a predetermined region (central region) and the second inference result obtained by performing inference on the first medical image data 600 are displayed in two stages on the display unit 114. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

[0093] The operation of the learning phase of the present invention will be described with reference to FIG.

[0094] S800: The medical imaging apparatus 100 captures an image of a subject and acquires medical image data. The information terminal 110 (image acquisition unit 502) acquires the medical image data from the medical imaging apparatus 100.

[0095] S802: The operator selects whether or not to reduce the data size (data amount) of the medical image data via the operation unit 112. If the data size of the medical image data is not to be reduced, the process proceeds to S804. If the data size of the medical image data is to be reduced, the process proceeds to S810.

[0096] S804: The information terminal 110 (transmission unit 506) transmits the medical image data to the information processing device 130. The information processing device 130 (teacher data generation unit 200) generates teacher data that pairs the medical image data with a correct answer label. The information processing device 130 (learning unit 202) generates a trained model by learning the medical image data in association with the teacher data using a neural network.

[0097] S806: The information processing device 130 (the storage unit 204) stores the trained model trained by the training unit 202 and the associated information.

[0098] S810: The information terminal 110 (processing unit 504) performs data size reduction processing on the medical image data acquired by the image acquisition unit 502. The processing unit 504 generates medical image data with a data size smaller than the data size (data amount) of the medical image data. This processing generates medical image data with a plurality of data sizes.

[0099] S812: The information terminal 110 (transmission unit 506) transmits medical image data of multiple data sizes to the information processing device 130. The information processing device 130 (teacher data generation unit 200) generates teacher data that pairs each medical image data with a correct answer label. The information processing device 130 (learning unit 202) generates multiple trained models by using a neural network to learn each medical image data in association with the teacher data.

[0100] S814: The information processing device 130 (the storage unit 204) stores the multiple trained models trained by the training unit 202 and the associated information, as shown in FIG.

[0101] The operation of the inference phase of the present invention will be described with reference to FIG.

[0102] S900: The medical imaging apparatus 100 captures an image of a subject and acquires medical image data. The information terminal 110 (image acquisition unit 502) acquires the medical image data from the medical imaging apparatus 100.

[0103] S902: The operator selects whether or not to reduce the data size (data amount) of the medical image data via the operation unit 112. If the data size of the medical image data is not to be reduced, the process proceeds to S904. If the data size of the medical image data is to be reduced, the process proceeds to S910.

[0104] S904: The information terminal 110 (transmission unit 506) transmits the medical image data to the information processing device 130. The information processing device 130 (inference unit 206) uses the trained model stored in the storage unit 204 to perform inference on the medical image data.

[0105] S906: The information processing device 130 (the inference unit 206) transmits the inference result obtained by performing the inference on the medical image data to the information terminal 110. The display unit 114 displays the inference result obtained by performing the inference on the medical image data together with the medical image based on the medical image data.

[0106] S910: The information terminal 110 (processing unit 504) performs data size reduction processing on the medical image data acquired by the image acquisition unit 502. The processing unit 504 generates medical image data with a data size smaller than the data size (data amount) of the medical image data. This processing generates medical image data with a plurality of data sizes.

[0107] S912: The information terminal 110 (the inference unit 206) performs multiple inferences on the multiple medical image data using the multiple trained models stored in the storage unit 204. The inference unit 206 identifies a trained model suitable for medical image data whose data size has been reduced. The inference unit 206 also identifies a trained model suitable for medical image data whose data size has not been reduced. The inference unit 206 performs inferences on the multiple medical image data using the trained models suitable for the multiple medical image data.

[0108] S914: The display unit 114 displays a first inference result obtained by performing inference on the medical image data whose data size has been reduced, together with a medical image based on the medical image data. Then, the display unit 114 displays a second inference result obtained by performing inference on the medical image data whose data size has not been reduced. The display unit 114 displays the inference results in the order in which they were transmitted from the information terminal 110 (inference unit 206).

[0109] As described above, the information processing system of the present invention includes an information terminal 110 that acquires medical image data from a medical imaging device 100, and an information processing device 130 that performs inference on the medical image data using a trained model, the information terminal 110 and the information processing device 130 are connected via a network 120, the information terminal 110 reduces the data size of the medical image data and transmits it to the information processing device 130, the information processing device 130 performs inference on the medical image data with the reduced data size, and transmits a first inference result to the information terminal 110. Thus, the operator can quickly recognize the inference result at an early stage.

[0110] The information terminal 110 transmits the remaining medical image data that has not been transmitted to the information processing device 130 to the information processing device 130, and the information processing device 130 performs inference on the medical image data obtained by integrating the medical image data with the reduced data size and the remaining medical image data, and transmits the second inference result to the information terminal 110. Therefore, the operator can recognize the accurate inference result in the next stage. EXAMPLES

[0111] An information processing system (information processing device 130) according to a second embodiment of the present invention will be described with reference to Fig. 10. The difference from the first embodiment is that the information processing system includes an integration processing unit 210 that integrates multiple inference results. The integration processing unit 210 includes an inference result storage unit 800 that stores multiple inference results inferred by the inference unit 206, and an addition processing unit 802 that adds up the multiple inference results.

[0112] The inference unit 206 performs inference on the second medical image data with reduced data size, using the trained model stored in the storage unit 204. The inference unit 206 transmits a first inference result (initial inference result) obtained by performing inference on the second medical image data, to the information terminal 110 via the network 120. The display unit 114 displays the first inference result obtained by performing inference on the second medical image data with reduced data size.

[0113] Moreover, the inference unit 206 transmits a first inference result obtained by performing inference on the second medical image data to the integration processing unit 210. The inference result storage unit 800 stores the first inference result.

[0114] The inference unit 206 performs inference on the first medical image data whose data size has not been reduced, using the trained model stored in the storage unit 204. The inference unit 206 transmits a second inference result (next inference result) obtained by performing inference on the first medical image data to the integrated processing unit 210. The inference result storage unit 800 stores the second inference result. Here, the inference result storage unit 800 stores the first inference result and the second inference result, but may store three or more inference results.

[0115] The addition processing unit 802 performs an addition process on multiple inference results (first inference result and second inference result) stored in the inference result storage unit 800. At this time, the objects of addition in the addition processing unit 802 are second medical image data generated from first medical image data obtained by imaging a subject, and multiple inference results obtained by inferring the first medical image data. In other words, the objects of addition in the addition processing unit 802 are multiple inference results obtained by inferring medical image data (multiple items) whose original medical image data is the same.

[0116] The addition processing unit 802 can adjust the addition weight. The addition processing unit 802 performs the addition process by weighting the second inference result heavier than the weighting of the first inference result. This is because the second inference result is inferred from the first medical image data whose data size has not been reduced.

[0117] The integration processing unit 210 transmits an inference result obtained by integrating a plurality of inference results to the information terminal 110 via the network 120. The display unit 114 displays the integrated inference result.

[0118] As described above, the information processing system of the present invention includes the integration processing unit 210 that integrates a plurality of inference results. By integrating a plurality of inference results, it is possible to improve the inference accuracy.

[0119] A computer program for realizing the functions of the above-mentioned embodiments can be supplied to a computer via a network or memory (not shown), and the computer program can be executed by a processor (not shown). This is a computer program for causing a computer to execute the above-mentioned information processing method. In other words, the computer program is a program for causing a computer to realize the functions of an information processing device. The memory stores the computer program. [Explanation of symbols]

[0120] 100 Medical imaging device 110 Information terminal 112 Operation section 114 Display section 120 Network 130 Information processing equipment 200 Teacher Data Generation Unit 202 Learning Department 204 Storage section 206 Reasoning section 502 Image acquisition unit 504 Processing section 506 Transmitter 508 Receiving unit

Claims

1. an information terminal for acquiring medical image data from a medical imaging device; an information processing device having an inference unit that performs inference on the medical image data using a trained model; the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical image data and transmits the data to the information processing device; the inference unit performs inference on the medical image data whose data size has been reduced, and transmits a first inference result to the information terminal; An information processing system characterized in that the information terminal transmits remaining medical image data that has not been transmitted to the information processing device to the information processing device, the inference unit performs inference on the medical image data, and the information processing device transmits a second inference result to the information terminal.

2. The information processing system according to claim 1, characterized in that the inference unit selects a trained model according to characteristics of medical image data acquired from the information terminal, and performs inference on the medical image data using the selected trained model.

3. The information processing system according to claim 1, further comprising a memory unit that links and stores the trained model with additional information that identifies a type of the trained model, and the inference unit identifies the type of the trained model by referring to the additional information.

4. The information processing system according to claim 3, characterized in that the supplementary information includes information identifying at least one of the characteristics of the learned medical image data, the characteristics of the teacher data, the diagnostic purpose, a class indicating the classification target, and a model structure.

5. 2. The information processing system according to claim 1, wherein, when the inference unit detects a lesion in the medical image data, the information processing device transmits detection information indicating that the lesion has been detected to the information terminal.

6. The information processing system according to claim 5 , further comprising a display unit that displays the detection information transmitted to the information terminal.

7. The information processing system according to claim 1 , wherein the information terminal comprises an image acquisition unit that acquires the medical image data from the medical imaging device, and a processing unit that performs a thinning process on the medical image data.

8. The information processing system according to claim 7, characterized in that the processing unit performs a thinning process on the first medical image data acquired by the image acquisition unit, generates second medical image data having a data size smaller than the data size of the first medical image data, and generates third medical image data which is the remaining medical image data by subtracting the second medical image data from the first medical image data, and the second medical image data and the third medical image data are transmitted to the information processing device.

9. The information processing system according to claim 8, characterized in that the inference unit uses the learned model to perform inference on the second medical image data to generate a first inference result, and performs inference on the first medical image data that is an integration of the second medical image data and the third medical image data to generate a second inference result.

10. The information processing system according to claim 8, further comprising a display unit that displays a first inference result obtained by performing inference on the second medical image data together with a medical image based on the first medical image data.

11. The information processing system according to claim 10, characterized in that the display unit displays a first inference result obtained by performing inference on the second medical image data, and then displays a second inference result obtained by performing inference on the first medical image data which is an integrated version of the second medical image data and the third medical image data.

12. 10. The information processing system according to claim 2, further comprising an integration processing unit that integrates the first inference result and the second inference result.

13. The information processing system according to claim 12, characterized in that the integration processing unit includes an inference result memory unit that stores a plurality of inference results inferred by the inference unit, and an addition processing unit that adds up the plurality of inference results.

14. an information terminal for acquiring medical image data from a medical imaging device; an information processing device having an inference unit that performs inference on the medical image data using a trained model; the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical image data and transmits the data to the information processing device; the information processing device performs inference on the medical image data whose data size has been reduced, and transmits a first inference result to the information terminal; An information processing system characterized in that the information terminal transmits other medical image data contained in the medical image data that has not been transmitted to the information processing device to the information processing device, and the information processing device performs inference on the other medical image data and transmits a second inference result to the information terminal.

15. an information terminal for acquiring medical image data from a medical imaging device; an information processing device having an inference unit that performs inference on the medical image data using a trained model; the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical image data and transmits the data to the information processing device; The information processing device is an information processing system that performs inference on the medical image data whose data size has been reduced and transmits a first inference result to the information terminal, the information terminal includes an image acquisition unit that acquires first medical image data from the medical imaging device, and a processing unit that performs a thinning process on the first medical image data; An information processing system characterized in that the processing unit performs a thinning process on the first medical image data acquired by the image acquisition unit, generates second medical image data having a data size smaller than the data size of the first medical image data, and generates third medical image data which is the remaining medical image data by subtracting the second medical image data from the first medical image data, and the second medical image data and the third medical image data are transmitted to the information processing device.

16. An information terminal connected via a network to an information processing device that performs inference on medical image data, an acquisition unit for acquiring medical image data from a medical imaging device; A processing unit that reduces a data size of the medical image data and generates the medical image data having the reduced data size; a transmission unit that transmits the medical image data whose data size has been reduced to the information processing device; a receiving unit that receives, via the network, from the information processing device, a first inference result in which inference has been performed by the information processing device on the medical image data whose data size has been reduced; The transmission unit transmits remaining medical image data that has not been transmitted to the information processing device to the information processing device, The information terminal is characterized in that the receiving unit receives a second inference result, which is an inference performed by the information processing device on the medical image data, from the information processing device via the network.

17. acquiring medical image data from a medical imaging device; and performing inference on the medical image data using the trained model, Reducing the data size of the medical image data; transmitting the medical image data whose data size has been reduced from the information terminal to the information processing device; A step of performing inference on the medical image data whose data size has been reduced by the information processing device, and transmitting a first inference result to the information terminal; transmitting the remaining medical image data that has not been transmitted to the information processing device to the information processing device, causing the information processing device to perform inference on the medical image data, and transmitting a second inference result to the information terminal; 13. An information processing method comprising: An information processing method comprising the steps of:

18. A program causing a computer to execute the information processing method according to claim 17.

Images