Information processing system, medical imaging device, information terminal, information processing method and program

Abstract

To provide an information processing system for executing inference for medical moving image data using a learned model and outputting a result of the inference.SOLUTION: An information processing system includes an information terminal 110 for acquiring medical moving image data from a medical imaging device 100, and an information processing device 130 having an inference part 206 for executing inference for the medical moving image data using a learned model. The information terminal 110 is connected to the information processing device 130 through a network 120. The information terminal 110 reduces a data size of the medical moving image data and transmits the reduced data to the information processing device 130. The inference part 206 executes inference for the medical moving image data whose data size is reduced, and transmits a result of the inference to the information terminal 110.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system, a medical imaging apparatus, an information terminal, an information processing method, and a program for performing inference using a trained model. [Background technology]

[0002] There is known a Computer Aided Diagnosis (CAD) system that analyzes medical data acquired by a medical imaging device and presents diagnostic support information to a doctor. The CAD system applies machine learning technology to medical data, such as medical image data, and outputs diagnostic support information.

[0003] Patent Document 1 discloses that in order to improve the reliability of analysis based on machine learning, multiple processed medical signals related to a subject are acquired by performing at least one of different imaging methods and different signal processing. Patent Document 1 also discloses that inference is performed using multiple trained models for each of the multiple processed medical signals. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-42810 A Summary of the Invention [Problem to be solved by the invention]

[0005] In Patent Document 1, inference is performed using multiple trained models for each of multiple processed medical signals, but performing inference on medical video image data is not considered. Therefore, it is desirable to perform inference on medical video image data using trained models and output the inference results.

[0006] Therefore, an object of the present invention is to provide an information processing system that can perform inference on medical video image data using a trained model and output the inference results. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, the information processing system includes an information terminal that acquires medical video data of a subject from a medical imaging device, and an information processing device having an inference unit that performs inference on the medical video data using a learned model, the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical video data and transmits it to the information processing device, the inference unit performs inference on the medical video data with reduced data size, and the information processing device transmits the inference result to the information terminal.

[0008] The system also includes a medical imaging device that images a subject and generates medical video data, and an information processing device having an inference unit that performs inference on the medical video data using a learned model, wherein the medical imaging device and the information processing device are connected via a network, the medical imaging device reduces the data size of the medical video data and transmits it to the information processing device, the inference unit performs inference on the medical video data with reduced data size, and the information processing device transmits the inference result to the medical imaging device. Effect of the Invention

[0009] According to the present invention, it is possible to perform inference on medical video image data using a trained model and output the inference results. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing a configuration of an information processing system according to the present invention. [Diagram 2] FIG. 1 is a diagram showing a configuration of an information processing device according to the present invention. [Diagram 3] FIG. 2 is a diagram showing the configuration of a storage unit in the information processing device of the present invention. [Figure 4] 3A and 3B are diagrams showing an example of a screen display form in a display unit of the present invention. [Diagram 5] FIG. 1 is a diagram showing a configuration of an information terminal according to the present invention. [Figure 6] FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. [Figure 8] FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. [Figure 9] FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. [Figure 10] FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. [Figure 11] 4 is a flowchart showing the operation of a learning phase of the present invention. [Figure 12] 4 is a flow chart showing the operation of the inference phase of the present invention. [Figure 13] FIG. 11 is a diagram showing a second embodiment of the present invention. [Figure 14] FIG. 11 is a diagram showing a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. EXAMPLES

[0012] FIG. 1 shows the configuration of an information processing system according to the present invention.

[0013] The information processing system of the present invention comprises an information terminal 110, a network 120, and an information processing device 130. The information terminal 110 may be a single information terminal or may be a plurality of information terminals. The medical imaging device 100 images a subject and generates medical image data.

[0014] An operation unit 112 and a display unit 114 are connected to the information terminal 110. The operation unit 112 receives various instructions from an operator and transmits the various instructions to the information terminal 110 and the medical imaging device 100. The operation unit 112 is composed of, for example, a mouse, a keyboard, a button, a panel switch, a foot switch, a trackball, a joystick, etc. The display unit 114 displays a GUI for inputting various instructions in the operation unit 112, and displays medical image data acquired from the medical imaging device 100.

[0015] In FIG. 1, operation unit 112 and display unit 114 are provided separately from information terminal 110, but information terminal 110 may have the functions of operation unit 112 and display unit 114 internally.

[0016] The medical imaging apparatus 100 is an apparatus that generates medical image data of a subject, such as an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, or an ultrasonic diagnostic apparatus.

[0017] X-ray CT scanners are equipped with an X-ray source and an X-ray detector. While rotating the X-ray source and the X-ray detector around the subject, the subject is irradiated with X-rays from the X-ray source and the data detected by the X-ray detector is projected to generate CT image data.

[0018] The MRI apparatus generates a predetermined magnetic field for a subject placed within a static magnetic field, and generates MRI image data by performing a Fourier transform on the acquired data.

[0019] The ultrasound diagnostic device transmits ultrasound to a subject, receives the ultrasound reflected from the subject, and generates ultrasound image data. The ultrasound diagnostic device can also generate ultrasound image data (moving image data) and still image data in real time.

[0020] The medical image data (CT image data, MRI image data, ultrasound image data, etc.) generated by the medical imaging device 100 is three-dimensional image data (volume data) or two-dimensional image data. The medical image data also includes raw data. The medical image data may be medical video image data consisting of multiple frame data. The medical video image data may be generated from multiple raw data.

[0021] The medical imaging apparatus 100 is connected to an information terminal 110. The information terminal 110 is a PC terminal, a mobile phone such as a smartphone, a notebook terminal, a tablet terminal, or the like. The information terminal 110 can also set subject information and can associate medical data acquired from the medical imaging apparatus 100 with the subject information. The information terminal 110 can also display various data such as medical video data and measurement data acquired from the medical imaging apparatus 100.

[0022] The information terminal 110 and the information processing device 130 are connected to a network 120. The network 120 includes a communication network outside the hospital, such as wireless communication (Wi-Fi), the Internet, a wireless base station, a provider, a communication line, etc. The network 120 may also include an intranet, which is a communication network within the hospital. The information terminal 110 can communicate with the information processing device 130 via the network 120. The information terminal 110 can transmit medical video data to the information processing device 130. The information processing device 130 can transmit an inference result obtained by performing inference on the medical video data to the information terminal 110.

[0023] 2 shows the configuration of an information processing device 130 of the present invention. The information processing device 130 includes a teacher data generating unit 200 that generates teacher data using medical video data, a learning unit 202 that performs learning on the medical video data using the teacher data generated by the teacher data generating unit 200, a storage unit 204 that stores a trained model generated by the learning unit 202, and an inference unit 206 that performs inference on the medical video data using the trained model.

[0024] The components (functions) of the information processing device 130 are realized by, for example, a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program (software) stored in a memory. The information processing device 130 has a processor and a memory inside. The processor can execute each process of the information processing device 130 according to the program stored in the memory, and can cause the information processing device 130 to function as a teacher data generating unit 200, a learning unit 202, a memory unit 204, an inference unit 206, etc.

[0025] The teacher data generating unit 200 is connected to the network 120 and can acquire medical video data. The teacher data generating unit 200 generates teacher data using the medical video data. The teacher data generated by the teacher data generating unit 200 is determined according to the inference task performed by the neural network and the target of classification.

[0026] Examples of inference tasks performed by neural networks include classification tasks to classify the classes of medical video data, detection tasks to detect what is captured in what position in the medical video data, and segmentation tasks to extract target areas from the medical video data.

[0027] When training a neural network to perform a classification task, the teacher data generation unit 200 generates teacher data that pairs medical video data with a correct answer label, which is a label indicating an object that appears in the medical video data.

[0028] On the other hand, when performing a detection task using a neural network, the teacher data generation unit 200 generates teacher data that pairs medical video data with a correct image to which an ROI (Region Of Interest) indicating the position of an object captured in the medical video data and a correct label that is a label indicating the object have been assigned.

[0029] When performing a segmentation task using a neural network, the teacher data generation unit 200 generates teacher data that pairs medical video data with a correct image to which position information of pixels of a target area that appears in the medical video data and a correct label that indicates the target area have been assigned.

[0030] For example, a neural network is trained to perform a task of segmenting the presence or absence, type, and area of ​​a lesion from medical video data acquired from the information terminal 110. In this case, the training data generating unit 200 generates training data that pairs medical video data having a lesion area with information on a correct answer label indicating the type of lesion for the medical video data and a correct answer image with position information of the pixels of the lesion.

[0031] Furthermore, the teacher data generating unit 200 may perform preprocessing of the medical video data in accordance with the neural network that is learning in the learning unit 202. For example, when the subject of inference by the neural network is medical video data, the teacher data generating unit 200 performs processes such as noise removal, filtering, image cropping, and resolution change on the acquired medical video data.

[0032] 2 shows a form in which the teacher data generating unit 200 is included inside the information processing device 130, the information terminal 110 may include the teacher data generating unit 200 therein. That is, the teacher data generating unit 200 may be included as a component of the information terminal 110. For example, after the teacher data as described above is generated by the information terminal 110, the inference device may be trained by the learning unit 202 of the information processing device 130 via the network 120.

[0033] The learning unit 202 is connected to the teacher data generating unit 200. The learning unit 202 generates a trained model by using a neural network to learn medical video data in association with teacher data. Here, the trained model indicates parameters determined by performing a learning process up to a predetermined standard and information on a model corresponding to the parameters. Note that the trained model may be used to train another model as transfer learning, or further training processing may be performed on the trained model.

[0034] A neural network includes multiple layers. Among neural networks, CNN (Convolutional Neural Network), which is a type of deep learning technology, has multiple intermediate layers between the input layer and the output layer, although not shown. The multiple intermediate layers include a convolution layer, a pooling layer, an upsampling layer, and a synthesis layer. The convolution layer is a layer that performs convolution processing on a group of input values. In the convolution layer, the input medical video data is convolved to extract features of the medical video data.

[0035] The pooling layer is a layer that performs processing to make the number of output value groups smaller than the number of input value groups by thinning out or synthesizing input value groups. The upsampling layer is a layer that performs processing to make the number of output value groups larger than the number of input value groups by duplicating input value groups or adding values ​​interpolated from the input value groups. The synthesis layer is a layer that inputs value groups such as the output value group of a certain layer or the pixel value group that constitutes medical video image data from multiple sources and synthesizes them by concatenating or adding them. The number of intermediate layers can be changed at any time depending on the learning content.

[0036] The storage unit 204 is connected to the learning unit 202. The storage unit 204 stores a trained model trained to extract the type and area of ​​a lesion in medical video data, for example. The trained model is generated using a neural network, for example. In addition to CNN, RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), which is one of deep learning technologies among neural network technologies, and models derived from CNN and RNN, other machine learning technologies such as support vector machines, logistic regression, and random forests may be used, or a method based on a rule base may be used. In addition, a trained model combining CNN and RNN, or CNN and LSTM may be used. By combining CNN, which can recognize the type and area of ​​a lesion, with RNN or LSTM, which can handle time-series data, it is possible to extract the type and area of ​​a lesion in medical video data.

[0037] The inference unit 206 is connected to the network 120 and can acquire medical video data and measurement data. The inference unit 206 is connected to the storage unit 204 and can perform inference using a trained model stored in the storage unit 204.

[0038] The inference unit 206 performs inference on the newly generated medical video data using, for example, a trained model trained to extract the type and area of ​​a lesion in the medical video data. If the newly generated medical video data contains a lesion, the inference unit 206 can output the type and area of ​​the lesion.

[0039] 3 shows a configuration of the storage unit 204 of the information processing device 130 of the present invention. The storage unit 204 associates a trained model trained in the learning unit 202 with incidental information that identifies the type of the trained model and stores them. Specifically, the storage unit 204 assigns first incidental information 310 to a first trained model 300 trained in the learning unit 202, and stores the first trained model 300. The storage unit 204 assigns second incidental information 312 to a second trained model 302 trained in the learning unit 202, and stores the second trained model 302. The storage unit 204 assigns third incidental information 314 to a third trained model 304 trained in the learning unit 202, and stores the third trained model 304.

[0040] In FIG. 3, three trained models are stored in the storage unit 204 together with the accompanying information, but the storage unit 204 can also store four or more trained models together with the accompanying information. In addition, each of the trained models is different in at least one of the characteristics of the trained medical video data, the characteristics of the teacher data, the diagnostic purpose, the class indicating the classification target, the model structure, the region information, and the site information. Each trained model can be specified or identified based on the accompanying information assigned to each trained model. In addition, the inference unit 206 in the information processing device 130 can select a trained model according to the characteristics of the medical video data acquired from the information terminal 110, and use the selected trained model to perform inference on the medical video data acquired from the information terminal 110.

[0041] The additional information includes the type of trained model. The inference unit 206 can identify the type of trained model by referring to the additional information. The additional information includes information that identifies at least one of the characteristics of trained medical video data, the characteristics of teacher data, the diagnostic purpose, the class indicating the classification target, the model structure, the region information, and the site information.

[0042] FIG. 4 shows an example of a screen display form in the display unit 114 of the present invention. The display unit 114 displays a selection screen 400 for selecting the type of trained model used by the inference unit 206. In other words, the selection screen 400 is a selection unit for selecting the trained model used by the inference unit 206. Icons suitable for diagnostic purposes are displayed on the selection screen. By selecting each icon, the operator can select a trained model for detecting stenosis (plaque), a trained model for detecting reflux, a trained model for detecting discrepancy, and the like. In addition, the operator can select all trained models by selecting the ALL icon.

[0043] Here, the icon corresponding to stenosis and reflux is selected. The inference unit 206 can specify the trained model corresponding to stenosis and reflux by referring to the additional information (diagnosis purpose). In this manner, the trained model for detecting stenosis and reflux is selected.

[0044] The display unit 114 also displays a medical image 402 based on the medical video data. The medical image 402 is, for example, an ultrasound image, and is a video image updated in real time. The inference unit 206 performs inference using a learned model corresponding to stenosis and reflux. As shown in FIG. 4, for example, when the inference unit 206 detects stenosis in the ultrasound image, the information processing device 130 transmits detection information and a region in which the stenosis is detected to the information terminal 110. The display unit 114 displays detection information 406 in which the stenosis (plaque) is detected. The display unit 114 also displays a region 404 in which the stenosis is detected. That is, when the inference unit 206 in the information processing device 130 detects a lesion in the ultrasound image, the information processing device 130 transmits detection information in which the lesion is detected to the information terminal 110. The display unit 114 displays detection information 406 in which the lesion is detected.

[0045] 5 shows the configuration of an information terminal 110 of the present invention. The information terminal 110 has a function of thinning out medical video image data, a function of performing pre-processing for inference, a function of transmitting and receiving information, and the like.

[0046] The information terminal 110 has a function of reducing the data size of medical video data. The information terminal 110 includes an image acquisition unit 502 that acquires medical video data from the medical imaging device 100, a processing unit 504 that performs a data size reduction process on the medical video data, a transmission unit 506 that transmits the medical video data processed by the processing unit 504 to an external device such as the information processing device 130 via the network 120, and a reception unit 508 that receives information transmitted from the external device such as the information processing device 130 via the network 120. The display unit 114 displays medical images (video images or still images) based on the medical video data and information transmitted from the external device such as the information processing device 130.

[0047] The processing unit 504 performs a thinning process on the first medical video data acquired by the image acquisition unit 502, and generates second medical video data having a data size smaller than the data size (data amount) of the first medical video data.

[0048] Here, the first medical video data is image data consisting of multiple frames. The processing unit 504 generates third medical video data, which is the remaining medical video data obtained by subtracting the second medical video data from the first medical video data. Here, the processing unit 504 generates medical video data (second medical video data, third medical video data) from the first medical video data, which has a smaller data size than the first medical video data.

[0049] The processing unit 504 outputs second medical video data generated by performing a thinning process on the first medical video data to the transmitting unit 506. Thereafter, the processing unit 504 outputs third medical video data, which is the remaining medical video data obtained by subtracting the second medical video data from the first medical video data, to the transmitting unit 506. The transmitting unit 506 transmits the second medical video data to an external device such as the information processing device 130, and then transmits the third medical video data to the external device such as the information processing device 130. Note that the transmitting unit 506 may transmit the second medical video data to an external device such as the information processing device 130, and then transmit the first medical video data to an external device such as the information processing device 130.

[0050] 2, the information processing device 130 (the inference unit 206) receives the second medical video data transmitted from the information terminal 110 (the transmission unit 506) via the network 120, and performs inference on the second medical video data. The inference unit 206 performs inference on the second medical video data using the trained model stored in the storage unit 204.

[0051] The inference unit 206 can specify the type of trained model by referring to the above-mentioned accompanying information. For example, the inference unit 206 specifies the type of trained model by using characteristics of trained medical video data. The inference unit 206 can specify a trained model suitable for thinned medical video data. The inference unit 206 performs inference on the second medical video data by using the trained model specified by the accompanying information.

[0052] The inference unit 206 extracts lesion information (lesion type and area) from the second medical video data. The inference unit 206 transmits the inference result obtained by performing inference on the second medical video data to the information terminal 110 via the network 120. If a lesion is present in the second medical video data, the inference unit 206 transmits the lesion information (lesion type and area). If a lesion is not present in the second medical video data, the inference unit 206 transmits information indicating that a lesion is not present in the second medical video data. The information terminal 110 (receiving unit 508) receives the inference result (lesion information) obtained by performing inference on the second medical video data.

[0053] The display unit 114 displays the inference result obtained by performing inference on the second medical video data whose data size has been reduced together with the first medical video data whose data size has not been reduced. Here, the display unit 114 does not need to display the second medical video data whose data size has been reduced.

[0054] For example, assume that the first medical video data has a 1024×1024 matrix and 1000 frames, and the second medical video data with reduced data size has a 1024×1024 matrix and 500 frames. Since the second medical video data has about half the data size (data amount) of the first medical video data, the transfer time of the second medical video data is shorter than the transfer time of the first medical video data. By transmitting the second medical video data, which has a data size smaller than the data size (data amount) of the first medical video data, to the information processing device 130 (inference unit 206) to perform inference, the inference result can be output quickly.

[0055] Furthermore, the inference unit 206 identifies a trained model suitable for medical video data obtained by integrating the second medical video data and the third medical video data. The inference unit 206 performs inference on the medical video data obtained by integrating the second medical video data and the third medical video data using a trained model suitable for non-thinned medical video data. The display unit 114 displays a first inference result obtained by inference on the second medical video data, and then displays a second inference result obtained by inference on the medical video data obtained by integrating the second medical video data and the third medical video data. This makes it possible to output an accurate inference result.

[0056] 6 to 10 show the processing form of the processing unit 504 of the present invention. Here, it is assumed that the medical video data (frame data) is ultrasound image data.

[0057] As shown in FIG. 6(a), generation of medical video data 600-614 starts at time T1. For example, transmission and reception of ultrasound starts at time T1, and ultrasound image data is generated (updated) in real time. The processing unit 504 can also set time T1 by recognizing that images of organs (liver, heart, blood vessels, etc.) are captured in the ultrasound image data for a certain period of time. The operator may set via the operation unit 112 that medical video data will be generated from time T1. The display unit 114 displays the medical video data 600-614.

[0058] At time T2, the generation of the medical video data 600-614 is stopped. The display unit 114 may display the frame data 614 (last frame data) that was generated at time T2 as still image data. At this time, the frame data 614 of the medical video data and the frame data 616 of the still image data become the same frame data. Time T2 is, for example, the timing when a freeze button is pressed to stop the generation of the medical video data. After time T2, the still image data corresponding to the frame data 614 continues to be displayed on the display unit 114.

[0059] As shown in Fig. 6(a), the processing unit 504 performs frame data thinning processing on first medical video data 600-614 (number of frame data: N) captured from time T1 to time T2. Then, the processing unit 504 generates second medical video data 620-632 (number of frame data: N / 2) having a data size smaller than the data size of the first medical video data 600-614. As shown in Fig. 6(b), the processing unit 504 selects odd-numbered frame data from the first medical video data 600-614 to generate second medical video data 620-632. The frame data 600 and frame data 620, the frame data 604 and frame data 624, and the frame data 612 and frame data 632 are the same frame data.

[0060] The processing unit 504 subtracts the second medical video data 620-632 (number of frame data: N / 2) from the first medical video data 600-614 (number of frame data: N) and generates third medical video data 642-654 (number of frame data: N / 2) which is the remaining medical video data. The processing unit 504 generates the third medical video data 642-654 from the even-numbered frame data of the first medical video data 600-614. The frame data 602 and frame data 642, the frame data 606 and frame data 646, and the frame data 614 and frame data 654 are the same frame data.

[0061] The processing unit 504 outputs the second medical video data 620-632 consisting of odd-numbered frame data to the transmitting unit 506. The transmitting unit 506 transmits the second medical video data 620-632 to the information processing device .

[0062] The inference unit 206 performs inference on the second medical video data 620-632 using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned accompanying information. The inference unit 206 identifies a trained model suitable for the medical video data from which frame data has been thinned out.

[0063] Here, the auxiliary information of the trained model stored in the memory unit 204 includes characteristics of the trained medical video data, and the characteristics of the medical video data include at least one of data size reduction information, frame data thinning information, data size, matrix size, area information, periodic information, etc.

[0064] The data size reduction information is information indicating how much the data size has been reduced compared to the original medical video data. For example, if the original medical video data has been reduced to 1 / 2 size, the data size reduction information is 1 / 2. If the original medical video data has been reduced to 1 / 3 size, the data size reduction information is 1 / 3. Furthermore, the frame data thinning information is information indicating how much the data has been thinned compared to the original medical video data. For example, if frame data of the original medical video data are alternately selected, the thinning information is 1 / 2.

[0065] For example, the learning unit 202 generates a trained model by previously learning medical video data whose frame data has been thinned out in association with teacher data. The storage unit 204 stores the trained model together with the accompanying information. The accompanying information is information indicating that training has been performed using medical video data whose frame data has been thinned out. Since the second medical video data 620-632 are medical video data whose frame data has been thinned out, the inference unit 206 performs inference on the second medical video data 620-632 using the trained model identified by the accompanying information.

[0066] The inference unit 206 transmits the inference result obtained by performing inference on the second medical video data 620-632 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the second medical video data 620-632 from which frame data has been thinned out. The operator can quickly recognize the inference result for the medical video data.

[0067] The display unit 114 displays the inference results obtained by performing inference on the second medical moving image data 620-632 together with still image data corresponding to the frame data constituting the first medical moving image data 600-614. Since the inference results obtained by performing inference on the second medical moving image data 620-632 are displayed on the display unit 114 together with the still image data, the operator can make a diagnosis using the still image data while referring to the inference results.

[0068] Time T2 is the timing when a freeze button that stops the generation of medical video data is pressed. The second medical video data 620-632 includes an imaging area captured as still image data and a peripheral area of ​​the imaging area. In the case of an ultrasound diagnostic device, an operator switches between multiple imaging sections while moving an ultrasound probe. The display unit 114 displays medical video data corresponding to multiple imaging sections. When the freeze button is pressed while medical video data corresponding to multiple imaging sections is displayed, the second medical video data 620-632 includes an imaging area corresponding to the still image data and a peripheral area of ​​the imaging area. Note that time T2 may be the timing when adjacent frame data in the medical video data are correlated and there is no change in the adjacent frame data.

[0069] When the inference unit 206 performs inference on the second medical video data 620-632, inference is performed on the imaging area corresponding to the still image data and the surrounding area of ​​the imaging area. This makes it possible to reduce overlooking of lesions and the like not only in the imaging area corresponding to the still image data but also in the surrounding area of ​​the imaging area.

[0070] In this way, when the operator uses the still image data to make a diagnosis, the operator can refer to the inference result obtained by inference on the imaging region corresponding to the still image data and the peripheral region of the imaging region. For example, if the still image data does not include a lesion, but the second medical video data 620-632 includes a lesion, the inference result obtained by inference on the second medical video data 620-632 includes the presence of a lesion. Thus, the operator can recognize that the lesion is present in the peripheral region of the imaging region corresponding to the still image data, which is the frame data 614 (last frame data) generated at time T2. The still image data 614 may be any one of the frame data constituting the first medical video data 600-614.

[0071] The processing unit 504 also has an image storage unit (not shown) that stores various image data. The image storage unit stores the still image data displayed on the display unit 114 in association with the inference result obtained by performing inference on the second medical moving image data 620-632. Therefore, when the operator reads out the still image data from the image storage unit and displays it on the display unit 114, the inference result obtained by performing inference on the second medical moving image data 620-632 is displayed, so that the operator can recognize that there is a lesion in the peripheral area of ​​the imaging area. The operator can check whether or not a reexamination is required for the subject.

[0072] Furthermore, the transmitting unit 506 transmits the thinned-out third medical video data 642-654 to the information processing device 130. The inference unit 206 performs inference on the first medical video data 600-614 obtained by integrating the second medical video data 620-632 and the third medical video data 642-654. The information processing device 130 performs processing to restore the second medical video data 620-632 and the third medical video data 642-654, from which frame data has been thinned, to the original first medical video data 600-614.

[0073] The inference unit 206 identifies a trained model suitable for medical video data whose frame data is not thinned. The inference unit 206 performs inference on the first medical video data 600-614 using the trained model suitable for medical video data whose frame data is not thinned.

[0074] The inference unit 206 transmits the inference result obtained by performing inference on the first medical moving image data 600-614 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the first medical moving image data 600-614 together with the moving image data based on the first medical moving image data 600-614. Since inference is performed on the first medical moving image data 600-614 whose frame data has not been thinned out, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

[0075] In this way, a first inference result obtained by performing inference on the second medical video data 620-632 whose frame data has been thinned out, and a second inference result obtained by performing inference on the first medical video data 600-614 whose frame data has not been thinned out are displayed in two stages on the display unit 114. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

[0076] The processing unit 504 may divide the first medical video data 600-614 into first half medical video data 600-604 which become second medical video data, and second half medical video data 606-614 which become third medical video data. Since the first half medical video data 600-604 are frame data that can be acquired in an early time period, the operator can recognize the inference result of the medical video data more quickly. Furthermore, the number of frames of the medical video data divided into the first half medical video data can be arbitrarily set by the operator via the operation unit 112. The operation unit 112 has an operation unit (setting unit) that sets the number of frames of the medical video data divided into the first half medical video data.

[0077] For example, if the operator sets the number of frames to "3", the processing unit 504 can divide the first medical video data 600-614 into the first half of the medical video data 600-604 (3 frame data) which becomes the second medical video data, and the second half of the medical video data 606-614 (N-3 frame data) which becomes the third medical video data. Also, if the operator sets the number of frames to "5", the processing unit 504 can divide the first medical video data 600-614 into the first half of the medical video data (5 frame data) which becomes the second medical video data, and the second half of the medical video data (N-5 frame data) which becomes the third medical video data.

[0078] As shown in Fig. 7, the processing unit 504 can perform a matrix size reduction process on the first medical video data 600-614. At time T1, generation of the first medical video data 600-614 is started. The processing unit 504 generates second medical video data 700-714 having frame data with a smaller matrix size than the matrix size of the frame data constituting the first medical video data 600-614. For example, if the frame data constituting the first medical video data 600-614 has a matrix size of 1024 x 1024, the frame data constituting the second medical video data 700-714 has a matrix size of 512 x 512.

[0079] The processing unit 504 outputs the second medical video data 700-714 to the transmitting unit 506. The transmitting unit 506 transmits the second medical video data 700-714 to the information processing device .

[0080] The inference unit 206 performs inference on the second medical video data 700-714 using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for the reduced matrix size.

[0081] The storage unit 204 stores the trained model together with the additional information (matrix size). The additional information is information indicating that training has been performed using medical video data composed of frame data of a predetermined matrix size. The inference unit 206 performs inference on the second medical video data 700-714 using the trained model specified by the additional information.

[0082] The inference unit 206 transmits the inference results obtained by performing inference on the second medical video data 700-714 consisting of frame data that has been subjected to a matrix size reduction process to the information terminal 110 via the network 120. The display unit 114 displays the inference results obtained by performing inference on the second medical video data 700-714 together with the medical video data based on the first medical video data 600-614. The operator can quickly recognize the inference results.

[0083] The display unit 114 displays the inference results obtained by performing inference on the second medical moving image data 700-714 together with still image data corresponding to frame data (e.g., frame data 614) constituting the first medical moving image data 600-614. Since the inference results obtained by performing inference on the second medical moving image data 700-714 are displayed on the display unit 114 together with the still image data, the operator can make a diagnosis using the still image data while referring to the inference results.

[0084] Time T2 is the timing when the freeze button for stopping the generation of the medical video data is pressed. The second medical video data 700-714 includes the imaging area captured as still image data and the surrounding area of ​​the imaging area. When the inference unit 206 performs inference on the second medical video data 700-714, inference is performed on the imaging area corresponding to the still image data and the surrounding area of ​​the imaging area. Therefore, it is possible to reduce overlooking of lesions and the like not only in the imaging area corresponding to the still image data but also in the surrounding area of ​​the imaging area.

[0085] 8, the processing unit 504 extracts a region of interest set in the first medical video data 600-614, and generates second medical video data 800-814 having a smaller data size than the first medical video data 600-614. The processing unit 504 extracts a region of interest 850 in each frame of data, and generates second medical video data 800-814 corresponding to the region of interest 850. Here, the medical video data 800-814 is generated to have the same size as the region of interest 850.

[0086] Specifically, the operator sets a region of interest 850 in still image data 680 via operation unit 112. Then, the operator switches to the motion imaging mode. The timing at which the mode is switched to the moving image imaging mode is designated as time T3.

[0087] The region of interest 850 is set when performing a measurement mode such as blood flow measurement, elasticity measurement, statistical measurement, etc. CFM (color flow mapping), which is a type of blood flow measurement, is a mode in which ultrasonic waves are transmitted and received in the same direction in the region of interest 850, the direction, speed, and dispersion of blood flow are detected, and the information is displayed in color on an ultrasonic image. Elasticity measurement is a mode in which ultrasonic waves are transmitted and received in the region of interest 850 and correlation calculations of the ultrasonic waves are performed to measure distortion and elasticity. Statistical measurement is a mode in which ultrasonic waves are transmitted and received in the region of interest 850 and a histogram based on the luminance value of the ultrasonic signal is measured. In other words, the region of interest 850 is set to an area required for diagnosis in each measurement.

[0088] The processing unit 504 outputs the second medical moving image data 800-814 corresponding to the region of interest 850 to the transmitting unit 506. The transmitting unit 506 transmits the second medical moving image data 800-814 to the information processing device .

[0089] The inference unit 206 performs inference on the second medical video data 800-814 using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for a predetermined region.

[0090] The learned model is stored together with the accompanying information in the storage unit 204. The accompanying information is information indicating that the model has been learned using medical video data consisting of frame data of a region of interest corresponding to the type of measurement mode (blood flow measurement, elasticity measurement, statistical measurement, etc.).

[0091] Since the second medical video data 800-814 is medical video data consisting of frame data of the region of interest 850 of blood flow measurement, the inference unit 206 performs inference on the second medical video data 800-814 using a learned model identified by the supplementary information (blood flow measurement + region of interest).

[0092] The inference unit 206 transmits the inference results obtained by performing inference on the second medical moving image data 800-814 to the information terminal 110 via the network 120. The display unit 114 displays the inference results obtained by performing inference on the second medical moving image data 800-814 together with the still image data 616 corresponding to the frame data constituting the first medical moving image data 600-614. The operator can quickly recognize the inference results.

[0093] The display unit 114 displays the inference results obtained by performing inference on the second medical moving image data 800-814 together with the still image data 616 corresponding to the frame data (e.g., frame data 614) constituting the first medical moving image data 600-614. Since the inference results obtained by performing inference on the second medical moving image data 800-814 are displayed on the display unit 114 together with the still image data, the operator can make a diagnosis using the still image data while referring to the inference results.

[0094] Time T4 is the timing when the freeze button for stopping the generation of the medical video data is pressed. The second medical video data 800-814 includes the region of interest 850 and the surrounding area of ​​the region of interest 850 captured as the still image data 616 corresponding to the frame data constituting the first medical video data 600-614. When the inference unit 206 performs inference on the second medical video data 800-814, inference is performed on the region of interest 850 and the surrounding area of ​​the region of interest 850. This makes it possible to reduce overlooking not only the region of interest 850 in the still image data 616 corresponding to the frame data constituting the first medical video data 600-614, but also a lesion or the like in the surrounding area of ​​the region of interest 850.

[0095] As shown in FIG. 9, the processing unit 504 can extract medical video data for each period. Ultrasound image data in the circulatory system is generated in synchronization with a biosignal such as an electrocardiogram (ECG) acquired from the subject. For example, the processing unit 504 measures the heartbeat phase based on the subject's biosignal in parallel with the generation of time-series ultrasound image data for the subject. A predetermined threshold is set for the electrocardiogram waveform measured from the subject, and a time-series trigger signal (R wave, P wave, etc.) is generated. The processing unit 504 can measure the interval between trigger signals (trigger signal interval) and measure period information based on the interval between the acquired trigger signals. Note that period information can also be measured from a blood flow signal (blood flow direction, speed, and dispersion) detected using CFM.

[0096] The processing unit 504 extracts second medical moving image data 900-906 (first cycle) from the first medical moving image data generated in real time using cycle information (cardiac cycle) of the subject. The second medical moving image data 900-906 is medical moving image data corresponding to one cycle detected first using cycle information (cardiac cycle). In other words, the information terminal 110 extracts medical moving image data corresponding to one cycle detected first. Note that the extracted medical moving image data does not have to be the one cycle detected first, and the operator can also arbitrarily extract medical moving image data corresponding to one cycle using cycle information (cardiac cycle) via the operation unit 112.

[0097] The processing unit 504 outputs the second medical video data 900-906 (first cycle), which is one cycle detected first, to the transmitting unit 506. The transmitting unit 506 transmits the second medical video data 900-906 (first cycle) to the information processing device 130. It is possible to set so that the third medical video data 908-914 (second cycle), which is the cycle next to the first cycle, is not transmitted to the information processing device 130. In other words, the information terminal 110 can also be set so as not to transmit medical video data other than the first cycle to the information processing device 130.

[0098] The inference unit 206 performs inference on the second medical video data 900-906 (first cycle) using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for the medical video data corresponding to the cycle information (cardiac cycle).

[0099] For example, the learning unit 202 generates a trained model by previously learning medical video data corresponding to cycle information in association with teacher data. The memory unit 204 stores the trained model together with accompanying information. The accompanying information of the trained model is information indicating that the trained model has been trained using medical video data corresponding to cycle information. Since the second medical video data 900-906 are medical video data corresponding to the first cycle, the inference unit 206 performs inference on the second medical video data 900-906 using the trained model identified by the cycle information (cardiac cycle) which is the accompanying information.

[0100] The inference unit 206 transmits the inference results obtained by performing inference on the second medical video data 900-906 to the information terminal 110 via the network 120. As shown in Fig. 9(a), the display unit 114 displays the inference results obtained by performing inference on the second medical video data 900-906 together with the third medical video data 908-914 (second cycle) and medical video data from the second cycle onwards.

[0101] That is, the information terminal 110 extracts medical moving image data corresponding to at least one cycle using cycle information based on a biological signal acquired from a subject, and transmits the extracted data to the information processing device 130. Then, the inference unit 206 performs inference on the medical moving image data corresponding to at least one cycle, and the information processing device 130 transmits the inference result to the information terminal 110. The display unit 114 displays the inference result obtained by performing inference on the second medical moving image data 900-906 corresponding to the first cycle together with the third medical moving image data 908-914 (second cycle) and medical moving image data from the second cycle onward, which are generated in real time, so that the operator can quickly recognize the inference result.

[0102] When the display unit 114 displays the third medical video data 908-914 (second cycle) and medical video data from the second cycle onwards, if the inference results obtained by inference on the second medical video data 900-906 (first cycle) have not been transmitted to the information terminal 110 via the network 120, the display unit 114 displays information indicating that inference is in progress.

[0103] The processing unit 504 also has an image storage unit that stores various image data. The image storage unit stores the second medical moving image data 900-906 (first cycle) in association with an inference result obtained by performing inference on the second medical moving image data 900-906 (first cycle). Thus, when the operator reads out the second medical moving image data 900-906 (first cycle) from the image storage unit and displays it on the display unit 114, the operator can recognize that there is a lesion in the second medical moving image data 900-906 (first cycle) by displaying the inference result obtained by performing inference on the second medical moving image data 900-906 (first cycle). The operator can confirm whether or not a reexamination is required for the subject.

[0104] In this way, the display objects displayed together with the inference results obtained by performing inference on the second medical video data 900-906 (first cycle) are the third medical video data 908-914 (second cycle) and medical video data from the second cycle onwards. The storage object stored together with the inference results obtained by performing inference on the second medical video data 900-906 (first cycle) is the second medical video data 900-906 (first cycle). In other words, the display object displayed on the display unit 114 and the storage object stored in the image storage unit are different.

[0105] As shown in Fig. 9(b), the display unit 114 displays information indicating that inference is in progress for the third medical video data 908-914 (second cycle) and medical video data from the second cycle onwards. Here, when the frame data 908 of the third medical video data is being generated, the inference result performed by the inference unit 206 has not been transmitted. When the frame data 908 of the third medical video data is being generated, inference is being performed by the inference unit 206. At this time, the display unit 114 displays information indicating that inference is in progress.

[0106] The display unit 114 displays the inference results obtained by performing inference on the second medical video data 900-906, together with the third medical video data 910-914 and the second and subsequent cycles of medical video data.

[0107] 10, the processing unit 504 extracts medical video data for each period. The processing unit 504 extracts medical video data 900-906 (first period), medical video data 908-914 (second period), and medical video data 916-922 (third period) from the first medical video data generated in real time using period information (cardiac period). The medical video data 900-906 are medical video data corresponding to the period detected first using the period information (cardiac period). The medical video data 908-914 (second period) are medical video data corresponding to the period detected second using the period information (cardiac period). The medical video data 916-922 (third period) are medical video data corresponding to the period detected third using the period information (cardiac period).

[0108] The operator sets a cycle of interest (number of cycles) via the operation unit 112 (setting unit). The processing unit 504 extracts medical video data corresponding to the number of cycles set by the operation unit 112 (setting unit). Here, it is assumed that the first cycle to the third cycle (three cycles) are set as the cycle of interest.

[0109] The processing unit 504 outputs the second medical video data (medical video data 900-906 (first cycle), medical video data 908-914 (second cycle), medical video data 916-922 (third cycle)) to the transmitting unit 506. The transmitting unit 506 transmits the second medical video data to the information processing device 130. The third medical video data 950- (fourth cycle and after), which is the next detected cycle, is not transmitted to the information processing device 130.

[0110] The inference unit 206 performs inference on the second medical video data 900-922 (first cycle to third cycle) using the trained model stored in the storage unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned accompanying information. The inference unit 206 identifies a trained model suitable for the medical video data corresponding to multiple pieces of cycle information (multiple cardiac cycles).

[0111] For example, the learning unit 202 generates a trained model by previously learning medical video data corresponding to a plurality of pieces of periodic information in association with teacher data. The storage unit 204 stores the trained model together with the accompanying information. The accompanying information is information indicating that the training was performed using medical video data corresponding to a plurality of pieces of periodic information. Since the second medical video data 900-922 (first to third cycles) are medical video data corresponding to the first to third cycles, the inference unit 206 performs inference on the second medical video data 900-922 (first to third cycles) using the trained model specified by the periodic information of the accompanying information.

[0112] The inference unit 206 transmits the inference results obtained by performing inference on the second medical moving image data 900-922 (first to third cycles) to the information terminal 110 via the network 120. The display unit 114 displays the inference results obtained by performing inference on the second medical moving image data 900-922 (first to third cycles) together with the video data corresponding to the third medical moving image data 950- (fourth cycles and after). Since the display unit 114 displays the inference results obtained by performing inference on the second medical moving image data 900-922 (first to third cycles) together with the video data generated in real time, the operator can quickly recognize the inference results.

[0113] The operation of the learning phase of the present invention will be described with reference to FIG.

[0114] The information terminal 110 (image acquisition unit 502) acquires the medical video data from the medical imaging device 100.

[0115] S802: The operator selects whether or not to reduce the data size (amount of data) of the medical video data via the operation unit 112. If the data size of the medical video data is not to be reduced, the process proceeds to S804. If the data size of the medical video data is to be reduced, the process proceeds to S810.

[0116] S804: The information terminal 110 (transmission unit 506) transmits the medical video data to the information processing device 130. The information processing device 130 (teacher data generation unit 200) generates teacher data that pairs the medical video data with a correct answer label. The information processing device 130 (learning unit 202) generates a trained model by learning the medical video data in association with the teacher data using a neural network.

[0117] S806: The information processing device 130 (the storage unit 204) stores the trained model trained by the training unit 202 and the associated information.

[0118] S810: The information terminal 110 (processing unit 504) performs data size reduction processing on the medical video data acquired by the image acquisition unit 502. The processing unit 504 generates medical video data with a data size smaller than the data size (data amount) of the medical video data. This processing results in the generation of medical video data with a plurality of data sizes. Here, as the data size reduction processing, thinning processing, matrix size reduction processing, region of interest extraction processing, etc. are performed on the medical video data.

[0119] S812: The information terminal 110 (transmission unit 506) transmits medical video data of multiple data sizes to the information processing device 130. The information processing device 130 (teacher data generation unit 200) generates teacher data that pairs each medical video data with a correct label. The information processing device 130 (learning unit 202) uses a neural network to learn each medical video data in association with the teacher data, thereby generating multiple trained models.

[0120] S814: The information processing device 130 (the storage unit 204) stores a plurality of trained models trained by the training unit 202 and associated information (types of data size reduction processing) as shown in FIG.

[0121] The operation of the inference phase of the present invention will be described with reference to FIG.

[0122] The information terminal 110 (image acquisition unit 502) acquires the medical video data from the medical imaging device 100. S900: The medical imaging device 100 captures an image of a subject and acquires medical video data.

[0123] S902: The operator selects whether or not to reduce the data size (data amount) of the medical video data via the operation unit 112. If the data size of the medical video data is not to be reduced, the process proceeds to S904. If the data size of the medical video data is to be reduced, the process proceeds to S910.

[0124] S904: The information terminal 110 (the transmission unit 506) transmits the medical video data to the information processing device 130. The information processing device 130 (the inference unit 206) uses the trained model stored in the storage unit 204 to perform inference on the medical video data.

[0125] S906: The information processing device 130 (the inference unit 206) transmits the inference result obtained by performing inference on the medical video data to the information terminal 110. The display unit 114 displays the inference result obtained by performing inference on the medical video data together with still image data based on the medical video data. The display unit 114 can switch between displaying the still image data and the medical video data to be displayed.

[0126] S910: The information terminal 110 (processing unit 504) performs data size reduction processing on the medical video data acquired by the image acquisition unit 502. The processing unit 504 generates medical video data with a data size smaller than the data size (data amount) of the medical video data.

[0127] S912: The information terminal 110 (the inference unit 206) performs inference on the medical video data using the trained model stored in the storage unit 204. The inference unit 206 identifies a trained model suitable for the medical video data whose data size has been reduced using the supplementary information (type of data size reduction process). The inference unit 206 performs inference on the medical video data using the trained model suitable for the medical video data whose data size has been reduced.

[0128] S914: The display unit 114 displays a first inference result obtained by performing inference on the medical video data whose data size has been reduced, together with still image data based on the medical video data. The display unit 114 may also display a second inference result obtained by performing inference on the medical video data whose data size has not been reduced. The display unit 114 displays the inference results in the order in which they were transmitted from the information terminal 110 (inference unit 206). The display unit 114 can switch between displaying the still image data and the medical video data to be displayed.

[0129] As described above, the information processing system of the present invention includes an information terminal 110 that acquires medical video data from a medical imaging device 100, and an information processing device 130 having an inference unit 206 that performs inference on the medical video data using a trained model. The information terminal 110 and the information processing device 130 are connected via a network 120. The information terminal 110 reduces the data size of the medical video data and transmits it to the information processing device 130. The inference unit 206 performs inference on the medical video data with the reduced data size, and transmits the inference result to the information terminal 110.

[0130] The information terminal 110 is an information terminal connected to an information processing device 130 that performs inference on medical video data via a network 120. The information terminal 110 includes an acquisition unit 502 that acquires medical video data from the medical imaging device 100, and a processing unit 504 that reduces the data size of the medical video data and generates medical video data with reduced data size. The information terminal 110 also includes a transmission unit 506 that transmits the medical video data with reduced data size to the information processing device 130, and a reception unit 508 that receives, via the network 120, from the information processing device 130, an inference result obtained by performing inference on the medical video data with reduced data size in the information processing device 130.

[0131] The display unit 114 displays the inference result together with the medical video image (ultrasound image) generated in real time, so that the operator can quickly recognize the inference result. In this way, it is possible to perform inference on the medical video image data using the trained model and output the inference result. EXAMPLES

[0132] An information processing system (information processing device 130) according to a second embodiment of the present invention will be described with reference to Fig. 13. The difference from the first embodiment is that an integration processing unit 210 that integrates multiple inference results is provided. The integration processing unit 210 includes an inference result storage unit 800 that stores multiple inference results inferred by the inference unit 206, and an addition processing unit 802 that adds up the multiple inference results.

[0133] The inference unit 206 performs inference on the second medical video data with reduced data size, using the trained model stored in the storage unit 204. The inference unit 206 transmits a first inference result (initial inference result) obtained by performing inference on the second medical video data, to the information terminal 110 via the network 120. The display unit 114 displays the first inference result obtained by performing inference on the second medical video data with reduced data size.

[0134] Moreover, the inference unit 206 transmits a first inference result obtained by performing inference on the second medical video image data to the integration processing unit 210. The inference result storage unit 800 stores the first inference result.

[0135] The inference unit 206 performs inference on the third medical video data with reduced data size, using the trained model stored in the memory unit 204. The inference unit 206 transmits a second inference result (next inference result) obtained by performing inference on the third medical video data to the integrated processing unit 210. The inference result memory unit 800 stores the second inference result. Here, the inference result memory unit 800 stores the first inference result and the second inference result, but may store three or more inference results.

[0136] The addition processing unit 802 performs an addition process on a plurality of inference results (a first inference result and a second inference result) stored in the inference result storage unit 800. At this time, the objects of addition in the addition processing unit 802 are a plurality of inference results obtained by inferring the second medical video image data generated from the first medical video image data obtained by imaging the subject, and the third medical video image data.

[0137] The second medical video data and the third medical video data are generated at different extraction periods. Also, the second medical video data and the third medical video data may be generated by different data reduction processes.

[0138] The addition unit 802 can adjust the addition weighting, and can perform the addition process by weighting the second inference result heavier than the first inference result.

[0139] The integration processing unit 210 transmits an inference result obtained by integrating a plurality of inference results to the information terminal 110 via the network 120. The display unit 114 displays the integrated inference result.

[0140] As described above, the information processing system of the present invention includes the integration processing unit 210 that integrates a plurality of inference results. By integrating a plurality of inference results, it is possible to improve the inference accuracy. EXAMPLES

[0141] An information processing system (information processing device 130) according to a third embodiment of the present invention will be described with reference to Fig. 14. The difference from the first and second embodiments is that the medical imaging device 100 has the functions of the information terminal 110 described above.

[0142] The medical imaging apparatus 100 is an apparatus that generates medical moving image data of a subject, such as an X-ray CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, or an ultrasonic diagnostic apparatus.

[0143] The medical imaging device 100 includes an image generation unit 510 that images a subject and generates medical video image data, a processing unit 504 that performs data size reduction processing on the medical video image data, a transmission unit 506 that transmits the medical video image data processed by the processing unit 504 to an external device such as an information processing device 130 via the network 120, and a receiving unit 508 that receives information transmitted from an external device such as the information processing device 130 via the network 120.

[0144] The information processing system of this embodiment includes a medical imaging device 100 that images a subject and generates medical video data, and an information processing device 130 having an inference unit 206 that performs inference on the medical video data using a trained model. The medical imaging device 100 and the information processing device 130 are connected via a network 120. The medical imaging device 100 reduces the data size of the medical video data and transmits it to the information processing device 130, the inference unit 206 performs inference on the medical video data with the reduced data size, and the information processing device 130 transmits the inference result to the medical imaging device 100.

[0145] The medical imaging apparatus 100 is a medical imaging apparatus connected to an information processing device 130 that performs inference on medical video data via a network 120. The medical imaging apparatus 100 includes an image generating unit 502 that generates medical video data by imaging a subject, and a processing unit 504 that reduces the data size of the medical video data to generate medical video data with reduced data size. The medical imaging apparatus 100 also includes a transmitting unit 506 that transmits the medical video data with reduced data size to the information processing device 130, and a receiving unit 508 that receives, from the information processing device 130, an inference result obtained by performing inference on the medical video data with reduced data size in the information processing device 130 via the network 120. Note that the details of the functions of the processing unit 504, the transmitting unit 506, and the receiving unit 508 have been described in the first embodiment, and therefore repeated description will be omitted.

[0146] The display unit 114 displays the inference result together with the medical video data (ultrasound image) generated in real time, so that the operator can quickly recognize the inference result. In this way, it is possible to perform inference on the medical video data using the trained model and output the inference result.

[0147] A computer program for realizing the functions of the above-mentioned embodiments can be supplied to a computer via a network or memory (not shown), and the computer program can be executed by a processor (not shown). This is a computer program for causing a computer to execute the above-mentioned information processing method. In other words, the computer program is a program for causing a computer to realize the functions of an information processing device. The memory stores the computer program. Moreover, each embodiment can be realized in combination. [Explanation of symbols]

[0148] 100 Medical imaging device 110 Information terminal 112 Operation section 114 Display section 120 Network 130 Information processing equipment 200 Teacher Data Generation Unit 202 Learning Department 204 Storage section 206 Reasoning section 502 Image acquisition unit 504 Processing section 506 Transmitter 508 Receiving unit

Claims

1. an information terminal that acquires medical video data of a subject from a medical imaging device; an information processing device having an inference unit that performs inference on the medical video image data by using a trained model; the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical video data and transmits it to the information processing device; the inference unit performs inference on the medical video image data whose data size has been reduced, and the information processing device transmits an inference result to the information terminal; An information processing system characterized in that the information terminal transmits the remaining medical video image data that has not been transmitted to the information processing device to the information processing device, the inference unit performs inference on medical video image data in which the reduced data size and the remaining medical video image data are integrated, and the information processing device transmits the inference result to the information terminal.

2. The information processing system according to claim 1, characterized in that the information terminal is provided with a processing unit that thins out frame data from first medical video image data to generate second medical video image data having a reduced data size.

3. The information processing system according to claim 1, characterized in that the information terminal is provided with a processing unit that performs a matrix size reduction process on first medical video image data and generates second medical video image data having a reduced data size.

4. The information processing system according to claim 1 , wherein the information terminal includes a processing unit that extracts a region of interest from first medical video data and generates second medical video data corresponding to the region of interest.

5. The information processing system according to claim 1, characterized in that the inference unit selects a trained model based on characteristics of the medical video data acquired from the information terminal, and performs inference on the medical video data using the selected trained model.

6. The information processing system according to claim 1, further comprising a memory unit that links and stores the trained model with additional information that identifies a type of the trained model, and the inference unit identifies the type of the trained model by referring to the additional information.

7. The information processing system according to claim 1, characterized in that the information terminal comprises an image acquisition unit that acquires the medical video data from the medical imaging device, and a processing unit that performs thinning processing on a plurality of frame data constituting the medical video data.

8. The information processing system according to claim 1, further comprising a display unit that displays the inference results obtained by performing inference on the medical video image data whose data size has been reduced, together with the medical video image data whose data size has not been reduced.

9. The information processing system according to claim 1, further comprising a display unit that displays the inference results obtained by performing inference on the medical video image data whose data size has been reduced, together with still image data based on the medical video image data.

10. The information processing system according to claim 9, further comprising a memory unit that links and stores the still image data displayed on the display unit with the inference results obtained by performing inference on the medical video image data whose data size has been reduced.

11. The information processing system according to claim 8, characterized in that the display unit displays a first inference result obtained by performing inference on the medical video image data whose data size has been reduced, and then displays a second inference result obtained by performing inference on the medical video image data whose data size has not been reduced.

12. 12. The information processing system according to claim 11, further comprising an integration processing unit that integrates the first inference result and the second inference result.

13. The information processing system according to claim 12, characterized in that the integration processing unit includes an inference result memory unit that stores a plurality of inference results inferred by the inference unit, and an addition processing unit that adds up the plurality of inference results.

14. a medical imaging device for imaging a subject and generating medical video data; an information processing device having an inference unit that performs inference on the medical video image data by using a trained model; the medical imaging device and the information processing device are connected via a network, the medical imaging device reduces a data size of the medical video image data and transmits the data to the information processing device, the inference unit performs inference on the medical video image data whose data size has been reduced, and the information processing device transmits an inference result to the medical imaging device, An information processing system characterized in that the medical imaging device transmits the remaining medical video image data that has not been transmitted to the information processing device to the information processing device, the inference unit performs inference on medical video image data in which the reduced data size and the remaining medical video image data are integrated, and the information processing device transmits the inference result to the medical imaging device.

15. an information terminal that acquires medical video data of a subject from a medical imaging device; an information processing device having an inference unit that performs inference on the medical video image data by using a trained model; the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical video data and transmits it to the information processing device; the inference unit performs a first inference on the medical video image data whose data size has been reduced, and the information processing device transmits a result of the first inference to the information terminal; the information terminal transmits to the information processing device other medical video data included in the medical video data that has not been transmitted to the information processing device; The information processing system, wherein the inference unit performs a second inference on the other medical video image data, and the information processing device transmits a result of the second inference to the information terminal.

16. A medical imaging apparatus connected via a network to an information processing apparatus that performs inference on medical video image data, an image generating unit that captures an image of a subject and generates medical video image data of the subject; a processing unit that reduces a data size of the medical video data and generates the medical video data having the reduced data size; a transmission unit that transmits the medical video data whose data size has been reduced to the information processing device; a receiving unit that receives, via the network, from the information processing device, an inference result obtained by performing inference on the medical video image data whose data size has been reduced by using a trained model in the information processing device; the transmission unit further transmits to the information processing device the remaining medical video image data that has not been transmitted to the information processing device; The medical imaging device is characterized in that the receiving unit receives from the information processing device via the network the inference results performed by the information processing device on medical video image data in which the reduced data size and the remaining medical video image data are integrated.

17. A medical imaging apparatus connected via a network to an information processing apparatus that performs inference on medical video image data, an image generating unit that captures an image of a subject and generates medical video image data of the subject; a processing unit that reduces a data size of the medical video data and generates the medical video data having the reduced data size; a transmission unit that transmits the medical video data whose data size has been reduced to the information processing device; a receiving unit that receives, via the network, from the information processing device, an inference result obtained by performing inference on the medical video image data whose data size has been reduced by using a trained model in the information processing device; a processing unit that performs a matrix size reduction process on the first medical video data to generate second medical video data having a reduced data size; A medical imaging apparatus comprising:

18. A medical imaging apparatus connected via a network to an information processing apparatus that performs inference on medical video image data, an image generating unit that captures an image of a subject and generates medical video image data of the subject; a processing unit that reduces a data size of the medical video data and generates the medical video data having the reduced data size; a transmission unit that transmits the medical video data whose data size has been reduced to the information processing device; a receiving unit that receives, via the network, from the information processing device, an inference result obtained by performing inference on the medical video image data whose data size has been reduced by using a trained model in the information processing device; a display unit that displays an inference result obtained by performing an inference on the medical video data whose data size has been reduced, together with still image data based on the medical video data; a storage unit that associates and stores the still image data displayed on the display unit with an inference result obtained by performing an inference on the medical video image data whose data size has been reduced; A medical imaging apparatus comprising:

19. A medical imaging apparatus connected via a network to an information processing apparatus that performs inference on medical video image data, an image generating unit that captures an image of a subject and generates medical video image data of the subject; a processing unit that reduces a data size of the medical video data and generates the medical video data having the reduced data size; a transmission unit that transmits the medical video data whose data size has been reduced to the information processing device; a receiving unit that receives, via the network, from the information processing device, a first inference result obtained by performing a first inference on the medical video image data whose data size has been reduced by using a trained model in the information processing device; the transmission unit further transmits, to the information processing device, other medical video data included in the medical video data that has not been transmitted to the information processing device; The receiving unit performs a second inference on the other medical video image data by using a trained model, and the information processing device receives a result of the second inference from the information processing device. Imaging device for.

20. An information terminal connected via a network to an information processing device that performs inference on medical video image data, an acquisition unit that acquires medical video image data of a subject from a medical imaging device; a processing unit that reduces a data size of the medical video data and generates the medical video data having the reduced data size; a transmission unit that transmits the medical video data whose data size has been reduced to the information processing device; a receiving unit that receives, via the network, from the information processing device, an inference result obtained by performing inference on the medical video image data whose data size has been reduced by using a trained model in the information processing device; the transmission unit further transmits to the information processing device the remaining medical video image data that has not been transmitted to the information processing device; The information terminal is characterized in that the receiving unit receives from the information processing device via the network an inference result obtained by performing inference on medical video image data in which the reduced data size and the remaining medical video image data are integrated, using a trained model in the information processing device.

21. 1. An information processing method executed by a processor, comprising: acquiring medical video image data of a subject from a medical imaging device; and performing inference on the medical video image data using the trained model, reducing a data size of the medical video data; transmitting the medical video data whose data size has been reduced from the information terminal to an information processing device; a step of performing inference on the medical video image data whose data size has been reduced by the information processing device, and transmitting an inference result to the information terminal; transmitting remaining medical video image data that has not been transmitted to the information processing device from the information terminal to the information processing device; a step of performing inference on the medical video data obtained by integrating the medical video data whose data size has been reduced and the remaining medical video data by the information processing device, and transmitting an inference result to the information terminal; 13. An information processing method comprising:

22. 1. An information processing method executed by a processor, comprising: acquiring medical video image data of a subject from a medical imaging device; and performing inference on the medical video image data using the trained model, reducing a data size of the medical video data; transmitting the medical video data whose data size has been reduced from the information terminal to an information processing device; a step of performing a first inference on the medical video image data whose data size has been reduced by the information processing device, and transmitting a first inference result to the information terminal; transmitting, to the information processing device, other medical video data included in the medical video data that has not been transmitted to the information processing device; a step of performing a second inference on the other medical video image data by the information processing device, and transmitting a second inference result to the information terminal; 13. An information processing method comprising:

23. 23. A program for causing a computer to execute the information processing method according to claim 21 or 22.

Images