Machine Learning Model Evaluation Device, Machine Learning Model Evaluation Method, and Program

Abstract

To provide a machine learning model evaluation device, a machine learning model evaluation method, and a program that can fully utilize an analysis medical AI application.SOLUTION: A machine learning model evaluation device according to the embodiment includes an acquisition unit, a processing unit, and a calculation unit. The acquisition unit acquires a machine learning model that generates output data for medical data input. The processing unit generates target output data by inputting target medical data generated on the basis of a second condition different from a first condition under which semi-medical data is generated to the machine learning model. The calculation unit calculates, on the basis of a comparison result between reference output data generated by inputting the reference medical data to the machine learning model and the target output data, goodness of fit of the second condition to the machine learning model.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to a machine learning model evaluation apparatus, a machine learning model evaluation method, and a program.

Background Art

[0002] In recent years, in medical treatment, various artificial intelligence (AI) application software for analytical medical treatment (hereinafter referred to as analytical medical treatment AI applications) has been supplied by vendors. The analytical medical treatment AI applications execute various analyzes and calculations on behalf of technicians and the like. Therefore, medical institutions can improve the efficiency of medical treatment by introducing analytical medical treatment AI applications. Conditions used in this type of analytical medical treatment AI application, for example, imaging conditions of images input to the analytical medical treatment AI application used for image analysis, differ from medical institution to medical institution.

[0003] Although the types of analytical medical treatment AI applications announced in recent years tend to increase, the optimal usage conditions differ for each analytical medical treatment AI application. However, the processing inside the analytical medical treatment AI application is often black-boxed, and medical institutions have not been able to grasp the conditions suitable for the introduced analytical medical treatment AI application. For this reason, medical institutions have sometimes been unable to fully utilize the introduced analytical medical treatment AI application.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the embodiments disclosed in this specification and the drawings is to enable the full utilization of the analytical diagnosis AI application. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problems. The problems corresponding to the respective effects of each configuration shown in the embodiments described later can also be regarded as other problems.

Means for Solving the Problem

[0006] The machine learning model evaluation device of the embodiment has an acquisition unit, a processing unit, and a calculation unit. The acquisition unit acquires a machine learning model that generates output data for an input of medical data. The processing unit generates target output data by inputting target medical data generated based on a second condition different from the first condition under which the reference medical data is generated into the machine learning model. The calculation unit calculates the degree of fitness of the machine learning model for the second condition based on the comparison result between the reference output data generated by inputting the reference medical data into the machine learning model and the target output data.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Best Mode for Carrying Out the Invention

[0008] Hereinafter, a machine learning model evaluation apparatus, a machine learning model evaluation method, and a program according to an embodiment will be described with reference to the drawings.

[0009] (First Embodiment) FIG. 1 is a diagram showing a configuration example of a machine learning model evaluation system 1 according to the first embodiment. The machine learning model evaluation system 1 includes, for example, a medical imaging device 100, a medical treatment device 200, and a machine learning model evaluation device 300. The medical imaging device 100, the medical treatment device 200, and the machine learning model evaluation device 300 are communicably connected via a communication network NW.

[0010] The medical imaging device 100 is a device that generates a medical image by scanning a subject P. The subject P is, for example, a human patient, but is not limited thereto, and may be other animals or inanimate objects. The medical imaging device 100 may be, for example, an X-ray CT device, and may also be an MRI device, an ultrasonic diagnostic device, a nuclear medicine diagnostic device, or the like. The medical imaging device 100 is a so-called modality. Hereinafter, as an example, the case where the medical imaging device 100 is an X-ray CT device will be described.

[0011] The medical treatment device 200 is configured to include, for example, a personal computer or a smartphone in which an analytical medical treatment AI application provided by a vendor is installed. The medical treatment device 200 acquires, for example, image data provided by the X-ray CT device 100. The medical treatment device 200 calculates an analysis result by analyzing the image data with the analytical medical treatment AI application. The analytical medical treatment AI application includes a machine learning model (trained model).

[0012] There are various types of AI applications for medical diagnosis and treatment, such as Visualization-based, Workflow-based, measurement-based, and diagnosis-based applications. For Visualization-based applications, there are, for example, applications that execute Tracking (tracking), Stabilizer (stabilization), Needle (needle puncture), interpolation (spatiotemporal), super-resolution, magnification, emphasis, etc. on images to obtain analysis results. For Workflow-based applications, there are, for example, applications that generate a schedule to save time and obtain analysis results.

[0013] For measurement-based applications, there are, for example, applications that execute Segmentation (segmentation), Measurement (measurement), Quantification (quantification), etc. to obtain analysis results. For diagnosis-based applications, there are, for example, applications that execute Detection (detection), Triage (severity diagnosis), findings, prognosis prediction, etc. to obtain analysis results. The medical diagnosis and treatment device 200 may be incorporated into the X-ray CT device 100, which is a modality, or may be a workstation or cloud server connected to the X-ray CT device 100.

[0014] The machine learning model evaluation device 300 collects medical images from the X-ray CT device (medical imaging device) 100 and evaluates the machine learning model introduced into the medical diagnosis and treatment device 200 using the collected medical images. Details of the machine learning model will be described later. The machine learning model evaluation device 300 may be incorporated into the X-ray CT device 100 or the medical diagnosis and treatment device 200, or may be a workstation or cloud server connected to the X-ray CT device 100 or the medical diagnosis and treatment device 200.

[0015] FIG. 2 is a diagram showing a configuration example of the X-ray CT apparatus 100 in the first embodiment. The X-ray CT apparatus 100 includes, for example, a gantry device 110, a couch device 130, and a console device 140. In FIG. 2, for convenience of explanation, both a view of the gantry device 110 as seen from the Z-axis direction and a view as seen from the X-axis direction are shown, but actually, there is only one gantry device 110. In the embodiment, the rotation axis of the rotating frame 117 or the longitudinal direction of the top plate 133 of the couch device 130 in the non-tilt state is defined as the Z-axis direction, an axis that is orthogonal to the Z-axis direction and horizontal with respect to the floor surface is defined as the X-axis direction, and a direction that is orthogonal to the Z-axis direction and perpendicular to the floor surface is defined as the Y-axis direction, respectively.

[0016] The gantry device 110 includes, for example, an X-ray tube 111, a wedge 112, a collimator 113, an X-ray high voltage device 114, an X-ray detector 115, a data acquisition system (hereinafter, DAS: Data Acquisition System) 116, a rotating frame 117, and a control device 118.

[0017] The X-ray tube 111 generates X-rays by irradiating thermoelectrons from the cathode (filament) toward the anode (target) by applying a high voltage from the X-ray high voltage device 114. The X-ray tube 111 includes a vacuum tube. For example, the X-ray tube 111 is a rotating anode type X-ray tube that generates X-rays by irradiating thermoelectrons to a rotating anode.

[0018] The wedge 112 is a filter for adjusting the amount of X-rays irradiated from the X-ray tube 111 to the subject P. The wedge 112 attenuates the X-rays passing through itself so that the distribution of the amount of X-rays irradiated from the X-ray tube 111 to the subject P becomes a predetermined distribution. The wedge 112 is also called a wedge filter or a bow-tie filter. The wedge 112 is, for example, made by processing aluminum to have a predetermined target angle and a predetermined thickness.

[0019] The collimator 113 is a mechanism for narrowing the irradiation range of the X-rays that have passed through the wedge 112. The collimator 113 narrows the irradiation range of the X-rays, for example, by forming a slit with a combination of a plurality of lead plates. The collimator 113 may also be referred to as an X-ray aperture.

[0020] The X-ray high voltage device 114 has, for example, a high voltage generator and an X-ray control device. The high voltage generator has an electric circuit including a transformer and a rectifier, etc., and generates a high voltage to be applied to the X-ray tube 111. The X-ray control device controls the output voltage of the high voltage generator according to the amount of X-rays to be generated by the X-ray tube 111. The high voltage generator may perform voltage boosting by the above-described transformer, or may perform voltage boosting by an inverter. The X-ray high voltage device 114 may be provided on the rotating frame 117, or may be provided on the side of a fixed frame (not shown) of the gantry device 110.

[0021] The X-ray detector 115 detects the intensity of the X-rays generated by the X-ray tube 111 and incident after passing through the subject P. The X-ray detector 115 outputs an electrical signal (which may also be an optical signal, etc.) corresponding to the detected intensity of the X-rays to the DAS 116. The X-ray detector 115 has, for example, a plurality of X-ray detection element arrays. Each of the plurality of X-ray detection element arrays has a plurality of X-ray detection elements arranged in the channel direction along an arc centered on the focal point of the X-ray tube 111. The plurality of X-ray detection element arrays are arranged in the slice direction (column direction, row direction).

[0022] The X-ray detector 115 is, for example, an indirect detector having a grid, a scintillator array, and a photosensor array. The scintillator array has a plurality of scintillators. Each scintillator has a scintillator crystal. The scintillator crystal emits light in an amount corresponding to the intensity of the incident X-rays. The grid is disposed on the surface of the scintillator array where the X-rays are incident, and has an X-ray shielding plate having a function of absorbing scattered X-rays. Note that the grid may also be called a collimator (one-dimensional collimator or two-dimensional collimator). The photosensor array has, for example, photosensors such as photomultiplier tubes (PMTs). The photosensor array outputs an electrical signal corresponding to the amount of light emitted by the scintillator. The X-ray detector 115 may be a direct conversion type detector having a semiconductor element that converts the incident X-rays into an electrical signal.

[0023] The DAS 116 has, for example, an amplifier, an integrator, and an A / D converter. The amplifier performs an amplification process on the electrical signal output by each X-ray detection element of the X-ray detector 115. The integrator integrates the amplified electrical signal over a view period. The A / D converter converts the electrical signal indicating the integration result into a digital signal. The DAS 116 outputs detection data based on the digital signal to the console device 140.

[0024] The rotating frame 117 is an annular rotating member that rotates while holding the X-ray tube 111, the wedge 112, and the collimator 113 and the X-ray detector 115 facing each other. The rotating frame 117 is rotatably supported by a fixed frame around the subject P introduced therein. The rotating frame 117 further supports the DAS 116. The detection data output by the DAS 116 is transmitted from a transmitter having a light-emitting diode (LED) provided on the rotating frame 117 to a receiver having a photodiode provided on a non-rotating portion (for example, a fixed frame) of the gantry device 110 by optical communication, and is transferred to the console device 140 by the receiver. Note that the method of transmitting the detection data from the rotating frame 117 to the non-rotating portion is not limited to the method using the above-described optical communication, and any non-contact type transmission method may be adopted. The rotating frame 117 is not limited to an annular member as long as it can support and rotate the X-ray tube 111 and the like, and may be a member such as an arm.

[0025] The X-ray CT apparatus 100 is, for example, a Rotate / Rotate-Type X-ray CT apparatus (third-generation CT) in which both the X-ray tube 111 and the X-ray detector 115 are supported by the rotating frame 117 and rotate around the subject P. However, the present invention is not limited to this, and a Stationary / Rotate-Type X-ray CT apparatus (fourth-generation CT) in which a plurality of X-ray detection elements arranged in an annular shape are fixed to the fixed frame and the X-ray tube 111 rotates around the subject P may be used.

[0026] The control device 118 includes, for example, a processing circuit having a processor such as a CPU (Central Processing Unit), and a drive mechanism including a motor and an actuator. The control device 118 receives an input signal from the input interface 143 attached to the console device 140 or the gantry device 110, and controls the operations of the gantry device 110 and the bed device 130.

[0027] The control device 118 rotates the rotary frame 117, tilts the gantry device 110, or moves the top plate 133 of the bed device 130, for example. When tilting the gantry device 110, the control device 118 rotates the rotary frame 117 about an axis parallel to the Z-axis direction based on the tilt angle (pitch angle) input to the input interface 143. The control device 118 grasps the rotation angle of the rotary frame 117 based on the output of a sensor (not shown) or the like. Further, the control device 118 provides the rotation angle of the rotary frame 117 to the processing circuit 150 as needed. The control device 118 may be provided in the gantry device 110 or may be provided in the console device 140.

[0028] The control device 118 self-propels the gantry device 110 along the moving rail to perform this scan imaging or perform scan imaging, which is positioning imaging performed before the execution of this scan imaging.

[0029] The bed device 130 is a device that places the subject P to be scanned and introduces it into the rotary frame 117 of the gantry device 110. The bed device 130 includes, for example, a base 131, a bed driving device 132, a top plate 133, and a support frame 134. The base 131 includes a housing that supports the support frame 134 so as to be movable in the vertical direction (Y-axis direction). The bed driving device 132 includes a motor and an actuator. The bed driving device 132 moves the top plate 133 on which the subject P is placed along the support frame 134 in the longitudinal direction (Z-axis direction) of the top plate 133. The top plate 133 is a plate-shaped member on which the subject P is placed.

[0030] The console device 140 includes, for example, a memory 141, a display 142, an input interface 143, a communication interface 144, and a processing circuit 150. In the present embodiment, the console device 140 is described as being separate from the gantry device 110, but a part or all of the components of the console device 140 may be included in the gantry device 110.

[0031] The memory 141 is implemented by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, etc. Further, the memory 141 may include a storage medium such as a ROM (Read Only Memory) or a register. Further, the memory 141 may include a storage medium such as a ROM (Read Only Memory) or a register.

[0032] The memory 141 stores, for example, detection data, projection data, reconstructed images, etc. These data may be stored in an external memory (such as a NAS (Network Attached Storage), etc.) that the X-ray CT apparatus 100 can communicate with, instead of (or in addition to) the memory 141.

[0033] The display 142 displays various kinds of information. For example, the display 142 displays a CT image generated by the processing circuit 150, a GUI (Graphical User Interface) that receives various operations by an operator (such as a doctor, etc.), etc. The display 142 is, for example, an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube) display, an organic EL (Electro Luminescence) display, etc. The display 142 may be provided on the gantry device 110. The display 142 may be a desktop type, or a display device (such as a tablet terminal) that can communicate wirelessly with the main body of the console device 140.

[0034] The input interface 143 receives various input operations by an operator (such as a doctor, etc.) and outputs an electrical signal indicating the content of the received input operation to the processing circuit 150. For example, the input interface 143 receives input operations such as collection conditions when collecting detection data or projection data (described later), reconstruction conditions when reconstructing a CT image, and image processing conditions when generating a post-processing image from a CT image.

[0035] The input interface 143 is realized by, for example, a mouse, a keyboard, a touch panel, a trackball, a switch, a button, a joystick, a foot pedal, a camera, an infrared sensor, a microphone, etc. The input interface 143 may be provided in the gantry device 110. Further, the input interface 143 may be realized by a display device (for example, a tablet terminal) capable of wireless communication with the main body of the console device 140.

[0036] Note that in this specification, the input interface 143 is not limited to only those equipped with physical operation components such as a mouse and a keyboard. For example, an electric signal processing circuit that receives an electric signal corresponding to an input operation from an external input device provided separately from the device and outputs this electric signal to the control circuit is also included in the examples of the input interface 143.

[0037] The communication interface 144 includes, for example, a NIC (Network Interface Card) and a wireless communication module. The communication interface 144 communicates with other external devices including the diagnostic device 200 and the machine learning model evaluation device 300 via the communication network NW.

[0038] The processing circuit 150 controls the overall operation of the X-ray CT apparatus 100. The processing circuit 150 executes, for example, a system control function 151, a preprocessing function 152, a reconstruction processing function 153, an image processing function 154, an output control function 155, etc. The processing circuit 150 realizes these functions, for example, by a hardware processor executing a program stored in the memory 141.

[0039] A hardware processor refers to circuitry such as, for example, a CPU, a GPU (Graphics Processing Unit), an application specific integrated circuit (ASIC), a programmable logic device (e.g., a simple programmable logic device (SPLD) or a complex programmable logic device (CPLD), a field programmable gate array (FPGA)).

[0040] Instead of storing a program in the memory 141, it may be configured to directly incorporate the program into the circuitry of the hardware processor. In this case, the hardware processor realizes its functions by reading and executing the program incorporated into the circuitry. The above program may be stored in the memory 141 in advance, or may be stored in a non-transitory storage medium such as a DVD or a CD-ROM, and installed from the non-transitory storage medium into the memory 141 when the non-transitory storage medium is mounted on a drive device (not shown) of the console device 140. The hardware processor is not limited to being configured as a single circuit, and may be configured as one hardware processor by combining a plurality of independent circuits to realize each function. Also, a plurality of components may be integrated into one hardware processor to realize each function.

[0041] Each component included in the console device 140 or the processing circuit 150 may be realized by being decentralized and implemented by a plurality of hardware. The processing circuit 150 may be realized not by a configuration included in the console device 140 but by a separate processing device capable of communicating with the console device 140. The processing device may be, for example, a workstation connected to the X-ray CT device 100 or a cloud server capable of collectively executing processing equivalent to that of the processing circuit 150.

[0042] The system control function 151 controls various functions of the processing circuit 150 based on the input operations received by the input interface 143. The system control function 151 acquires imaging conditions (hereinafter, the first conditions) when imaging the subject P based on the input operations received by the input interface 143. The first conditions are the conditions for imaging to obtain image data. The first conditions are the conditions obtained by measuring the subject. The first conditions include, for example, each condition of ray quality, body weight, exposure (exposure dose), image quality, contrast agent amount, contrast timing, and reconstruction parameters. The system control function 151 transmits the acquired first conditions to the machine learning model evaluation device 300 via the communication interface 144. The first conditions are, for example, the conditions when the processing results by the medical device 200 described later are appropriate.

[0043] The preprocessing function 152 performs preprocessing such as logarithmic conversion processing, offset correction processing, sensitivity correction processing between channels, and beam hardening correction on the detection data output by the DAS 116 to generate projection data, and stores the generated projection data in the memory 141.

[0044] The reconstruction processing function 153 performs reconstruction processing on the projection data generated by the preprocessing function 152 by methods such as filtered back-projection method and successive approximation reconstruction method to generate a CT image which is one of the medical images, and stores the generated CT image in the memory 141.

[0045] The image processing function 154 converts the CT image into three-dimensional image or cross-sectional image data of an arbitrary cross-section by a known method based on the input operations received by the input interface 143. The conversion into a three-dimensional image may be performed by the preprocessing function 152.

[0046] The output control function 155 causes a display 142 to display data of a CT image generated through a reconstruction process by the reconstruction processing function 153 or an image (for example, a three-dimensional image, a cross-sectional image, a high-resolution image) generated by the image processing function 154 (hereinafter referred to as image data). Further, the output control function 155 transmits the image data as image data to a medical device 200 via a communication interface 144. The output control function 155 may transmit the image data to other external devices including a machine learning model evaluation device 300. The image data is an example of medical image data.

[0047] FIG. 3 is a diagram showing a configuration example of the medical device 200 in the first embodiment. The medical device 200 includes, for example, a communication interface 202, an input interface 204, a display 206, a memory 208, and a processing circuit 210.

[0048] The communication interface 202 includes, for example, a NIC, a wireless communication module, and the like. The communication interface 202 communicates with other external devices including the X-ray CT device 100 and the machine learning model evaluation device 300 via a communication network NW. The input interface 204 has the same configuration as any of those exemplified as the input interface 143 described in the X-ray CT device 100, receives various input operations by an operator, and outputs an electrical signal indicating the content of the received input operation to the processing circuit 210.

[0049] The display 206 displays various types of information. For example, the display 206 displays a processing result of the processing circuit 210, a GUI for receiving various operations by an operator, and the like. The display 206 is, for example, an LCD, an organic EL display, or the like. The memory 208 has the same configuration as any of those exemplified as the memory 141 described in the X-ray CT device 100. The memory 208 stores a machine learning model 10 supplied by the vendor. A plurality of types of machine learning models 10 may be stored in the memory 208.

[0050] The processing circuit 210 includes, for example, a data acquisition function 212, a data processing function 214, and a processing result output control function 216. The processing circuit 210 realizes these functions by, for example, a hardware processor (computer) executing a program stored in the memory 208. The hardware processor has the same configuration as any of those exemplified as the hardware processor described in the X-ray CT apparatus 100.

[0051] The data acquisition function 212 acquires the image data transmitted by the X-ray CT apparatus 100 via the communication interface 202. The data processing function 214 processes the image data acquired by the data acquisition function 212 using the machine learning model 10 stored in the memory 208.

[0052] The machine learning model 10 generates output data for an input of medical data. The medical data is, for example, the image data transmitted by the X-ray CT apparatus 100. The machine learning model 10 outputs diagnostic data as output data by inputting the image data as input data.

[0053] The data processing function 214 further generates diagnostic data (hereinafter referred to as reference diagnostic data) that is generated by inputting image data (hereinafter referred to as reference image data) based on a first condition that is the input data into the machine learning model 10. The reference diagnostic data is the correct answer data of the machine learning model associated with the reference image data when the reference image data is input into the machine learning model 10. The data processing function 214 transmits the generated reference image data to the machine learning model evaluation apparatus 300 via the communication interface 202. The reference image data is an example of reference medical data. The reference diagnostic data is an example of reference output data.

[0054] The processing result output control function 216 causes the medical data generated by the data processing function 214 to be displayed on the display 206 or output as sound using a speaker (not shown). The processing result output control function 216 may further transmit the processing result to an external device, such as a machine learning model evaluation device 300, via the communication interface 202.

[0055] When the analytical medical AI application is a Visualization-based application, the medical data is, for example, data of an image with an adjusted display mode. When the analytical medical AI application is a Workflow-based application, the medical data is, for example, data of the work procedure of a technician or the like. When the analytical medical AI application is a measurement-based application, the medical data is, for example, data of the measurement result. When the analytical medical AI application is a diagnostic-based application, the medical data is, for example, data indicating the diagnostic result.

[0056] FIG. 4 is a diagram showing a configuration example of the machine learning model evaluation device 300 in the first embodiment. The machine learning model evaluation device 300 includes, for example, a communication interface 302, an input interface 304, a display 306, a memory 308, and a processing circuit 310. In the embodiment, the machine learning model evaluation device 300 is provided independently of the medical device 200, but the machine learning model evaluation device 300 may be included in a personal computer or a smartphone included in the medical device 200. In this case, the machine learning model evaluation device 300 may be included in a personal computer or a smartphone in which an application that performs the function of the machine learning model evaluation device 300 is installed, and a communication interface or the like that can be shared by the medical device 200 and the machine learning model evaluation device 300 may be shared.

[0057] The communication interface 302 includes, for example, a NIC, a wireless communication module, etc. The communication interface 302 communicates with the X-ray CT apparatus 100 and other external apparatuses via the communication network NW. The input interface 304 has the same configuration as any of the input interfaces 143 described in the X-ray CT apparatus 100, accepts various input operations by the operator, and outputs an electrical signal indicating the content of the accepted input operation to the processing circuit 310.

[0058] The display 306 displays various kinds of information. For example, the display 306 displays the processing result of the processing circuit 310, a GUI for accepting various operations by the operator, etc. The memory 308 has the same configuration as any of those exemplified as the memory 141 described in the X-ray CT apparatus 100. The display 306 is an example of an output unit. The display 306 is an example of a display unit. The output unit may be other than the display 306 that displays an image. The output unit may be, for example, a speaker or a transmission interface.

[0059] The processing circuit 310 includes, for example, an acquisition function 312, a processing function 314, a calculation function 316, and an output control function 318. The processing circuit 310 realizes these functions, for example, by a hardware processor (computer) executing a program stored in the memory 308. The acquisition function 312 is an example of an acquisition unit. The processing function 314 is an example of a processing unit. The calculation function 316 is an example of a calculation unit. The output control function 318 is an example of an output control unit.

[0060] The acquisition function 312 includes, for example, a first condition acquisition function 322, an image data acquisition function 324, and a machine learning model acquisition function 326. The first condition acquisition function 322 acquires the first condition transmitted by the X-ray CT apparatus 100 via the communication interface 302. The image data acquisition function 324 acquires the image data transmitted by the X-ray CT apparatus 100 via the communication interface 302.

[0061] The machine learning model acquisition function 326 acquires the same machine learning model 10 as the machine learning model 10 stored in the memory 308 of the diagnostic device 200 among the machine learning models 10 supplied by the vendor. The machine learning model acquisition function 326 may acquire, via the communication interface 302, the same machine learning model 10 as the machine learning model 10 stored in the memory 208 of the diagnostic device 200 transmitted by the diagnostic device 200.

[0062] The processing function 314 includes, for example, a simulation processing function 332 and a target data generation function 334. The simulation processing function 332 generates a second condition obtained by adjusting a part of the first condition acquired by the first condition acquisition function 322. The simulation processing function 332 generates, for example, a second condition in which the weight is reduced by 1 kg among the first conditions and the other conditions are made to match the first condition. Similarly, a large number of second conditions in which slight changes are made to one or more parameters among the first conditions are generated. The simulation processing function 332 generates a large amount of simulation data (hereinafter, target image data) of an image captured under a second condition different from the first condition for the machine learning model 10. The simulation processing function 332 is an example of a simulation processing unit. The target image data is an example of target medical data.

[0063] The simulation processing function 332 may change the parameters under any conditions. The simulation processing function 332 may sequentially change a plurality of types of parameters. The simulation processing function 332 may determine, for example, parameters to be changed according to the attributes of the patient, the purpose of the examination, the type and state of the lesion.

[0064] The target data generation function 334 inputs the target image data generated by the simulation processing function 332 as input data of the machine learning model 10 acquired by the machine learning model acquisition function 326, respectively. The target data generation function 334 generates target diagnostic data for each of a large amount of target image data. The target diagnostic data is an example of target output data.

[0065] The calculation function 316 compares the reference medical data transmitted by the medical device 200 with the target medical data generated by the target data generation function 334. Based on the comparison result of the reference medical data and the target medical data, the calculation function 316 calculates the degree of fitness of the machine learning model 10 for the second condition. The degree of fitness indicates, for example, the degree to which correct data is output when a condition (for example, the second condition) is input as data into the machine learning model.

[0066] The degree of fitness is calculated according to the degree to which the medical data (output data) is correct data when the image data (medical data) captured under the imaging conditions is input into the machine learning model 10. For example, the degree of fitness for the machine learning model 10 of the first condition is "1". The degree of fitness for the machine learning model 10 of the second condition is calculated, for example, based on the degree of fitness for the machine learning model 10 of the first condition.

[0067] Since the reference medical data transmitted by the medical device 200 is correct data, it can be said that the machine learning model 10 is a model that correctly obtains the output data when the reference image data is used as the input data. Therefore, the degree of fitness for the machine learning model 10 of the first condition, which is the condition for generating the reference image data, is, for example, "1". On the other hand, the degree of fitness for the machine learning model 10 of the second condition, which is the condition for generating the target image data, is calculated as, for example, "1" or "0", and when the degree of fitness is "1", the degree of fitness is higher than when the degree of fitness is "0". The degree of fitness for the machine learning model 10 of the second condition may be calculated as a value greater than "0" and less than "1".

[0068] When the difference between the reference medical data and the target medical data when comparing the reference medical data and the target medical data is within the specified range, the calculation function 316 calculates the degree of fitness for the machine learning model 10 of the second condition as "1". When the difference between the compared reference medical data and the target medical data exceeds the specified range, the calculation function 316 calculates the degree of fitness for the machine learning model 10 of the second condition as 0.

[0069] Whether the difference between the reference medical data and the target medical data falls within the specified range depends on the type of the analytical medical AI application including the machine learning model 10. For example, when the analytical medical AI application is a Visualization-based application, if the generated comparison target image is within a range where it can be determined to be the same when visually recognized by a technician or the like, the calculation function 316 calculates the fitness for the machine learning model 10 of the second condition as "1", assuming that the difference between the reference medical data and the target medical data falls within the specified range.

[0070] When the analytical medical AI application is a Workflow-based application, if the difference in time taken for comparison is within a predetermined time, the calculation function 316 calculates the fitness for the machine learning model 10 of the second condition as "1", assuming that the difference between the reference medical data and the target medical data falls within the specified range. When the analytical medical AI application is a measurement-based application, if the difference in measurement results of the comparison target is within a predetermined numerical value, the calculation function 316 calculates the fitness for the machine learning model 10 of the second condition as "1", assuming that the difference between the reference medical data and the target medical data falls within the specified range. When the analytical medical AI application is a diagnosis-based application, if the diagnosis results of the comparison target match, the calculation function 316 calculates the fitness for the machine learning model 10 of the second condition as "1", assuming that the difference between the reference medical data and the target medical data falls within the specified range.

[0071] When the calculation function 316 sets the fitness for the machine learning model 10 of the second condition as "1", it determines the allowable range for the second condition. For example, when the calculation function 316 calculates the fitness for the machine learning model 10 of the second condition as "1", assuming that the difference between the reference medical data and the target medical data falls within the specified range, it defines the range between the maximum value and the minimum value of the second condition for which the target image data was generated as the allowable range.

[0072] The output control function 318 causes the second condition and the target medical data generated by the data processing function 214 to be displayed on the display 206 or output as sound using a speaker (not shown). The processing result output control function 216 further transmits the second condition and the target medical data to an external device, such as the machine learning model evaluation device 300, via the communication interface 202. The output control function 318 causes the allowable range of the second condition when the fitness of the machine learning model 10 calculated by the calculation function 316 is calculated as "1" to be displayed on the display 306. The output control function 318 may display, instead of or in addition to the display of the allowable range of the second condition, the fitness of the second condition with respect to the machine learning model 10 on the display 306.

[0073] Next, the processing in the machine learning model evaluation system 1 will be described. In the machine learning model evaluation system 1, a diagnosis using the medical device 200 is performed based on an image of the subject P imaged by the X-ray CT device 100. On the other hand, the suitability of the machine learning model used in the medical device 200 is evaluated using the machine learning model evaluation device 300 using the image imaged by the X-ray CT device 100.

[0074] FIG. 5 is a diagram showing an outline of an example of the processing flow of the machine learning model evaluation system 1 in the first embodiment. In the machine learning model evaluation system 1, first, imaging conditions (first condition) are set in the X-ray CT device 100, and after the imaging of the subject P by the X-ray CT device 100 is completed, an image subjected to image data generation processing is generated. The generated image is transmitted to the medical device 200, and the first condition is transmitted to the machine learning model evaluation device 300.

[0075] In the medical device 200, processing using the machine learning model 10 is performed on the image data transmitted by the X-ray CT device 100, and medical data is generated. The generated medical data is used for medical treatment by medical personnel such as doctors, and is also transmitted to the machine learning model evaluation device 300 as reference medical data. The image data transmitted from the X-ray CT device 100 to the medical device 200 is transmitted to the machine learning model evaluation device 300 as reference image data.

[0076] In the machine learning model evaluation apparatus 300, a simulation using the second condition generated based on the first condition transmitted by the X-ray CT apparatus 100 is executed, and target image data is generated. Processing using the machine learning model 10 is performed on this target image data, and target medical data is generated. The machine learning model evaluation apparatus 300 generates target medical data and compares it with the reference medical data transmitted by the medical apparatus 200. The machine learning model evaluation apparatus 300 sequentially generates target image data and target medical data while gradually changing the setting conditions of the second condition, calculates the degree of fitness of the second condition with respect to the machine learning model 10, and determines the allowable range of the second condition. Hereinafter, a more specific processing procedure in the machine learning model evaluation system 1 will be described.

[0077] FIG. 6 is a sequence diagram showing the processing flow of the machine learning model evaluation system 1 in the first embodiment. In the machine learning model evaluation system 1, when imaging the subject P, in the X-ray CT apparatus 100, first, the first condition for imaging the subject P is set (step S101). After the first condition is set, the X-ray CT apparatus 100 images the subject P (step S103).

[0078] Subsequently, the X-ray CT apparatus 100 performs image processing on the obtained detection data, projection data, etc. imaged in the processing circuit 150 to generate image data (step S105). The X-ray CT apparatus 100 transmits the generated image data to the medical apparatus 200 and transmits the first condition set in step S101 to the machine learning model evaluation apparatus 300 (step S107).

[0079] The medical apparatus 200 to which the image data is transmitted by the X-ray CT apparatus 100 receives and acquires the transmitted image data in the data acquisition function 212. Subsequently, the medical apparatus 200 processes the image data transmitted by the X-ray CT apparatus 100 as input data of the machine learning model 10 stored in the memory 208 in the data processing function 214 (step S201) and generates medical data (step S203).

[0080] Subsequently, the medical device 200 outputs the generated medical data by causing it to be displayed on the display 206 or the like in the processing result output control function 216 (step S205), and provides the medical data to medical staff such as a doctor who operates the medical device 200. The medical staff who operates the medical device 200 diagnoses the subject P, for example, while visually recognizing the medical data output to the medical device 200.

[0081] The medical device 200 uses the image data transmitted by the X-ray CT device 100 and the medical data generated in step S205 as reference image data and reference medical data, respectively, in the data processing function 214. The data processing function 214 transmits the reference image data and the reference medical data to the machine learning model evaluation device 300 (step S207).

[0082] On the other hand, the machine learning model evaluation device 300 receives and acquires the first condition transmitted by the X-ray CT device 100 in the first condition acquisition function 322. Subsequently, the machine learning model evaluation device 300 generates a second condition based on the acquired first condition in the simulation processing function 332 of the processing function 314 (step S301).

[0083] Subsequently, the machine learning model evaluation device 300 generates target image data based on the second condition in the simulation processing function 332 of the processing function 314 (step S303). In the processing function 314, a large number of second conditions are generated, and a large number of target image data based on the large number of second conditions are generated.

[0084] Subsequently, the simulation processing function 332 processes the target image data as input data of the machine learning model 10 acquired by the machine learning model acquisition function 326 (step S305), and generates target medical data for each of the second conditions (step S307).

[0085] Subsequently, in the calculation function 316, the machine learning model evaluation apparatus 300 calculates the degree of conformity of the second condition for which the target image data is generated by comparing the reference medical data with the target medical data (step S311). The calculation function 316 determines whether the difference between the reference medical data and the target medical data is within a specified range as a result of comparing the reference medical data with the target medical data (step S313).

[0086] When it is determined that the difference between the reference medical data and the target medical data is within the specified range, the calculation function 316 calculates the degree of conformity of the second condition with respect to the machine learning model 10 as "1", and includes the second condition at the time of generating the target image data in the comparison target data set within the allowable range (step S315). When it is determined that the difference between the reference medical data and the target medical data is not within the specified range, the calculation function 316 calculates the degree of conformity of the second condition with respect to the machine learning model 10 as "0", skips step S315, and advances the process to step S317.

[0087] Subsequently, the calculation function 316 determines whether the calculation of the degree of conformity has been completed (step S317). The calculation function 316 determines that the calculation of the degree of conformity has been completed, for example, when the upper limit value and the lower limit value of the allowable range of the second condition are set, or when the degree of conformity of all the second conditions is calculated.

[0088] When it is determined that the setting of the allowable range of the second condition has not been completed, the calculation function 316 returns the process to step S311. When the calculation function 316 determines that the setting of the allowable range of the second condition has been completed, the output control function 318 outputs predetermined information, such as by causing the display 306 to display the degree of conformity of the second condition with respect to the machine learning model 10 calculated by the calculation function 316 and the allowable range of the second condition (step S319). In this way, the process in the machine learning model evaluation system 1 is terminated.

[0089] FIG. 7 is a diagram showing an example of an image that maps the allowable range of imaging conditions displayed on the display 306 in the first embodiment. The imaging conditions include "image quality", "exposure", "image quality", "contrast agent amount", "contrast timing", "reconstruction parameters", and "body weight". The output control function 318 in the machine learning model evaluation apparatus 300 causes the display 306 to display an allowable range map image GA10 indicating the allowable range for each of these imaging conditions.

[0090] By causing the output control function 318 to display the allowable range for each imaging condition as the allowable range map image GA10 on the display 306, an operator such as a medical practitioner can easily grasp the allowable range for each condition. In this example, the output control function 318 shows the allowable range map image GA10 for seven imaging conditions, but it may show an allowable range map image for a number of imaging conditions less than or greater than seven. The allowable range may be shown in a form other than the form of the allowable range map image GA10 shown in FIG. 7.

[0091] The machine learning model evaluation system 1 of the first embodiment calculates the degree of conformity to the machine learning model 10 of the second condition by comparing the reference medical data for the reference image data with the target medical data for the target image data. Since the reference image data is correct data, the degree of conformity to the machine learning model 10 of the first condition is "1". Therefore, since the degree of conformity to the machine learning model 10 of the second condition can be accurately evaluated, a plurality of types of analytical medical AI applications supplied by the vendor can be fully utilized.

[0092] When the machine learning model evaluation system 1 of the first embodiment calculates the degree of conformity to the machine learning model 10 of the second condition as "1", it determines the allowable range for the second condition and causes the display 306 to display the allowable range. Therefore, it is possible to notify the operator of the range of imaging conditions in which the analytical medical AI application including the machine learning model 10 can be accurately utilized.

[0093] In the above-described first embodiment, the output control function 318 indicates the allowable range of imaging conditions (second condition) based on the degree of fitness for the machine learning model 10, but it may also display a tolerance margin for the allowable range of imaging conditions. FIG. 8 is a diagram showing an example of an image indicating the tolerance margin of the allowable range of imaging conditions displayed on the display 306 in the first embodiment.

[0094] In this example, the output control function 318 causes the display 306 to display a reference condition map image GA20 indicating the first condition when generating the reference image data, together with the allowable range map image GA10. By displaying the allowable range map image GA10 and the reference condition map image GA20, for example, with respect to the imaging conditions of "exposure", the range indicated by the arrow image YA10 between the allowable range map image GA10 and the reference condition map image GA20 can be recognized by the operator as the tolerance margin.

[0095] Furthermore, the allowable range map image GA10 and the reference condition map image GA20 may be displayed on the GUI, and the imaging conditions may be specified inside the allowable range map image GA10 or the like by operating the input interface 304. The allowable range map image GA10 and the reference condition map image GA20 may be displayed on a display other than the display 306 of the machine learning model evaluation apparatus 300, for example, the display 142 provided in the console apparatus of the X-ray CT apparatus 100 or the display 206 provided in the diagnostic apparatus 200.

[0096] In the above-described first embodiment, the output control function 318 indicates the allowable range of imaging conditions based on the degree of fitness of the second condition for the machine learning model 10, but it may also display the adjustable range of imaging conditions. FIG. 9 is a diagram showing an example of an image indicating the adjustable range of imaging conditions displayed on the display 306 in the first embodiment.

[0097] In this example, the output control function 318 causes the display 306 to display the allowable exposure range as an imaging condition using the first arrow GA21 to the fifth arrow GA25. The output control function 318 shows the second arrow GA22 as the current set value, and emphasizes and displays that the second arrow GA22 indicates the current set value by displaying the second arrow GA22 as a set arrow thicker than the other arrows from the first arrow GA21 to the fifth arrow GA25. The output control function 318 shows the out-of-allowable-range arrow GA40 as a dashed line above the first arrow GA21 and below the fifth arrow GA25.

[0098] The output control function 318 displays a set value icon GA30 on the right side of the first arrow GA21 to the fifth arrow GA25. The set value icon GA30 is a GUI. When the set value icon GA30 is operated in the up and down movement directions, the set arrow indicating the set value moves up and down. The set value icon GA30 can move between the right sides of the first arrow GA21 to the fifth arrow GA25 and cannot move beyond the side of the out-of-allowable-range arrow GA40.

[0099] The output control function 318 displays a first message image MA11 in the central upper part on the right side of the first arrow GA21 to the fifth arrow GA25. The first message image MA11 includes the characters "Allowable range" and parentheses surrounding the first arrow GA21 to the fifth arrow GA25, and notifies the operator that the range indicated by the first arrow GA21 to the fifth arrow GA25 is the allowable exposure range.

[0100] The output control function 318 displays a second message image MA12 in the upper left part on the left side of the first arrow GA21 to the fifth arrow GA25. The second message image MA12 notifies the operator that the thick arrow is the current set value. The output control function 318 displays a third message image MA13 in the lower left part on the left side of the first arrow GA21 to the fifth arrow GA25. The third message image MA13 includes the characters "It can be lowered further" and notifies the operator that the exposure amount can be lowered.

[0101] In this way, the machine learning model evaluation apparatus 300 can easily make the operator recognize the imaging conditions, specifically, the range that can be set for exposure in this case, by causing the output control function 318 to display the first arrow GA21 to the fifth arrow GA25 and the first message image MA11 to the third message image MA13 on the display 306. Further, since the set value icon GA30 can move within the range corresponding to the first arrow GA21 to the fifth arrow GA25, the operator can easily set the imaging conditions, specifically, exposure in this case.

[0102] Also, by determining the allowable range of the imaging conditions, for example, when continuously capturing a plurality of images, it may be examined how much the imaging conditions can change (or blur). In this case, for example, at the initial stage of continuously capturing images, although a certain burden is imposed on the subject P to some extent, images are captured under imaging conditions that are easy to capture, and after determining the allowable range of the imaging conditions, images may be captured under imaging conditions where the burden on the subject P within the capturable range is reduced. Further, when continuously capturing images, the imaging conditions may be changed during the continuous imaging, or when capturing images at regular intervals, the imaging conditions may be changed after several imaging operations are completed.

[0103] (Second Embodiment) Next, the second embodiment will be described. The machine learning model evaluation system 1 of the second embodiment is mainly different in part of the configuration of the machine learning model evaluation apparatus 300 as compared with the first embodiment. Hereinafter, the machine learning model evaluation system 1 of the second embodiment will be described centering on the differences from the first embodiment. FIG. 10 is a diagram showing a configuration example of the machine learning model evaluation apparatus 300 in the second embodiment.

[0104] The memory 308 of the machine learning model evaluation apparatus 300 according to the second embodiment stores image data 350 with imaging condition data. The image data 350 with imaging condition data includes the image data and the imaging conditions when the image data was captured. The image data 350 with imaging condition data is, for example, data obtained by accumulating the image data obtained when measuring a subject in past medical practices at a medical facility where the X-ray CT apparatus 100 is installed and the imaging conditions when the image data was captured.

[0105] The machine learning model evaluation apparatus 300 according to the second embodiment is installed in a facility that can collect a plurality of image data 350 with imaging condition data from, for example, a large-scale medical facility or a plurality of medical facilities. The machine learning model evaluation apparatus 300 according to the second embodiment stores, for example, a large amount of image data 350 with imaging condition data in the memory 308.

[0106] The processing function 314 in the machine learning model evaluation apparatus 300 according to the second embodiment extracts data including imaging conditions approximated to the first conditions from the image data 350 with imaging condition data transmitted by the X-ray CT apparatus 100 as accompanying data. The calculation function 316 generates the image data included in the image data 350 with imaging condition data extracted by the processing function 314 as reference image data and the medical data as reference medical data. Other configurations are common to the first embodiment described above. The medical data of the first embodiment is data for a specific subject P, whereas the image data 350 with imaging condition data is data for a plurality of subjects. For this reason, the generated allowable range is not a value for a specific subject but a value generally used as a representative value.

[0107] The machine learning model evaluation system 1 according to the second embodiment has the same operational effects as the machine learning model evaluation system 1 of the first embodiment described above. Furthermore, since the machine learning model evaluation system 1 according to the second embodiment does not need to include a simulation processing function or the like, it can have a simpler configuration accordingly.

[0108] (Third Embodiment) Next, a third embodiment will be described. The machine learning model evaluation system 1 of the third embodiment is mainly different in part of the configuration of the machine learning model evaluation device 300 as compared with the first embodiment. FIG. 11 is a diagram showing an outline of an example of the processing flow of the machine learning model evaluation system 1 in the third embodiment.

[0109] In the machine learning model evaluation system 1 of the first embodiment, based on the result of comparing the medical data (reference medical data) output by the machine learning model 10 of the medical device 200 with the medical data (target medical data) output by the machine learning model 10 of the machine learning model evaluation device 300, the fitness of the machine learning model 10 for the second condition is calculated.

[0110] On the other hand, in the machine learning model evaluation system 1 of the third embodiment, based on the result of comparing the image data (reference image data) input to the machine learning model 10 of the medical device 200 with the image data (target image data) input to the machine learning model 10 of the machine learning model evaluation device 300, the fitness of the machine learning model 10 for the second condition is calculated. In the machine learning model evaluation device 300 of the third embodiment, in order to calculate the fitness of the machine learning model 10 for the second condition, the distribution of the input data of the learning data used for the learning of the machine learning model 10 is utilized.

[0111] FIG. 12 is a diagram showing a configuration example of the machine learning model evaluation device 300 in the third embodiment. The machine learning model evaluation device 300 in the third embodiment is mainly different from the machine learning model evaluation device 300 of the first embodiment in that it includes an estimation function 319 and the acquisition function 312 includes a first acquisition function 327 and a second acquisition function 328 instead of the first condition acquisition function 322 and the image data acquisition function 324. Also, the content simulated by the processing function 314 is different. The machine learning model acquisition function 326 in the acquisition function 312 is an example of a machine learning model acquisition unit.

[0112] The estimation function 319 estimates the distribution of the training data used for training the machine learning model 10, for example, the data distribution of the input data (image data) included in the training data (hereinafter, the first data distribution). The estimation of the data distribution includes encoding the machine learning model 10 and each data used for its training. The first acquisition function 327 of the acquisition function 312 acquires the first data distribution estimated by the estimation function 319. The first acquisition function 327 is an example of a first acquisition unit.

[0113] The simulation processing function 332 in the processing function 314 acquires, for example, the imaging conditions when imaging the image data in the cohort and the image data. The cohort can be any cohort, for example, the cohort of a medical facility where the X-ray CT apparatus 100 is installed, or the cohort of a plurality of medical facilities connected via the communication network NW.

[0114] The simulation processing function 332 generates the acquired imaging conditions as the second conditions as they are. The simulation processing function 332 generates the image data in the acquired cohort as the target image data to be input to the machine learning model 10 as it is. The simulation processing function 332 generates the data distribution of the generated target image data (hereinafter, the second data distribution). The second acquisition function 328 of the acquisition function 312 acquires the second data distribution generated by the simulation processing function 332. The second acquisition function 328 is an example of a second acquisition unit.

[0115] The second data distribution is a data distribution regarding the distribution of the image data included in the data set in the cohort, and is, for example, a partial or entire data distribution of the image data in the subject of the medical facility (hereinafter, the installation medical facility) where the X-ray CT apparatus 100 is installed. The second data distribution may be the data distribution of the image data other than the image data acquired by the simulation processing function 332. For example, it may be the image distribution of the image data of the patient (subject) of the installation medical facility stored in the memory 141 of the X-ray CT apparatus 100.

[0116] The calculation function 316 calculates the fitness of the second data distribution with respect to the machine learning model 10 based on the comparison result between the first data distribution and the second data distribution. Since the first data distribution is a data distribution with a high fitness for the machine learning model 10, the closer the second data distribution is to the first data distribution, the higher the tendency of its fitness for the machine learning model 10. The calculation function 316 defines the allowable range of imaging conditions based on the fitness of the second data distribution.

[0117] The second data distribution may be, for example, a data distribution estimated by the first data distribution. For example, when the data distribution is a distribution of imaging conditions related to the image quality of image data, in this case, for example, the allowable range of imaging conditions for imaging the image data constituting the second data distribution is an appropriate range as the allowable range of imaging conditions for imaging the image data when using the machine learning model 10. Therefore, by outputting the calculated allowable range, it is possible to assist a doctor or the like in setting appropriate imaging conditions.

[0118] When the data distribution of the image (hereinafter referred to as the target data distribution) when imaging the subject deviates from the first data distribution, the calculation function 316 may perform image processing to correct the image for which the target data distribution is obtained so that the target data distribution approximates or coincides with the first data distribution. FIG. 13 is a diagram showing an example of the relationship between the first data distribution and the target data distribution in the third embodiment.

[0119] The horizontal axis of the histogram shown in FIG. 13 is, for example, the image quality of the image data. The image quality of the image data is in an equivalent relationship with the dose of X-rays irradiated to the subject P when imaging the image, and the higher the dose of X-rays, the higher the image quality. When the target data distribution DB21 is shifted to the side where the image quality of the image data is lower with respect to the first data distribution DB11, by generating a corrected data distribution DB22 obtained by performing image processing to improve the image quality on each image data, the target data distribution DB21 can be made to overlap the first data distribution DB11.

[0120] The target data distribution may be adjusted by performing image processing on the image data included in the target data distribution DB21 and / or alternatively by adjusting the imaging conditions. Hereinafter, specific embodiments of the adjustment of the target data distribution will be described. FIGS. 14 to 17 are diagrams showing an example of the relationship between the first data distribution and the target data distribution in the third embodiment.

[0121] As shown in FIG. 14, in the case where the first data distribution DB11 is shifted to the side where the image quality (= dose) of the image data included in the target data distribution DB21 is lower. In this case, although the dose of X-rays increases when imaging the subject P to obtain the image data, the image quality can be improved.

[0122] As shown in FIG. 15, in the case where the first data distribution DB11 is shifted to the side where the image quality (= dose) of the image data included in the target data distribution DB21 is higher. In this case, by reducing the dose of X-rays when imaging the subject P to obtain the image data, an image with satisfactory image quality can be obtained while reducing the exposure dose of the subject P.

[0123] As shown in FIG. 16, in the case where the target data distribution DB21 almost overlaps with the first data distribution DB11. In this case, the image data can be obtained without changing the imaging conditions such as the dose of X-rays (= dose) when imaging the subject P to obtain the image data. As shown in FIG. 17, in the case where the target data distribution DB21 is included in the range on the higher-dose side with respect to the first data distribution DB11. In this case, it may not be necessary to change the imaging conditions, or the dose of X-rays when imaging the subject P to obtain the image data may be reduced.

[0124] In each of the above embodiments, the first data distribution and the second data distribution are each a data distribution of image data, but may also be data distributions of other data. For example, the first data distribution may be the data distribution of a data set of reference image data and reference medical data, and the second data set may be the data distribution of a data set of target image data input to the simulation processing function 332 and target medical data corresponding to this target image data.

[0125] (Fourth Embodiment) Next, the fourth embodiment will be described. The machine learning model evaluation system of the fourth embodiment is mainly different in the processing related to the calculation and output of the allowable range of imaging conditions (second condition) in the calculation function 316 and the output control function 318 as compared with the machine learning model evaluation system 1 of the first embodiment.

[0126] In the machine learning model evaluation system of the fourth embodiment, the calculation function 316 determines the allowable range of the second condition according to the subject using the medical data. Specifically, the calculation function 316 determines the allowable range of the second condition for the case where the subject is a user such as a doctor (human) (hereinafter, doctors, etc.), the case where it is a computer such as an AI, and the case where it is both a doctor, etc. or a computer.

[0127] The output control function 318 causes the display 306 to display the allowable range determined for each subject calculated by the calculation function 316. Hereinafter, a display example of the display 306 will be described. FIG. 18 is a diagram showing an example of an image representing the allowable range of imaging conditions displayed on the display 306 in the fourth embodiment.

[0128] The output control function 318 causes the display 306 to display a first allowable range image GA50, a second allowable range image GA52, and a third allowable range image GA54 as images indicating the allowable range of the dose. The first allowable range image GA50 is an image indicating the allowable range when the user is a doctor or the like. The second allowable range image GA52 is an image indicating the allowable range when the user is an analytical diagnosis AI application. The third allowable range image GA54 is an image indicating the allowable range when the user is both a doctor or the like and an analytical diagnosis AI application.

[0129] For example, when the user (doctor or the like or analytical diagnosis AI application) performs a diagnosis using the image data, when a doctor or the like performs a diagnosis as the user, for example, the diagnosis is performed while looking at an image based on the image data. On the other hand, when the analytical diagnosis AI application performs a diagnosis as the user, the diagnosis is performed based on numerical values or the like included in the image data without looking at the image. Thus, since the area of using the image data differs depending on the user, the calculation function 316 determines the allowable range of the second condition depending on the user.

[0130] The machine learning model evaluation system 1 of the fourth embodiment has the same operational effects as the machine learning model evaluation system 1 of the first embodiment described above. Further, in the machine learning model evaluation apparatus 300 of the fourth embodiment, when there are a plurality of users, for example, when the users are both a doctor or the like and an analytical diagnosis AI application, the calculation function 316 determines the range where the allowable ranges of the second condition for those plurality of users overlap as the allowable range. Therefore, it is possible to determine an allowable range suitable for each of the plurality of users.

[0131] (Fifth Embodiment) Next, the fifth embodiment will be described. In the machine learning model evaluation system 1 of the fifth embodiment, in the machine learning model evaluation apparatus 300, the allowable ranges of imaging conditions in the machine learning models 10 included in each of a plurality of, for example, two analytical diagnostic AI applications are superimposed and displayed on the display 306. FIG. 19 is a diagram showing an example of an image in which the allowable range of imaging conditions displayed on the display 306 in the fifth embodiment is shown as a map.

[0132] On the display 306, a first allowable range map image GA11 and a second allowable range map image GA12 are displayed. The first allowable range map image GA11 is an image showing the allowable range of imaging conditions for the machine learning model 10 included in the first analytical diagnostic AI application. The second allowable range map image GA12 is an image showing the allowable range of imaging conditions for the machine learning model 10 included in the second analytical diagnostic AI application.

[0133] The first analytical diagnostic AI application is an application with a wide allowable range of dose and a narrow allowable range of contrast agent amount. The second analytical diagnostic AI application is an application with a narrow allowable range of dose and a wide allowable range of contrast agent amount. In the machine learning model evaluation system 1 of the fifth embodiment, for two analytical diagnostic AI applications, the allowable ranges of imaging conditions can be displayed on the display 306 for each analytical diagnostic application.

[0134] The machine learning model evaluation system 1 of the fifth embodiment has the same operational effects as the machine learning model evaluation system 1 of the first embodiment described above. Further, the machine learning model evaluation system 1 of the fifth embodiment displays the allowable ranges of imaging conditions for a plurality of analytical diagnostic AI applications. For this reason, an analytical diagnostic AI application corresponding to the attributes and state of the subject P can be used. The machine learning model evaluation system 1 of the fifth embodiment superimposes and displays the allowable ranges of imaging conditions for a plurality of analytical diagnostic AI applications. For this reason, the imaging conditions of a plurality of analytical diagnostic AI applications can be easily compared.

[0135] For example, when the subject P is a young patient, in order not to increase the dose even if the amount of the contrast agent increases, the first analytical and diagnostic AI application can be used. When the subject P is a patient suffering from kidney disease, the second analytical and diagnostic AI application can be used to suppress the amount of the contrast agent.

[0136] In each of the above embodiments, the machine learning model evaluation device 300 calculates the degree of fitness for the machine learning model 10 under the second condition, but the degree of fitness for the machine learning model 10 other than the second condition may also be calculated. For example, the machine learning model evaluation device 300 may calculate the degree of fitness of the input data input to the machine learning model 10, or may calculate the degree of fitness of a data set that is a set of the input data input to the machine learning model 10 and the output data output from the machine learning model 10.

[0137] In each of the above embodiments, the first condition is used as the imaging condition, but the first condition may be other conditions. For example, the first condition may be an image processing condition when image data is processed. Examples of the image processing condition include conditions such as noise reduction, resolution change, image size, brightness, contrast, WW (Window Level), WL (Window width), edge cooperation, object recognition, object enhancement, motion recognition, motion cooperation, Segmentation, Registration, biomarker image, etc. Noise reduction is performed, for example, by performing smoothing or using a filter. Examples of noise include additive noise, blur, smoothness, resampling, and those by a Band Pass Filter. Resolution includes, for example, spatial resolution, density resolution, and time resolution.

[0138] In addition, examples of the imaging condition include, in addition to those shown in each embodiment, image quality SN (S / N ratio), frame rate, field of view (FOV), aperture, halation, resolution at the time of imaging, imaging direction, SN (noise, contrast), shading, heart rate, etc. Examples of the dose include tube voltage, tube current, pulse width, filter, and beam quality.

[0139] In each of the above embodiments, the image data generated by the X-ray CT apparatus 100 is used as medical data. However, the image data used as medical data may be other image data. For example, when the image data generated by the X-ray CT apparatus 100 is displayed on the display 206, image processing such as making obstacles such as bones transparent or displaying organs in three dimensions may be performed on the image data to make the affected part easier to see. The image data used as medical data may be the image data after image processing is performed to make the affected part easier to see in this way.

[0140] According to at least one of the embodiments described above, an acquisition unit that acquires a machine learning model for generating output data for an input of medical data, a processing unit that generates target output data by inputting target medical data generated based on a second condition different from the first condition under which quasi-medical data is generated into the machine learning model, and a calculation unit that calculates the degree of fitness of the second condition with respect to the machine learning model based on a comparison result between reference output data generated by inputting the reference medical data into the machine learning model and the target output data. By having these components, it is possible to make full use of the analytical diagnosis AI application.

[0141] Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are also included in the invention described in the claims and its equivalent scope.

Description of Reference Numerals

[0142] 1 Machine Learning Model Evaluation System 10 Machine Learning Model 100 X-ray CT Apparatus (Medical Imaging Apparatus) 110 Gantry Device 111 X-ray Tube 112 Wedge 113 Collimator 114 X-ray high voltage device 115 X-ray detector 116 DAS 117 Rotating frame 118 Control device 130 Bed device 131 Base 132 Bed drive device 133 Top plate 134 Support frame 140 Console device 141, 208, 308 Memory 142, 206, 306 Display 143, 204, 304 Input interface 144, 202, 302 Communication interface 150, 210, 310 Processing circuit 151 System control function 152 Pretreatment function 153 Reconfiguration processing function 154 Image processing function 155, 318 Output control function 200 Diagnostic device 202 Communication interface 212 Data acquisition function 214 Data processing function 216 Processing result output control function 300 Machine learning model evaluation device 312 Acquisition function 314 Processing function 316 Calculation function 318 Output control function 319 Estimation function 322 First condition acquisition function 324 Image data acquisition function 326 Machine learning model acquisition function 327 First acquisition function 328 Second acquisition function 332 Simulation processing function 334 Target data generation function Image data with 350 imaging condition data DB11 First data distribution DB21 Target data distribution DB22 Correction data distribution GA10 Tolerance range map image GA11 First tolerance range map image GA12 Second tolerance range map image GA20 Reference condition map image NW Communication network P Subject

Claims

1. An acquisition unit that acquires a machine learning model for generating output data in response to input of medical data; A processing unit that generates target output data by inputting target medical data generated based on a second condition different from a first condition under which reference medical data is generated into the machine learning model; A calculation unit that calculates a degree of fitness of the second condition with respect to the machine learning model based on a comparison result between reference output data generated by inputting the reference medical data into the machine learning model and the target output data. A machine learning model evaluation apparatus comprising: A machine learning model evaluation apparatus.

2. The reference output data is correct answer data of the machine learning model associated with the reference medical data. The machine learning model evaluation apparatus according to Claim 1. The machine learning model evaluation apparatus according to Claim 1.

3. The medical data is medical image data. The machine learning model evaluation apparatus according to Claim 1 or 2. The machine learning model evaluation apparatus according to Claim 1 or 2.

4. The first condition is at least one of imaging conditions when imaging for obtaining the medical image data or image processing conditions when performing image processing on the medical image data. The machine learning model evaluation apparatus according to Claim 3. The machine learning model evaluation apparatus according to Claim 3.

5. The first condition is a condition obtained by measuring a subject. The machine learning model evaluation apparatus according to any one of Claims 1 to 4. The machine learning model evaluation apparatus according to any one of Claims 1 to 4.

6. The processing unit further includes a simulation processing unit that determines the second condition based on the first condition and generates simulation data based on the second condition as the target medical data. The machine learning model evaluation apparatus according to any one of Claims 1 to 5. Further comprising: The machine learning model evaluation apparatus according to any one of Claims 1 to 5.

7. The target medical data includes data obtained by accumulating data obtained by measuring a subject in the past. The machine learning model evaluation apparatus according to any one of Claims 1 to 5. The machine learning model evaluation apparatus according to any one of Claims 1 to 5.

8. The calculation unit determines an allowable range of the second condition based on the degree of fitness. The machine learning model evaluation apparatus according to any one of Claims 1 to 7. The machine learning model evaluation apparatus according to any one of Claims 1 to 7.

9. The calculation unit determines an allowable range of the second condition according to a user who uses the output data. The machine learning model evaluation apparatus according to Claim 8. The machine learning model evaluation apparatus according to Claim 8.

10. The apparatus further includes an output control unit that causes a display unit to display information indicating the degree of fitness. The machine learning model evaluation apparatus according to any one of Claims 1 to 9. The machine learning model evaluation apparatus according to any one of Claims 1 to 9.

11. The calculation unit determines an allowable range of the second condition for a plurality of the machine learning models. The output control unit superimposes a plurality of the second conditions and causes them to be displayed on the display unit. The machine learning model evaluation device according to claim 10.

12. When the difference between the reference output data and the target output data falls within a specified range, the calculation unit determines that the second condition is within the allowable range. The machine learning model evaluation device according to any one of claims 8 to 11.

13. The specified range varies depending on the type of the machine learning model. The machine learning model evaluation device according to claim 12.

14. A machine learning model acquisition unit that acquires a machine learning model that generates output data for input of medical data, A first acquisition unit that acquires a first data distribution regarding the distribution of reference medical data input to the machine learning model, A second acquisition unit that acquires a second data distribution regarding the distribution of target medical data input to the machine learning model in a cohort, A calculation unit that calculates the degree of fitness of the distribution of the target medical data with respect to the machine learning model based on a comparison result between the first data distribution and the second data distribution. Machine learning model evaluation device.

15. A computer of a machine learning model evaluation device, acquires a machine learning model that generates output data for input of medical data, generates target output data by inputting target medical data generated based on a second condition different from the first condition under which the reference medical data is generated into the machine learning model, calculates the degree of fitness of the second condition with respect to the machine learning model based on a comparison result between the reference output data generated by inputting the reference medical data into the machine learning model and the target output data. Machine learning model evaluation method.

16. causes a computer of a machine learning model evaluation device to acquire a machine learning model that generates output data for input of medical data, generate target output data by inputting target medical data generated based on a second condition different from the first condition under which the reference medical data is generated into the machine learning model, calculate the degree of fitness of the second condition with respect to the machine learning model based on a comparison result between the reference output data generated by inputting the reference medical data into the machine learning model and the target output data. Program.

Images