Image processing device

Abstract

To reduce the computational cost during neural network inference. [Solution] The image processing apparatus according to the embodiment comprises an acquisition unit, a determination unit, and an execution unit. When performing inference on a trained model, the acquisition unit acquires features from the trained model for a region of interest set based on the input data. The determination unit determines whether or not to continue the inference for the region of interest based on the features acquired by the acquisition unit. If the determination unit determines not to continue the inference, the execution unit cancels the inference, and if the determination unit determines to continue the inference, the execution unit continues the inference.

Description

Technical Field

[0001] Embodiments disclosed in the present specification and drawings relate to an image processing apparatus.

Background Art

[0002] When processing volume data using deep learning, due to issues such as memory constraints, instead of processing the full range of volume data, the volume data may be divided into fixed-size VOI (Volume of Interest), and processing may be performed on each of the divided VOIs.

[0003] However, when the VOI size is large, the processing may take time.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0005] One of the problems to be solved by the embodiments disclosed in the present specification and drawings is to reduce the computational cost during inference of a neural network. However, the problems to be solved by the embodiments disclosed in the present specification and drawings are not limited to the above-mentioned problems. Problems corresponding to the respective effects of the respective configurations shown in the embodiments described later can also be positioned as other problems.

Means for Solving the Problem

[0006] The image processing apparatus according to the embodiment comprises an acquisition unit, a determination unit, and an execution unit. When performing inference on a trained model, the acquisition unit acquires features from the trained model for each region of interest set based on the input data. The determination unit determines whether or not to continue the inference for each region of interest based on the features acquired by the acquisition unit. If the determination unit determines not to continue the inference, the execution unit cancels the inference, and if the determination unit determines to continue the inference, the execution unit continues the inference. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 shows an example of the configuration of an image processing apparatus according to an embodiment. [Figure 2] Figure 2 shows an example of the configuration of an X-ray CT apparatus according to this embodiment. [Figure 3] Figure 3 shows an example of a trained model according to the first embodiment. [Figure 4] Figure 4 shows an example of a convolution process according to the first embodiment. [Figure 5] Figure 5 is a flowchart showing the processing flow performed by the image processing device according to the first embodiment. [Figure 6] Figure 6 shows an example of processing performed by the image processing apparatus according to the first embodiment. [Figure 7] Figure 7 shows an example of processing performed by the image processing apparatus according to the first embodiment. [Figure 8] Figure 8 shows an example of a screen displayed by the image processing device according to the first embodiment. [Figure 9] Figure 9 shows an example of a screen displayed by the image processing device according to the first embodiment. [Figure 10] Figure 10 shows an example of a screen displayed by an image processing device according to a modified example of the first embodiment. [Figure 11] Figure 11 shows an example of a screen displayed by an image processing device according to a modified example of the first embodiment. [Figure 12] Figure 12 shows an example of a screen displayed by an image processing device according to a modified example of the first embodiment. [Figure 13] Figure 13 shows an example of a screen displayed by an image processing device according to a modified example of the first embodiment. [Figure 14] Figure 14 is a diagram showing an example of processing performed by the image processing device according to the second embodiment. [Modes for carrying out the invention]

[0008] The following describes in detail an embodiment of the image processing apparatus with reference to the drawings.

[0009] (First embodiment) First, an example of the configuration of an image processing apparatus according to the embodiment will be described using Figures 1 and 2. Figure 1 is a diagram showing the image processing apparatus 100 according to the embodiment. Figure 2 is a diagram showing an example of a medical imaging diagnostic apparatus incorporating the image processing apparatus 100 according to the embodiment. In Figure 2, the case in which the medical imaging diagnostic apparatus incorporating the image processing apparatus 100 is an X-ray CT apparatus 200 is described. However, the embodiment is not limited to the case where the medical imaging diagnostic apparatus is an X-ray CT apparatus, and the medical imaging diagnostic apparatus may be other medical imaging diagnostic apparatuses such as ultrasound diagnostic apparatuses, magnetic resonance imaging apparatuses, PET diagnostic apparatuses, etc. Furthermore, the image processing apparatus 100 according to the embodiment does not necessarily have to be a medical image processing apparatus, and may be an image processing apparatus for purposes other than medical image processing. Also, even if the image processing apparatus 100 is a medical image processing apparatus, the image processing apparatus 100 may not be incorporated into a medical imaging diagnostic apparatus and may function as a medical image processing apparatus on its own.

[0010] In Figure 1, the image processing device 100 includes a memory 132, an input device 134, a display 135, and a processing circuit 150. The processing circuit 150 includes a learning function 150a, an acquisition function 150b, a judgment function 150c, an inference execution function 150d, and a display control function 150e.

[0011] In this embodiment, each processing function performed by the learning function 150a, acquisition function 150b, judgment function 150c, inference execution function 150d, and display control function 150e is stored in memory 132 in the form of a program that can be executed by a computer. The processing circuit 150 is a processor that reads the program from memory 132 and executes it to realize the function corresponding to each program. In other words, the processing circuit 150, in the state in which each program has been read, will have the functions indicated within the processing circuit 150.

[0012] In Figure 1, the processing functions performed by the learning function 150a, acquisition function 150b, judgment function 150c, inference execution function 150d, and display control function 150e are described as being realized by a single processing circuit 150. However, the processing circuit 150 may be configured by combining multiple independent processors, and each processor may realize the functions by executing a program. In other words, each of the above functions may be configured as a program, and one processing circuit 150 may execute each program. As another example, a specific function may be implemented in a dedicated independent program execution circuit. In Figure 1, the learning function 150a, acquisition function 150b, judgment function 150c, inference execution function 150d, and display control function 150e are examples of the learning unit, acquisition unit, judgment unit, execution unit (inference unit), and display control unit, respectively. The display 135 is an example of the display unit.

[0013] The term "processor" used in the above description refers to, for example, a circuit such as a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). The processor implements functions by reading and executing programs stored in the memory 132.

[0014] Alternatively, instead of storing the programs in the memory 132, the programs may be directly embedded in the circuit of the processor. In this case, the processor implements functions by reading and executing the programs embedded in the circuit.

[0015] The processing circuit 150 performs machine learning by a learning function 150a to generate a trained model (neural network). The generated trained model is stored in the memory 132 as needed.

[0016] The processing circuit 150 acquires various types of information from the medical image diagnostic apparatus by an acquisition function 150b. The processing circuit 150 performs predetermined processing to be described later by a determination function 150c and an inference execution function 150d. The processing circuit 150 controls image generation, display, and the like by a display control function 150e. As an example, the processing circuit 150 causes the display 135 to display various generated images by the display control function 150e. In addition, the processing circuit 150 may perform overall control of the medical image diagnostic apparatus from which the image processing apparatus 100 acquires data, by the display control function 150e.

[0017] Memory 132 stores data acquired from the medical imaging diagnostic device and image data generated by the processing circuit 150. For example, memory 132 can be a semiconductor memory element such as RAM (Random Access Memory) or flash memory, a hard disk, or an optical disk.

[0018] The input device 134 receives various instructions and information inputs from the operator. The input device 134 is, for example, a pointing device such as a mouse or trackball, a selection device such as a mode switch, or an input device such as a keyboard. The display 135 displays a GUI (Graphical User Interface) for receiving input of imaging conditions, images generated by the processing circuit 150, etc., under the control of the processing circuit 150 which has a display control function 150e. The display 135 is, for example, a display device such as a liquid crystal display.

[0019] Figure 2 shows an example of an X-ray CT apparatus 200 incorporating the image processing apparatus 100 according to this embodiment.

[0020] For example, as shown in Figure 2, the X-ray CT apparatus 200 comprises a pedestal 10, a patient bed 30, and an image processing device 100.

[0021] In Figure 2, the rotation axis of the rotating frame 13 or the longitudinal direction of the top plate 33 of the bed device 30 in the non-tilted state is defined as the Z-axis direction. The axis direction perpendicular to the Z-axis direction and horizontal to the floor surface is defined as the X-axis direction. The axis direction perpendicular to the Z-axis direction and perpendicular to the floor surface is defined as the Y-axis direction.

[0022] The gantry unit 10 is a device that irradiates a subject P, such as a patient, with X-rays, detects the X-rays that have passed through the subject P, and outputs them to the image processing device 100. Specifically, the gantry unit 10 comprises an X-ray tube 11, an X-ray detector 12, a rotating frame 13, an X-ray high-voltage device 14, a control device 15, a wedge 16, an X-ray diaphragm 17, and a DAS (Data Acquisition System) 18. In Figure 2, for illustrative purposes, the gantry unit 10 is shown from the X-axis direction and from the Z-axis direction, but in reality, the X-ray CT scanner 200 has only one gantry unit 10.

[0023] The X-ray tube 11 is a vacuum tube having a cathode (filament) that generates thermionic electrons and an anode (target) that generates X-rays upon collision with these thermionic electrons. Specifically, the X-ray tube 11 generates X-rays by irradiating thermionic electrons from the cathode to the anode by applying a high voltage from the X-ray high-voltage device 14. For example, the X-ray tube 11 is a rotating anode type X-ray tube that generates X-rays by irradiating a rotating anode with thermionic electrons.

[0024] The X-ray detector 12 has multiple detection elements that detect X-rays irradiated from the X-ray tube 11 and passed through the subject P, and each detection element outputs an electrical signal to the DAS 18 corresponding to the detected dose of X-rays. Specifically, the X-ray detector 12 has multiple rows of detection elements arranged in the channel direction along the circumferential direction of the arc centered on the focal point of the X-ray tube 11, and each row of detection elements is arranged in multiple rows in the slice direction (also called the column direction or row direction).

[0025] For example, the X-ray detector 12 is an indirect conversion type detector having a collimator, a scintillator array, and a detection element array. The collimator is positioned on the X-ray incident side of the scintillator array and has an X-ray shielding plate that absorbs scattered X-rays. For example, the collimator is a one-dimensional collimator or a two-dimensional collimator. The collimator is also called a grid. The scintillator array is positioned on the X-ray incident side of the detection element array and has multiple scintillators. Each scintillator has a scintillator crystal that outputs light with a photon amount corresponding to the dose of incident X-rays. The detection element array has multiple detection elements, each detection element converts the amount of light output from the scintillator into an electrical signal corresponding to the amount of light. For example, the detection element array is constructed by arranging multiple subarrays, each of which is arranged in one dimension (n rows × 1 column) or two dimensions (n ​​rows × m columns) on the same plane, in the channel direction and slice direction. Furthermore, for example, the detection element is a light-receiving element such as a photodiode (PD) or a photomultiplier tube (PMT).

[0026] The rotating frame 13 is an annular frame that rotates the X-ray tube 11 and the X-ray detector 12 around a rotation axis (Z-axis). Specifically, the rotating frame 13 is supported by a fixed frame (not shown) so as to be rotatable around the rotation axis, with the X-ray tube 11 and the X-ray detector 12 facing each other. The rotating frame 13 rotates around the rotation axis under the control of the control device 15, thereby rotating the X-ray tube 11 and the X-ray detector 12 around the rotation axis. In addition to the X-ray tube 11 and the X-ray detector 12, the rotating frame 13 also supports an X-ray high-voltage device 14 and a DAS 18.

[0027] The X-ray high-voltage device 14 includes a high-voltage generator having an electrical circuit such as a transformer and a rectifier, and a function to generate a high voltage to be applied to the X-ray tube 11, and an X-ray control device that controls the output voltage according to the X-ray output irradiated by the X-ray tube 11. The high-voltage generator may be of the transformer type or the inverter type. The X-ray high-voltage device 14 may be installed on the rotating frame 13 within the stand device 10, or on a fixed frame (not shown) that rotatably supports the rotating frame 13.

[0028] The wedge 16 is a filter used to adjust the dose of X-rays emitted from the X-ray tube 11. Specifically, the wedge 16 is a filter that transmits and attenuates the X-rays emitted from the X-ray tube 11 so that the X-rays emitted from the X-ray tube 11 to the subject P have a predetermined distribution. For example, the wedge 16 is a filter made of aluminum processed to have a predetermined target angle and thickness. The wedge 16 is also called a wedge filter or a bow-tie filter.

[0029] The X-ray diaphragm 17 includes a lead plate or the like to narrow the irradiation area of ​​the X-rays that have passed through the wedge 16, and a slit is formed by combining multiple lead plates or the like.

[0030] DAS18 is a processing circuit that generates detection data based on electrical signals output from each detection element of the X-ray detector 12. Specifically, DAS18 amplifies the electrical signals output from each detection element of the X-ray detector 12 and generates detection data by converting the amplified electrical signals from analog signals to digital signals. The detection data generated by DAS18 is transmitted by optical communication from a transmitter having a light-emitting diode (LED) provided on the rotating frame 13 to a receiver having a photodiode provided on the non-rotating part of the stand device 10 (e.g., the support frame), and then transferred to the image processing device 100. That is, based on the data obtained by the X-ray detector 12, the processing circuit 150 of the image processing device 100 generates, for example, the input data described above. Note that the method of transmitting detection data from the rotating frame 13 to the non-rotating part of the stand device 10 is not limited to optical communication; any non-contact data transmission method may be used.

[0031] The control device 15 includes a drive mechanism such as a motor and an actuator, and a processing circuit that controls the drive mechanism. The control device 15 has the function of receiving input signals from an input interface attached to the image processing device 100 or the stand device 10, and controlling the operation of the stand device 10 and the bed device 30. For example, the control device 15 receives input signals and controls the rotation of the rotating frame 13, the tilt of the stand device 10, and the operation of the bed device 30 and the top plate 33.

[0032] The patient bed device 30 is a device for placing and moving the subject P to be scanned, and comprises a base 31, a patient bed drive device 32, a top plate 33, and a support frame 34. The base 31 is a housing that supports the support frame 34 so that it can move in the vertical direction. The patient bed drive device 32 is a motor or actuator that moves the top plate 33 on which the subject P is placed in the direction of the long axis of the top plate 33. The top plate 33, which is provided on the upper surface of the support frame 34, is a plate on which the subject P is placed. In addition to moving the top plate 33, the patient bed drive device 32 may also move the support frame 34 in the direction of the long axis of the top plate 33.

[0033] Next, the background of the embodiment will be explained.

[0034] When processing volume data using deep learning, due to memory constraints and other issues, it is sometimes necessary to divide the volume data into fixed-size Volumes of Interest (VOIs) and process each VOI separately, rather than processing the entire volume data at once.

[0035] One approach here is to narrow down the range of data to be processed beforehand using another pre-trained model, and then reduce the number of VOIs by splitting the data into VOIs. However, if the VOI size is large, processing may take a long time.

[0036] The embodiment is described in view of the above background, and the image processing apparatus 100 according to the embodiment comprises an acquisition unit, a determination unit, and an execution unit. When the acquisition unit performs inference on a trained model, it acquires features from the trained model for each region of interest into which the input data has been divided. The determination unit determines whether or not to continue inference for each region of interest based on the features acquired by the acquisition unit. The execution unit stops the inference if the determination unit determines not to continue the inference, and continues the inference if the determination unit determines to continue the inference.

[0037] In this way, by terminating inference for regions of interest where the features are not clearly defined during the feature extraction stage, the computational cost during neural network inference can be reduced, and processing can be sped up.

[0038] Figure 3 shows an example of a pre-trained model 1 (neural network) according to the embodiment. In Figure 3, the pre-trained model 1 (neural network) is explained as a CNN (Convolutional Neural Network) that performs object detection. However, the pre-trained model 1 (neural network) according to the embodiment is not limited to the example described in Figure 3. For example, the pre-trained model according to the embodiment may be various types of neural networks other than CNN, such as DNN (Deep Neural Network), Transformer, RNN (Recursive Neural Network), etc. Furthermore, the applications of the neural network are not limited to object detection, but may include region extraction, contour extraction, classification, disease detection, as well as translation using encoder-decoder models, question answering, image / video generation, and various other applications.

[0039] In the example shown in Figure 3, a two-dimensional color input image is input to the trained model 1 as input data 50. Subsequently, input data 50 is converted into data 51 for each of the R, G, and B channels.

[0040] In the example shown in Figure 3, the case where the data input to the trained model 1 is a two-dimensional image was described. However, the embodiment is not limited to this, and the data input to the trained model 1 may be a three-dimensional image, or a grayscale image may be input as input data 50 instead of a color image.

[0041] The trained model 1 includes an encoder section that compresses the data 51 using a convolution operation or the like to generate a feature map 52 that is compressed compared to the original data 51, and a decoder section that performs the reverse type of convolution operation on the obtained feature map 52c to generate data 53 of the original size again. From the data 53, which is binary data generated from the feature map 52 through the decoder, mask data 54 is generated in which object detection has been performed on the input data 50.

[0042] Here, Figure 4 shows an example of the convolution process performed in the encoder portion of the trained model 1. As shown in Figure 4, the convolution filter 42 is applied to region 41 of the image data in each layer of the neural network, generating data to be output to the next layer. The coefficients of the convolution filter 42 are determined by training the trained model 1. As shown in Figure 3, the convolution filter 42 is applied stepwise to the data 51 to generate feature maps 52a, 52b, and 52c. Comparing feature map 52a and feature map 52c, feature map 52a has a lower degree of data compression and tends to contain features from the shallower parts of the neural network, such as low-order features that reflect local structures like edge detection. In contrast, feature map 52c has a higher degree of data compression and tends to contain features from the deeper parts of the neural network, such as higher-level features that capture a more global structure.

[0043] Figure 5 is a flowchart showing the processing flow of the image processing device 100 according to the embodiment. In the flowchart of Figure 5, step S100 is the training stage processing of the trained model 1, and steps S200 to S600 show the inference stage processing of the trained model 1.

[0044] First, in step S100, the processing circuit 150 performs training on the trained model 1 (neural network) using the learning function 150a. Here, training the trained model 1 means determining the values ​​of the parameters of the trained model 1 (neural network), specifically determining the coefficients of the convolutional filters 42 in each layer of the trained model 1. The training of the trained model 1 is performed, for example, by being given a large number of pairs of input data 50 and corresponding mask data 54, and updating the parameters of the trained model 1 to minimize a predetermined loss function, for example by supervised learning. The parameter update in the trained model 1 is performed by calculating the gradient using backpropagation.

[0045] Next, we will explain the inference stage of the trained model 1.

[0046] Here, the processing circuit 150 performs steps S200 to S600 for each region of interest (ROI) obtained by dividing the input data. The input data may be, for example, two-dimensional image data or three-dimensional image data. The regions of interest obtained by dividing the input data will be two-dimensional or three-dimensional regions of interest. A three-dimensional region of interest is also called a voxel of interest (VOI).

[0047] In step S200, the processing circuit 150 then acquires features from the trained model 1 using the acquisition function 150b. In other words, when the processing circuit 150 performs inference on the trained model 1, it acquires features from the trained model 1 for each region of interest into which the input data has been divided, using the acquisition function 150b.

[0048] For example, as shown in Figure 6, the processing circuit 150 generates a feature map 52a by applying a convolutional filter 42 with filter coefficients determined during the learning phase to data 51 obtained from input data 50 during the inference phase using the inference execution function 150d, generates a feature map 52b by further applying another convolutional filter 42 with filter coefficients determined during the learning phase to feature map 52a, and generates a feature map 52c by further applying another convolutional filter 42 with filter coefficients determined during the learning phase to feature map 52b. The processing circuit 150 acquires information about these feature maps 52a, 52b, and 52c using the acquisition function 150b.

[0049] Next, in step S300, the processing circuit 150, using the determination function 150c, makes a determination for each region of interest in which processing is currently underway, based on the feature quantities acquired by the acquisition function 150b in step S200, to determine whether or not to continue inference.

[0050] If the determination function 150c determines to continue inference for the currently processed region of interest (step S300 Yes), the process proceeds to step S400, and inference continues for the currently processed region of interest (step S400). On the other hand, if the determination function 150c determines not to continue inference for the currently processed region of interest (step S300 No), the process proceeds to step S500, and inference is terminated for the currently processed region of interest (step 500).

[0051] This process will be explained using Figures 6 and 7. As shown in Figure 6, the processing circuit 150 makes a determination by sequentially determining the features in each of the multiple layers of the trained model 1 using the determination function 150c. For example, the processing circuit 150 sequentially determines feature maps 52a, 52b, and 52c in the trained model 1 using the determination function 150c, and if it determines that "features exist" in all determinations, it decides to continue inference (step S300 Yes). Here, the processing circuit 150, using the determination function 150c, determines whether the sum of all elements in the map included in that feature map exceeds a predetermined threshold. For example, if the sum of all elements in feature map 52a is "10", the sum of all elements in feature map 52b is "20", the sum of all elements in feature map 52c is "30", and the thresholds for the judgments regarding feature maps 52a, 52b, and 52c are "5", "10", and "15", respectively, then the processing circuit 150, using its judgment function 150c, will determine that the features "exist" at judgment points 55a, 55b, and 55c, respectively, and will continue the inference (step S300 Yes). In this case, the processing proceeds to step S400 for this region of interest, and the processing circuit 150 performs inference using its inference execution function 150d, generating data 53 and mask data 54 for that region of interest.

[0052] On the other hand, for example, as shown in Figure 7, if the sum of all elements in the map included in feature map 52a is "10", the sum of all elements in the map included in feature map 52b is "20", the sum of all elements in the map included in feature map 52c is "30", and the judgment thresholds for feature maps 52a, 52b, and 52c are "5", "30", and "15", respectively, then the processing circuit 150, using the judgment function 150c, determines that the value of feature map 52b is below the judgment threshold, and therefore the judgment function 150c determines that there are "no" features in feature map 52b. As a result, the processing circuit 150, using the judgment function 150c, decides not to continue inference for process 60 in the current region of interest (Step S300 No). In this case, the process proceeds to step S500 for this region of interest, and the processing circuit 150 stops inference.

[0053] Next, in step S600, the processing circuit 150 uses the display control function 150e to display the result of the determination in step S300 on the display unit, the display 135.

[0054] This situation is illustrated in Figure 8. In Figure 8, the voxels 70, 71, and 72 represented by each cube represent the voxels (VOIs) in each region of interest that are processed by the trained model 1. Here, the uncolored voxel 70 represents a voxel that the processing circuit 150 determined, based on its decision function 150c, not to continue inference in step S300. The colored voxels 71 and 72 represent voxels that the processing circuit 150 determined, based on its decision function 150c, to continue inference in step S300.

[0055] Furthermore, for voxels that the processing circuit 150 determines to continue inference for in step S300 using the determination function 150c, the processing circuit 150 uses the display control function 150e to display the feature quantities acquired by the acquisition function 150b in step S200 on the display unit, the display 135. For example, the processing circuit 150 may use the display control function 150e to color voxel 71 or voxel 72 with a color corresponding to the feature quantity acquired by the acquisition function 150b in step S200. For example, the processing circuit 150 may use the display control function 150g to color voxel 71 or voxel 72 with a darker color the larger the value of the feature quantity acquired by the acquisition function 150b in step S200. Another example is that the processing circuit 150 may use the display control function 150e to display the magnitude of the feature quantity for each voxel, for example, as shown in Figure 9, using a bar graph.

[0056] As another example, the processing circuit 150 may use the display control function 150e to superimpose the output of the inference results for each region of interest onto the judgment results for each region of interest and display them on the display unit, the display 135.

[0057] Furthermore, the processing circuit 150 may, via the display control function 150e, receive input from the user to change the threshold value for which the determination function 150c determines whether or not to continue inference in step S300.

[0058] For example, in neural networks, it is known that the information held by a feature map differs depending on the depth of the feature map. For instance, it is known that feature maps in shallow layers, i.e., layers close to the input image, extract local structures such as edge detection. On the other hand, it is known that feature maps in deep layers, i.e., layers far from the input image, extract abstract structures that span the entire image as feature maps. The processing circuit 150, using the display control function 150e, can not only change the threshold at which it can determine that a feature has been extracted by changing the judgment threshold, but can also change the "depth" at which feature maps are primarily extracted by changing the judgment threshold for each depth. The processing circuit 150 can improve the accuracy of inference by re-inferring based on the changed judgment threshold.

[0059] Thus, in the first embodiment, the decision function 150c determines whether or not to terminate the inference for each region of interest based on the features, and the inference is terminated accordingly. This reduces the computational cost during neural network inference and speeds up processing in the inference stage.

[0060] (Modified version of the first embodiment) The embodiments are not limited to the examples described above, and various user interfaces are conceivable.

[0061] For example, the display control function 150e may, when a user hovers the mouse over a specific region of interest or VOI and selects that region of interest or VOI, display the feature quantities of that region of interest superimposed on the results of the determination for that region of interest. In other words, the processing circuit 150 may, using the display control function 150e, superimpose the output of the feature quantities or inference results on the results of the determination for that region of interest and display them on the display unit, the display 135, for the region of interest selected by the user.

[0062] As another example, as shown in Figure 10, the processing circuit 150 may, using the display control function 150e, further display the result of the decision on whether or not to continue inference for each region of interest in natural language on the display unit, the display 135. For example, the processing circuit 150 may, using the display control function 150e, display the message "A feature for XX was found." superimposed on each voxel for regions of interest where the feature value exceeds a threshold, and display the message "A feature for XX was not found." superimposed on each voxel for regions of interest where the feature value falls below a threshold.

[0063] As another example, the processing circuit 150 may use the display control function 150e to distinguish the region of interest in which inference is being performed from other regions of interest and display it on the display unit. For example, as shown in Figure 11, the processing circuit 150 may use the display control function 150e to make the region of interest 80 in which inference is being performed blink, thereby distinguishing it from other regions of interest and displaying it on the display unit.

[0064] As another example, when calculations are performed by executing multiple algorithms (trained models), the features obtained from the neural network corresponding to each algorithm may be displayed to the user. For example, the processing circuit 150 may acquire features on an algorithm-by-algorithm basis and display the features in a gradient using colors of varying intensity according to the magnitude of the acquired features. For example, the display control function 150e may display algorithms with larger features using a gradient of darker colors on the display 135. Since a larger feature count is considered to indicate a higher probability of algorithm success, algorithms (trained models) displayed with a gradient of darker colors are considered to be more successful in their respective domains of interest. In other words, by viewing the features for each algorithm, the user can grasp the relative strengths and weaknesses of multiple algorithms (trained models) in their respective domains of interest. That is, when processing is performed using multiple trained models, the processing circuit 150 may use the display control function 150e to display information regarding the features of each of the multiple trained models on the display unit.

[0065] Figure 12 shows an example of such a case. In the case of Figure 12, multiple trained models (algorithms 1 to 4) are shown being executed on the region of interest 95. In this case, when the given region of interest 95 is processed by multiple trained models (algorithms), the processing circuit 150 uses the acquisition function 150b to acquire the feature quantities obtained when inference is performed on the region of interest 95 for each of the multiple trained models. The processing circuit 150 uses the display control function 150e to display information about the feature quantities of each of the multiple trained models on the display panel 90, displaying the information with a gradient of colors corresponding to the magnitude of the feature quantities, and sorting it by the magnitude of the feature quantities, and displaying it in the display areas 91a, 91b, 91c, and 91d. This allows the user to visually confirm which trained model (algorithm) has the largest feature quantities and is superior in the region of interest 95.

[0066] Furthermore, the display control function 150e may also superimpose the feature quantities in each region of interest onto the input data and display them on the display unit 135. Figure 13 shows an example of this. In Figure 13, the display control function 150e superimposes the feature quantities in each region of interest onto the input medical image, which is the input image, using circles 92, 93, etc., of a size corresponding to the size of the feature quantities, and displays them on the display unit. In this way, by simply resizing and superimposing the feature quantity map onto the input image for each region of interest, the user can visually confirm the feature quantities in each region of interest.

[0067] (Second embodiment) In the first embodiment and its modifications, the case in which inference is stopped for each region of interest according to the value of the feature was described. In the second embodiment, the case in which learning (parameter updating) is performed only in the encoder part for regions of interest that do not have features is described, not in the inference stage but in the learning stage. That is, in the second embodiment, the processing circuit 150 comprises an acquisition function 150b, a determination function 150c, and a learning function 150a as an execution unit. When the processing circuit 150 performs learning of the trained model using the acquisition function 150b, it acquires features from the trained model for each region of interest into which the input data has been divided. The processing circuit 150 uses the determination function 150c to determine for each region of interest whether or not to continue learning the trained model in that region, based on the features acquired by the acquisition function 150b. Furthermore, the processing circuit 150, acting as an execution unit, stops learning for the region of interest if the determination function 150c determines, based on the learning function 150a, that learning should not continue for that region of interest. If the determination function 150c determines that learning should continue, the processing circuit 150 continues learning for that region of interest. An example of this is shown in Figure 14. If the processing circuit 150, based on the determination function 150c, determines, for example, at determination point 55b, that the acquired feature quantity is below a threshold, the processing circuit 150 does not update the decoder part parameters for the current region of interest, as shown in process 60, that is, it does not perform learning for the decoder part of the trained model 1. This reduces the computational cost required to train the trained model 1.

[0068] (Other embodiments) While various data visualization methods have been described in the embodiments described so far, the methods for data visualization are not limited to the examples above. For example, in the confidence level determination of generated sentences in an LLM (Large Language Model) using a Transformer, the confidence level of a sentence may be visualized using the probability used when selecting the next word (transition probability). As an example, the processing circuit 150 determines, by the determination function 150c, that sentences composed of sequences of word pairs with high transition probabilities are more reliable than sentences that do not have high transition probabilities. The display control function 150e may display sentences that the determination function 150c has determined to have high confidence on the display 135, for example, in a darker color compared to sentences that do not have high confidence. This allows the user to visually confirm the confidence level of sentences in the LLM, improving user convenience.

[0069] Furthermore, the examples described so far have explained the case where the input data is divided into multiple regions of interest, and the acquisition function 150b calculates features for each divided region of interest. As an example, the case where the input data is divided into fixed-size VOIs, and the acquisition function 150b calculates features for each divided VOI, has been described. However, the embodiments are not limited to these.

[0070] For example, the input data does not necessarily have to be divided into fixed-size VOIs.

[0071] For example, when performing inference on a trained model, the acquisition function 150b may acquire features from the trained model for regions of interest set based on the input data. The decision function 150c then determines whether or not to continue inference based on the acquired features. Here, the acquisition function 150b is not required to acquire features for each region of interest, nor is the decision function 150c required to determine whether or not to continue inference for each region of interest. In other words, the features calculated by the acquisition function 150b and the regions of interest do not have to have a one-to-one correspondence.

[0072] For example, the acquisition function 150b may calculate features using a two-stage trained model. In this case, the acquisition function 150b extracts or sets a region of interest, such as the liver region, based on the input data in the first stage of the trained model. Subsequently, the judgment function 150c makes a decision based on the acquired features whether or not to continue inference regarding the region of interest.

[0073] Furthermore, the region of interest may correspond to a portion of the input data, or it may correspond to the entire input data.

[0074] With respect to the above embodiments, the following additional notes are disclosed as aspects of the invention and selective features.

[0075] (Note 1) An X-ray CT apparatus provided in one aspect of the present invention comprises an X-ray detector and a processing circuit. The X-ray detector detects X-rays that have passed through a subject irradiated from an X-ray tube. The processing circuit generates input data based on the data obtained by the X-ray detector and, when performing inference on a trained model, obtains features from the trained model for a region of interest set based on the input data, and determines whether or not to continue the inference for the region of interest based on the obtained features. If it determines not to continue the inference, it stops the inference; if it determines to continue the inference, it continues the inference.

[0076] (Note 2) An image processing apparatus provided in one aspect of the present invention comprises a processing circuit. When the processing circuit performs inference on a trained model, it obtains features from the trained model for a region of interest set based on the input data, and determines whether or not to continue the inference for the region of interest based on the obtained features. If it determines not to continue the inference, it cancels the inference, and if it determines to continue the inference, it continues the inference.

[0077] (Note 3) The aforementioned region of interest may be a region of interest obtained by dividing the input data.

[0078] (Note 4) The aforementioned region of interest may be a region of interest that corresponds to a part of the input data.

[0079] (Note 5) The processing circuit acquires the feature quantities for each region of interest, Based on the acquired features, a decision may be made for each region of interest as to whether or not to continue the inference.

[0080] (Note 6) The image processing device may display the result of the determination on the display unit.

[0081] (Note 7) The aforementioned input data is two-dimensional or three-dimensional image data. The aforementioned region of interest may be two-dimensional or three-dimensional.

[0082] (Note 8) The processing circuit may perform the determination by sequentially determining the features in each of the multiple layers of the trained model.

[0083] (Note 9) The processing circuit may superimpose the output of the feature quantities or the inference results on the determination results for each region of interest selected by the user and display them on the display unit.

[0084] (Note 10) The processing circuit may accept input from the user to change the threshold value for the determination.

[0085] (Note 11) The processing circuit may further display the result of the determination in natural language on the display unit.

[0086] (Note 12) The processing circuit may also display the region of interest in which the inference is being performed on the display unit, distinguishing it from other regions of interest.

[0087] (Note 13) The aforementioned trained model consists of multiple trained models, The processing circuit may display information regarding the features of each of the plurality of trained models on the display unit.

[0088] (Note 14) The processing circuit may superimpose the feature quantities onto the input data and display them on the display unit.

[0089] (Note 15) An image processing apparatus provided in one aspect of the present invention comprises a processing circuit. When the processing circuit performs training of a pre-trained model, it obtains features from the pre-trained model for regions of interest into which the input data has been divided, and determines whether or not to continue training the pre-trained model in the region of interest based on the features. If it determines not to continue training in the region of interest, it stops training in the region of interest, and if it determines to continue training, it continues training in the region of interest.

[0090] According to at least one embodiment described above, the computational cost during neural network inference can be reduced.

[0091] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments are possible without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0092] 132 memory 134 Input device 135 displays 150 Processing Circuits 150a Learning function 150b acquisition function 150c judgment function 150d Inference Execution Function 150e Display Control Function

Claims

1. When performing inference on a trained model, the system includes an acquisition unit that acquires features from the trained model for a region of interest set based on the input data, A determination unit determines whether or not to continue the inference with respect to the region of interest based on the feature quantities acquired by the acquisition unit, If the determination unit determines that the inference should not be continued, the execution unit will terminate the inference, and if the determination unit determines that the inference should be continued, the execution unit will continue the inference. An image processing device equipped with the following features.

2. The aforementioned region of interest is a region of interest obtained by dividing the input data. The image processing apparatus according to claim 1.

3. The image processing apparatus according to claim 1, wherein the region of interest is a region of interest corresponding to a part of the input data.

4. The acquisition unit acquires the feature quantities for each of the regions of interest into which the input data has been divided. The image processing apparatus according to claim 1, wherein the determination unit determines whether or not to continue the inference for each region of interest based on the acquired feature quantities.

5. Display control unit that displays the result of the determination unit's determination on the display unit. The image processing apparatus according to claim 1, further comprising the following:

6. The aforementioned input data is two-dimensional or three-dimensional image data. The image processing apparatus according to claim 1, wherein the region of interest is a two-dimensional or three-dimensional region of interest.

7. The image processing apparatus according to claim 1, wherein the determination unit performs the determination by sequentially determining the feature quantities in each of the multiple layers of the trained model.

8. The image processing apparatus according to claim 5, wherein the display control unit overlays the output of the feature quantity or the inference result on the result of the determination for the region of interest selected by the user and displays it on the display unit.

9. The image processing apparatus according to claim 5, wherein the display control unit receives input from the user for a change in the threshold value of the determination.

10. The image processing apparatus according to claim 5, wherein the display control unit further causes the result of the determination to be displayed on the display unit in natural language.

11. The image processing apparatus according to claim 5, wherein the display control unit causes the region of interest in which the inference is being performed to be displayed on the display unit, distinguishing it from other regions of interest.

12. The aforementioned trained model consists of multiple trained models, The image processing apparatus according to claim 5, wherein the display control unit causes the display unit to display information regarding the feature quantities of each of the plurality of trained models.

13. The image processing apparatus according to claim 5, wherein the display control unit superimposes the feature quantities onto the input data and displays them on the display unit.

14. When training a pre-trained model, an acquisition unit acquires features from the pre-trained model for a region of interest set based on the input data, A determination unit, based on the feature quantities acquired by the acquisition unit, determines whether or not to continue training the trained model in the region of interest, If the determination unit determines that learning should not be continued for the area of ​​interest, the execution unit will discontinue learning for the area of ​​interest, and if the determination unit determines that learning should be continued, the execution unit will continue learning for the area of ​​interest. An image processing device equipped with the following features.

Images