Image processing method, image processing apparatus, and computer program

Abstract

Dynamically adjusting patches to accelerate the inference process of deep learning models and apply them to different inference tasks. [Solution] The image processing method according to this embodiment performs an inference task on image data using a trained deep learning model, first setting a plurality of patch parameters according to the inference task. Next, a plurality of patches are generated for each of the patch parameters, an inference result is obtained based on the plurality of generated patches, and an optimized patch parameter and a plurality of patches corresponding to the optimized patch parameter are determined from the plurality of patch parameters based on the inference result. Next, the inference task is performed using the deep learning model based on the plurality of patches corresponding to the determined optimized patch parameter and an inference result is obtained.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to an image processing method, an image processing apparatus, and a computer program.

Background Art

[0002] Deep learning is an algorithm that performs feature learning on data based on a neural network. The purpose of feature learning is to obtain a better representation method for data and construct a better deep learning model, and to learn this representation method in large-scale unlabeled data. Deep learning is particularly applicable to image processing tasks and can obtain good results. In image processing tasks, deep learning can be used for image processing such as segmentation, classification, and detection of images.

[0003] However, there are various problems with conventional deep learning. For example, a deep learning model usually requires a large number of parameters, has a complex computational graph, and in the task of segmenting high-resolution 3D images, it requires a large amount of computational resources, leading to an increase in inference time. Also, in a clinical environment, diagnosis and treatment decisions usually require quickly obtaining image processing results, and there is a requirement for the response speed of the deep learning model. Furthermore, hardware devices such as embedded or mobile platforms on medical devices usually have limited computational resources. Taking mobile medical devices or handheld ultrasound devices as an example, it is difficult to execute a complex deep learning-based segmentation model on such devices.

[0004] To solve the above various problems, the following prior arts have been proposed.

[0005] One conventional technique involves improving deep learning models using the sliding-window method. This method divides a complete image into fixed-size patches in a sliding window format, has the deep learning model perform inference in each patch, and then fuses the inference results from each patch into the final inference result. This method avoids the problem of insufficient computational resources required for inference on a complete image.

[0006] Another conventional technique uses the coarse-to-fine method to improve deep learning models. This method first uses a coarse model to identify priority target regions, and then uses a fine model to perform inference in those priority target regions. This approach can improve inference speed by reducing processing in the background region.

[0007] Furthermore, there is a technique called model compression that reduces the computational complexity of a deep learning model by reducing the number of parameters, pruning, quantization, or using a lightweight model architecture. There is also a technique that reduces computational complexity and improves inference speed while maintaining the performance of the deep learning model by distilling a complex deep learning model into a simplified model.

[0008] Furthermore, there is a technique that combines the sliding window method and the coarse-to-fine method (Coarse-to-fine + Sliding-Window), which generates the same patch for different tasks or objectives using pre-set patch parameters such as aspect ratio and length.

[0009] Therefore, the problems with such technologies are that it is difficult to obtain optimal inference results when faced with the needs, shapes, and volumes of different tasks and targets, and that they result in computational redundancy and long inference times. For example, in the conventional technologies described above, the same patch is usually generated for different tasks and targets using pre-set patch parameters such as aspect ratio and length. As a result, it is difficult to obtain optimal inference results when faced with the needs, shapes, and volumes of different tasks and targets, and this leads to computational redundancy and long inference times. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2020-25779 [Overview of the project] [Problems that the invention aims to solve]

[0011] One of the problems that the embodiments disclosed herein and in the drawings aim to solve is to dynamically adjust patches to accelerate the inference process of a deep learning model and to apply it to different inference tasks. However, the problems solved by the embodiments disclosed herein and in the drawings are not limited to the above problem. Problems corresponding to each effect of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0012] The image processing method according to this embodiment performs an inference task on image data using a trained deep learning model, and includes: a setting step of setting a plurality of patch parameters according to the inference task; a planning step of generating a plurality of patches for each of the patch parameters, obtaining an inference result based on the plurality of generated patches, and determining an optimized patch parameter and a plurality of patches corresponding to the optimized patch parameter from among the plurality of patch parameters based on the inference result; and an inference step of performing the inference task using the deep learning model and obtaining an inference result based on the plurality of patches corresponding to the determined optimized patch parameter. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 is a block diagram showing an example configuration of an image processing system according to the first embodiment. [Figure 2] Figure 2 is a block diagram showing an example configuration of an image processing apparatus according to the first embodiment. [Figure 3] Figure 3 is a schematic diagram showing input images with different patch parameters. [Figure 4] Figure 4 is a graph showing the relationship between the patch length ratio and the accuracy of the inference results. [Figure 5A] Figure 5A is a schematic diagram showing the plan for multiple patches. [Figure 5B] Figure 5B is a schematic diagram showing the plan for multiple patches. [Figure 5C] Figure 5C is a schematic diagram showing a plan for multiple patches. [Figure 5D] Figure 5D is a schematic diagram showing a plan for multiple patches. [Figure 5E] Figure 5E is a schematic diagram showing a plan for multiple patches. [Figure 5F] Figure 5F is a schematic diagram showing the plan for multiple patches. [Figure 6] Figure 6 is a graph showing the relationship between the time and accuracy of the inference results. [Figure 7]FIG. 7 is a schematic diagram showing a user interface for editing a patch by an editing function. [Figure 8] FIG. 8 is a flowchart showing an image processing method according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing a patch plan according to the first embodiment. [Figure 10] FIG. 10 is a block diagram showing a configuration example of an image processing apparatus according to the second embodiment. [Figure 11] FIG. 11 is a schematic diagram showing examples of clinical data and important information. [Figure 12] FIG. 12 is a block diagram showing a configuration example of an image processing apparatus according to the third embodiment. [Figure 13] FIG. 13 is a flowchart showing model update by a model update function. [Figure 14A] FIG. 14A is a schematic diagram for explaining the difference between the patch plan according to the comparative example and the patch plan by the image processing method according to the present invention, and is a diagram showing a plurality of patches planned by the comparative example. [Figure 14B] FIG. 14B is a schematic diagram for explaining the difference between the patch plan according to the comparative example and the patch plan by the image processing method according to the present invention, and is a diagram showing a plurality of patches planned by the comparative example. [Figure 14C] FIG. 14C is a schematic diagram for explaining the difference between the patch plan according to the comparative example and the patch plan by the image processing method according to the present invention, and is a diagram showing a plurality of patches planned by the image processing method according to the present embodiment.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of an image processing method, an image processing apparatus, and a computer program will be described with reference to the accompanying drawings.

[0015] (First Embodiment) Referring to Figure 1, an example of the configuration of the image processing system 100 will be described. Figure 1 is a block diagram showing an example of the configuration of the image processing system 100 according to the first embodiment.

[0016] The image processing system 100 comprises an image processing device 1, an operation terminal 2, an image data generation device 3, an image data preprocessing device 4, and a medical data storage device 5. Each device is connected to the others so as to be able to communicate with each other via a common signal transmission path, such as a bus or a network.

[0017] Image processing device 1 is a device that performs an inference task using a trained deep learning model on medical image data generated by image data generation device 3. Image processing device 1 may function as a server in, for example, an image processing system 100, and may be a workstation capable of performing high-speed information processing.

[0018] The operating terminal 2 is a terminal operated by an operator (e.g., a doctor or technician). The operating terminal 2 functions as a client in the image processing system 100. The operating terminal 2 may be a computer, a notebook computer, a smartphone, a tablet or wearable device, or a mobile or handheld device.

[0019] The operating terminal 2 may, for example, display a user interface UI (UI) as described later, so that the user can operate, input, or edit it.

[0020] Furthermore, the operating terminal 2 may be equipped with an input interface that accepts various operations from the user. The input interface converts the various operations received from the operator into electrical signals and transmits the electrical signals to the processing circuit of the operating terminal 2. The input interface of the operating terminal 2 may be a mouse, keyboard, buttons, panel switches, slider switches, trackballs, control panels, or touch panels.

[0021] Furthermore, the operating terminal 2 may be equipped with a display that shows various data and information. Examples of displays include liquid crystal displays, plasma displays, organic electro-luminescence (OLED) displays, and light-emitting diode (LED) displays. The display may also be a touch panel display that also functions as an input interface.

[0022] Image data generation device 3 is a device that generates image data of medical images (hereinafter abbreviated as "image data"). Image data generation device 3 may be a medical imaging diagnostic device (for example, an X-ray diagnostic device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, an ultrasound diagnostic device, a nuclear medicine imaging device, etc.). Image data generation device 3 transmits the generated image data to the medical data storage device 5.

[0023] The image data preprocessing device 4 is a device for preprocessing image data. The image data preprocessing device 4 comprises a processing circuit (not shown) and a memory (not shown) that stores computer programs that implement each function in the processing circuit. For example, the processing circuit includes a division function 41. Here, the division function 41 is an example of a "division step".

[0024] The segmentation function 41 distinguishes and divides the image data into regions of interest and regions of non-interest. As a result of the segmentation, for example, the segmentation function 41 divides the image data into regions of interest, including bone regions, and into regions of non-interest, including blood vessel regions. In this way, the segmentation function 41 extracts regions of interest by segmenting the image data. The position of each segmented region can be represented as positional information in the subject coordinate system of the image data. The segmentation by the segmentation function 41 can be modified using known image recognition techniques. For example, a threshold algorithm may be applied to segment the image data into regions of interest where the CT value is greater than 150HU, or coarse segmentation or a detection model (e.g., nnUNet Coarse model) may be applied to segment the regions of interest. The image data preprocessing device 4 transmits the preprocessing results from the segmentation function 41 to the medical data storage device 5.

[0025] The medical data storage device 5 is a device for storing medical data. The medical data storage device 5 stores, for example, image data, clinical data (described later), inference results based on deep learning models, and user editing results. The medical data storage device 5 may be a storage medium (for example, a magnetic storage medium, an electromagnetic storage medium, an optical storage medium, or a semiconductor memory), or it may be a drive device that reads and writes information to and from the storage medium. The medical data storage device 5 stores image data transmitted from the image data generation device 3 and preprocessing results transmitted from the image data preprocessing device 4.

[0026] Referring to Figure 2, an example of the configuration of the image processing apparatus 1 will be described. Figure 2 is a block diagram showing an example of the configuration of the image processing apparatus 1 according to the first embodiment.

[0027] The image processing device 1 comprises a processing circuit 11, a memory 12, and a communication IF 13. Each component is connected to the others via a bus, which is a common signal transmission path, enabling communication between them.

[0028] Memory 12 is a device for storing various types of data and information. Memory 12 may be a storage medium readable by the processor (for example, a magnetic storage medium, an electromagnetic storage medium, an optical storage medium, or a semiconductor memory), or it may be a drive device that reads and writes data and information to and from the storage medium. Memory 12 stores the computer programs that enable the processing circuit 11 to implement each functional unit.

[0029] The communication IF13 is an interface that communicates various types of data and information with each component included in the image processing system 100. The communication IF13 communicates various types of data and information with the operation terminal 2 and the medical data storage device 5.

[0030] The processing circuit 11 is a circuit that controls the overall operation of the image processing device 1. The processing circuit 11 includes at least one processor. A processor refers to a circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA)). If the processor is a CPU, the CPU realizes each function by reading and executing each program stored in memory 12. If the processor is an ASIC, each function is directly incorporated as a logic circuit within the ASIC's circuitry. The processor may be configured as a single circuit or as a combination of multiple independent circuits. The processing circuit 11 realizes the functions of each functional unit described later.

[0031] As shown in Figure 2, the processing circuit 11 includes an acquisition function 111, a setting function 112, a planning function 113, an inference function 114, and an output function 115. Here, the acquisition function 111 is an example of an "acquisition step". The setting function 112 is an example of a "setting step" and a "setting unit". The planning function 113 is an example of a "planning step" and a "planning unit". The inference function 114 is an example of an "inference step" and an "inference unit". The output function 115 is an example of an "output step".

[0032] The acquisition function 111 acquires various types of data and information. For example, the acquisition function 111 acquires image data and also acquires the results of the regions of interest and regions of non-interest obtained by the division function 41. The acquisition function 111 can acquire various types of data and information by accessing the medical data storage device 5.

[0033] The recommended settings for patch parameters using the setting function 112 will be explained with reference to Figures 3 and 4. Figure 3 is a schematic diagram showing input images with different patch parameters. Figure 4 is a graph showing the relationship between the patch length ratio and the accuracy of the inference results.

[0034] The configuration function 112 sets multiple patch parameters depending on the inference task performed by the deep learning model. Here, the patch parameters include, for example, base patch size, patch size ratio, base resolution, and resolution ratio. The patch parameters may also be parameters that determine the position, length, and resolution of other patches.

[0035] Furthermore, the setting function 112 sets recommended values ​​for each of the multiple patch parameters. Here, if the multiple patch parameters include, for example, base patch length, patch length ratio, base resolution, and resolution ratio, the recommended values ​​for each of the multiple patch parameters are, for example, base patch length = (128, 128, 128), patch length ratio = 0.9 to 1.35, base resolution = (1.0, 1.0, 1.0), and resolution ratio = 0.9 to 1.3. In the first embodiment, an example was described in which the recommended values ​​for the patch parameters are a range of values ​​for the patch parameters, but the recommended values ​​for the patch parameters may also be specific values ​​for the patch parameters.

[0036] Specifically, the setting function 112 sets recommended values ​​for multiple patch parameters based on parameters associated with the inference tasks performed by the deep learning model, from the history database DB stored in the medical data storage device 5. For example, the setting function 112 sets recommended values ​​for each of the multiple patch parameters based on the parameters of the organ segmentation data contained in the history database DB stored in the medical data storage device 5. Here, the organ segmentation data is the inference result of the segmentation inference task performed by the deep learning model, and the inference task performed by the deep learning model may be a tumor detection task, a lung lesion classification task, etc.

[0037] The setting function 112 may also set recommended values ​​for multiple patch parameters based on user input to the operation terminal 2. For example, the user specifies base patch length, patch length ratio, base resolution, and resolution ratio as multiple patch parameters on the operation terminal 2, and inputs their recommended values ​​via the operation terminal 2. The setting function 112 sets recommended values ​​for multiple patch parameters based on user input.

[0038] The setting function 112 may generate a corresponding input image for each patch parameter having a fixed interval within a preset range of patch parameters, obtain the inference result for each input image, and determine a recommended value for the patch parameter based on the inference result.

[0039] Specifically, we will explain using the example of determining a recommended value for the patch length ratio. For example, if the base patch length is (128, 128, 128), and within the preset range of 0.6 to 1.6 for the patch length ratio, a series of corresponding input images are generated for each patch length ratio with an interval of 0.05, and the inference result for each input image is obtained. In this case, the length of the input image = base patch length * patch length ratio. When the patch length ratio = 1.0, the length of the input image = (128, 128, 128), and the accuracy of the inference result is 0.972. However, when the patch length ratio = 1.25, the length of the input image = (160, 160, 160), and the accuracy of the inference result is 0.971. This yields a series of corresponding input images generated for each patch parameter with a fixed interval within the preset range shown in Figure 3.

[0040] Furthermore, the setting function 112 can also obtain the relationship between the patch length ratio included in the patch parameters, as shown in Figure 4, and the accuracy of the inference result, "DICE (%)" (the so-called "Dice coefficient"). From the relationship shown in Figure 4, it can be seen that the range of the patch length ratio that has little impact on the accuracy of the inference result is 0.9 to 1.35. Therefore, the setting function 112 determines the recommended value for the patch length ratio to be 0.9 to 1.35. Similarly, the setting function 112 determines the recommended values ​​for the base patch length, base resolution, and resolution ratio.

[0041] Here, the predetermined range for the patch length ratio, which is determined a priori, is 0.6 to 1.6. However, the predetermined range for the patch parameters may be determined by other methods, and the fixed interval for the patch length ratio is 0.05, but other intervals may be used. Neither the "predetermined range" nor the "fixed interval" for the patch parameters are limited; they should be used to determine the recommended values ​​for these patch parameters.

[0042] Refer to Figures 5A to 5F to explain the patch planning using the planning function 113. Figures 5A to 5F are schematic diagrams showing the planning of multiple patches.

[0043] The planning function 113 determines the patch parameters and plans the patch. Specifically, the planning function 113 generates multiple patches for each patch parameter.

[0044] The planning function 113 includes a patch generation function 113A, a patch adjustment function 113B, and a region of interest update function 113C. The patch generation function 113A, the patch adjustment function 113B, and the region of interest update function 113C are examples of the "patch generation step," the "patch adjustment step," and the "region of interest update step," respectively.

[0045] The patch generation function 113A generates a patch for each patch parameter.

[0046] Specifically, as shown in Figure 5A, the patch generation function 113A generates a patch (dashed line frame in Figure 5A) at the position of a corner point (also called an angle point, corner point, or corner point) of the region of interest (shaded area in Figure 5A) output from the image data preprocessor 4 (position indicated by a circle in Figure 5A). Here, a "corner point" of the region of interest refers, for example, to any of the eight endpoints of the rectangle circumscribing the region of interest in Figure 5F. The patch generation function 113A acquires these eight endpoints and generates a patch at one of them. Here, the patch generation function 113A, for example, selects the endpoint with the smallest coordinate value on mutually orthogonal XYZ coordinate systems to generate the patch.

[0047] Next, the patch adjustment function 113B adjusts the patch.

[0048] Specifically, as shown in Figure 5B, the patch adjustment function 113B sets the patch length based on the patch parameters. For the sake of explanation, we will use the following example: base resolution = (1.0, 1.0, 1.0), resolution ratio = 1.25, and base patch length * patch ratio = (128, 128, 128). The patch adjustment function 113B adjusts the patch length according to the length of the region of interest, that is, optimizes the patch length based on the length of the region of interest, in order to reduce the number of patches and accelerate the inference process of the deep learning model. At this time, the lengths on the mutually orthogonal XYZ axes of the region of interest are (Lx, Ly, Lz), and the lengths (x, y, z) of the patch on the XYZ axes are adjusted based on the following equation 1 using the base patch size, base resolution, resolution ratio, ceiling function (ceil), and INT function (int).

[0049]

number

[0050] Here, `int` indicates rounding down, and `ceil` indicates rounding up. This adjusts the patch length to (110, 96, 148). Thus, the patch length can be adapted to the length of the region of interest.

[0051] As shown in Figure 5C, the patch adjustment function 113B moves the position of the adjusted patch so that the area of ​​interest covered by the patch increases and the area of ​​non-interest covered by the patch decreases. Here, first, it is ensured that the area of ​​interest initially covered by the patch does not decrease, and then the position of the patch is adjusted so that it covers a larger area of ​​interest. Specifically, in Figure 5B, the upper left of the patch covers an excess area of ​​non-interest. Therefore, in Figure 5C, the patch is moved to the lower right so that it increases the area of ​​interest it covers and decreases the area of ​​non-interest it covers. Alternatively, the length of the patch may be reduced by moving the left and top edges of the patch to the lower right, thereby reducing the area of ​​non-interest it covers.

[0052] Next, the region of interest update function 113C updates the region of interest by subtracting the patch from the region of interest of the image data.

[0053] Specifically, as shown in Figure 5D, the region of interest update function 113C subtracts the patches shown in the dashed box from the region of interest shown in the shaded area in Figure 5C, and updates the region of interest to the new region of interest shown in the shaded area in Figure 5D.

[0054] Next, as shown in Figures 5E and 5F, the planning function 113 generates multiple patches in the updated new region of interest by executing the functions of the patch generation function 113A, patch adjustment function 113B, and region of interest update function 113C described above, until the volume of the updated new region of interest becomes zero. That is, in the updated new region of interest, the planning function 113 repeatedly executes the steps of patch generation by the patch generation function 113A, patch adjustment by the patch adjustment function 113B, and updating of the new region of interest by the region of interest update function 113C until the generated patches cover all regions of interest and the volume of the updated new region of interest becomes zero.

[0055] As shown in Figures 5D to 5F, in the first cycle, the planning function 113 generates the patch indicated by the dashed box in Figure 5D; in the second cycle, the planning function 113 generates further patches indicated by the right-hand dashed box in Figure 5E; and in the third cycle, the planning function 113 generates further patches indicated by the central dashed box in Figure 5F.

[0056] In subsequent cycles, the patch generation function 113A selects different endpoints from the eight endpoints of the rectangle circumscribing the region of interest that were used in the previous cycle to generate the patch.

[0057] Furthermore, the planning function 113 generates multiple patches for each patch parameter at the recommended values ​​set by the setting function 112. As a result, the planning function 113 generates multiple patches corresponding to each patch parameter, as shown in Figure 5F.

[0058] The patch parameters, patch length and position, number of cycles, and number of patches mentioned above are provided as examples only and are not limited to the values ​​mentioned herein.

[0059] Referring to Figure 6, we will explain the curve showing the relationship between inference time (seconds) and accuracy (%). Figure 6 is a graph showing the relationship between inference time and accuracy.

[0060] Next, the planning function 113 obtains inference results based on the generated multiple patches and, based on the inference results, determines the optimal patch parameter and the multiple patches corresponding to that optimal patch parameter from among the multiple patch parameters.

[0061] Specifically, the planning function 113 generates multiple patches for each patch parameter, thereby obtaining a series of patches for each patch parameter. The planning function 113 obtains inference results based on these multiple patches. Here, the inference results include the time and accuracy of the inference based on the multiple patches. This yields a trade-off curve, shown in Figure 6, which represents the relationship between the time and accuracy of the inference results.

[0062] Each point in the graph shown in Figure 6 corresponds to a single determined patch parameter and multiple patches generated and planned based on that patch parameter. For example, the inference result for the point indicated by the circle in Figure 6 corresponds to the patch parameter and multiple patches shown in Figure 5F.

[0063] In the first embodiment, the planning function 113 determines an optimized patch parameter and a plurality of patches corresponding to that optimized patch parameter from among a plurality of patch parameters based on the inference accuracy and inference time of the inference result.

[0064] Specifically, the planning function 113 pre-sets thresholds for inference accuracy and inference time. For example, the planning function 113 sets the threshold for inference accuracy to 97.2%. As a result, the planning function 113 selects the inference result with the shortest inference time among those inference results where the inference accuracy exceeds the threshold of 97.2% in the curve shown in Figure 6, i.e., it selects the inference result indicated by the circle in the figure. Based on this, the planning function 113 determines the patch parameter corresponding to this inference result as the optimized patch parameter, and determines multiple patches corresponding to this optimized patch parameter. The multiple patches determined by the planning function 113 are shown in the right-hand figure of Figure 6.

[0065] The planning function 113 may further include an editing function 113D, which is an example of an "editing step".

[0066] Refer to Figure 7 to explain the user interface UI for editing patches using the editing function 113D. Figure 7 is a schematic diagram showing the user interface UI for editing patches using the editing function 113D.

[0067] As shown in Figure 7, the editing function 113D displays the inference results and multiple patches determined by the planning unit in the user interface UI, making them editable by the user. The editing function 113D then adjusts the multiple patches according to the user's edits.

[0068] For example, if a user selects the "Edit" button in the user interface UI and then moves the middle patch of the three patches a certain distance to the bottom right using an operation such as "Drag," the edited three patches shown in the lower part of Figure 7 are obtained. Note that user editing is not limited to moving patches; users may also change the length of any of the patches or add or delete a certain number of patches.

[0069] As a result, the planning function 113 determines the optimization patch parameters and the multiple patches corresponding to those optimization patch parameters.

[0070] The inference function 114 performs inference tasks by performing inference on image data using a deep learning model. Specifically, the inference function 114 performs inference tasks using a deep learning model and obtains inference results based on the optimization patch parameters determined by the planning function 113 and a plurality of patches corresponding to those optimization patch parameters. The inference tasks are, for example, image processing tasks such as image segmentation, classification, and detection based on a deep learning model. The inference function 114 may also perform inference tasks by combining the processing flow executed by the image processing device 1 with techniques such as model pruning and model compression.

[0071] The output function 115 outputs various types of data and information. For example, the output function 115 outputs the inference results from the inference function 114.

[0072] (Image processing method) Referring to Figure 8, the processing flow of the image processing method according to the first embodiment will be described. Figure 8 is a flowchart of the image processing method according to the first embodiment.

[0073] In step S11, the acquisition function 111 acquires image data. Next, the process proceeds to step S12.

[0074] In step S12, the division function 41 of the image data preprocessing device 4 divides the image data acquired by the acquisition function 111 in step S11 into regions of interest and regions of non-interest. This divides the region of interest in the image data. Next, the process proceeds to step S13.

[0075] In step S13, the setting function 112 sets multiple patch parameters according to the inference task that the deep learning model performs on the image data. Specifically, the setting function 112 sets recommended values ​​for multiple patch parameters based on the parameters associated with the inference task in the history database DB stored in the medical data storage device 5. The setting function 112 may also set recommended values ​​for multiple patch parameters based on user input. The setting function 112 may generate a corresponding input image for each patch parameter that has a fixed interval within a preset range of patch parameters, obtain the inference result for each input image, and determine the recommended value for the patch parameter based on the inference result. In this way, recommended values ​​for multiple patch parameters are set according to the inference task. Next, the process proceeds to step S14.

[0076] In step S14, the planning function 113 generates multiple patches for each patch parameter set by the setting function 112 in step S13, obtains inference results based on the generated multiple patches, and determines the optimal patch parameter and the multiple patches corresponding to that optimal patch parameter from among the multiple patch parameters based on the inference results. A flowchart of the steps executed by the planning function 113 will be described later with reference to Figure 9. As a result, the optimal patch parameter and the multiple patches corresponding to that optimal patch parameter are determined based on the inference results. Next, the process proceeds to step S15.

[0077] In step S15, the inference function 114 performs an inference task using a deep learning model and obtains inference results based on the optimization patch parameters determined by the planning function 113 in step S14 and the multiple patches corresponding to those optimization patch parameters. This provides inference results based on the multiple patches corresponding to the optimization patch parameters. Next, the process proceeds to step S16.

[0078] In step S16, the output function 115 outputs the inference result from the inference function 114 in step S15. This completes the processing flow.

[0079] Referring to Figure 9, the processing flow for patch planning according to the first embodiment will be described. Figure 9 is a flowchart of patch planning according to the first embodiment.

[0080] In step S141, the patch generation function 113A selects one patch parameter from among the reference values ​​of multiple patch parameters set by the setting function 112, and generates a patch based on the selected patch parameter. Next, the process proceeds to step S142.

[0081] In step S142, the patch generation function 113A places the generated patches at non-empty corner point locations in the region of interest. Next, the process proceeds to step S143.

[0082] In step S143, the patch adjustment function 113B adjusts the patch generated by the patch generation function 113A. The patch adjustment function 113B sets the length of the patch generated by the patch generation function 113A based on the patch parameters, adjusts the length of the patch according to the length of the divided region of interest, and moves the position of the patch after length adjustment so that the range of the region of interest covered by the patch increases and the range of the region of non-interest covered decreases. In this way, the length and position of the patch generated by the patch generation function 113A are adjusted. Next, the process proceeds to step S144.

[0083] In step S144, the region of interest update function 113C updates the region of interest by subtracting the patch adjusted by the patch adjustment function 113B from the divided region of interest of the image data. Next, the process proceeds to step S145.

[0084] In step S145, the planning function 113 determines whether the volume of the new region of interest updated by the region of interest update function 113C is zero. If it is determined that the volume of the updated region of interest is zero, the process proceeds to step S146. If it is determined that the volume of the updated region of interest is not zero, the planning function 113 returns the processing flow to step S142 and uses the patch generation function 113A to place patches at other non-empty corner point locations of the updated region of interest. The cycle of steps S142 to S144 is executed until the volume of the updated region of interest becomes zero. In this way, the planning function 113 generates and plans multiple patches for each patch parameter.

[0085] In step S146, the planning function 113 determines whether user editing is required for the multiple planned patches. If user editing is required, the process proceeds to step S147. If user editing is not required, the process proceeds to step S148.

[0086] In step S147, the editing function 113D displays the multiple patches planned by the planning function 113 in the user interface UI, making them editable by the user. The editing function 113D then adjusts the multiple patches according to the user's edits. Next, the process proceeds to step S148.

[0087] In step S148, the planning function 113 obtains inference results based on the planned multiple patches, and determines the optimized patch parameters and the multiple patches corresponding to those optimized patch parameters from among the multiple patch parameters set by the setting function 112 based on the inference results. Then the patch planning process flow is completed, and the process proceeds to step S15.

[0088] The planning function 113 completes the processing flow for planning the patches and proceeds to step S15, described later, where the inference function 114 executes an inference task based on the planned patches.

[0089] As a result, the image processing apparatus 1 and image processing method according to the first embodiment can dynamically adjust patches to accelerate the inference process of a deep learning model and can be applied to different inference tasks.

[0090] Furthermore, according to the image processing apparatus 1 and image processing method of the first embodiment, the inference process of the deep learning model can be dynamically adjusted as needed, and the deep learning model can be trained based on the results of user selection and editing.

[0091] (Second embodiment) The image processing apparatus 1A according to the second embodiment will be described below with reference to Figures 10 and 11. Figure 10 is a block diagram showing an example configuration of the image processing apparatus 1A according to the second embodiment. Figure 11 is a schematic diagram showing an example of clinical data and important information. In the following description, only the differences between the second embodiment and the first embodiment will be described, and the common points will not be explained.

[0092] The difference between the second embodiment and the first embodiment is that the processing circuit 11A of the image processing device 1A further includes a clinical data analysis function 116.

[0093] In other words, as shown in Figure 10, the processing circuit 11A includes an acquisition function 111, a setting function 112, a planning function 113, an inference function 114, an output function 115, and a clinical data analysis function 116. Here, the clinical data analysis function 116 is an example of a "clinical data analysis step".

[0094] The clinical data analysis function 116 extracts important information from the subject's clinical data regarding the inference task performed by the deep learning model. Then, the planning function 113 determines the optimization parameters and multiple patches corresponding to those optimization parameters based on the important information in the subject's clinical data extracted by the clinical data analysis function 116.

[0095] Specifically, the clinical data analysis function 116 acquires clinical data of the subject from the medical data storage device 5, for example. The clinical data of the subject includes information such as the subject's personal information, the purpose of the examination, the department of the examination, the examination site, and the imaging technique, as shown in the upper part of Figure 11. The clinical data analysis function 116 extracts important information from the clinical data of the subject regarding the inference task performed by the deep learning model, and extracts from this clinical data, for example, as shown in the lower part of Figure 11, that the purpose of the examination is preoperative evaluation for lumbar spine surgery and the examination site is spinal artery CTA as important information.

[0096] Next, the planning function 113 determines the inference objectives in the inference task based on the key information extracted by the clinical data analysis function 116, and then determines the algorithm and accuracy requirements for each inference objective according to the current clinical course. Based on these accuracy requirements, the planning function 113 determines the optimization patch parameters.

[0097] For example, in general thoracolumbar spine surgery, evaluation of the spinal arteries, especially the AKA (Adamkiewicz artery), a branch of the spinal artery, is extremely important. Damage to this AKA branch during surgery can lead to spinal ischemia. To evaluate this AKA branch, it is necessary to perform bone removal processing on the image data. However, since the diameter of the spinal artery is only 0.1 mm to 1.2 mm, the accuracy of bone detection in bone removal processing is required to be extremely high.

[0098] As a result, the planning function 113 determines the optimized patch parameters and multiple patches corresponding to those optimized patch parameters based on the important information extracted by the clinical data analysis function 116.

[0099] The clinical data and important information presented above are merely examples, and the information included is not limited to these examples, provided that optimization parameters can be determined based on this important information.

[0100] As a result, according to the image processing method and image processing apparatus 1A of the second embodiment, it is possible to determine optimized patch parameters based on important information in clinical data, generate a number of corresponding patches, optimize the patch parameters based on clinical data, and further accelerate the inference process of the deep learning model.

[0101] (Third embodiment) The image processing apparatus 1B according to the third embodiment will now be described with reference to Figure 12. Figure 12 is a block diagram showing an example configuration of the image processing apparatus 1B according to the third embodiment. In the following description, only the differences between the third embodiment and the first embodiment will be described, and the common points will not be explained.

[0102] The third embodiment differs from the first embodiment in that the processing circuit 11B of the image processing device 1B further includes a model update function 117.

[0103] In other words, as shown in Figure 12, the processing circuit 11B includes an acquisition function 111, a setting function 112, a planning function 113, an inference function 114, an output function 115, and a model update function 117. Here, the model update function 117 is an example of a "model update step".

[0104] The model update function 117 stores the optimization patch parameters determined by the planning function 113 and the user's edit results for multiple patches corresponding to those optimization patch parameters, and trains the deep learning model based on these edit results.

[0105] Refer to Figure 13 to explain the processing flow of model updates using the model update function 117. Figure 13 is a flowchart of model updates using the model update function 117.

[0106] As shown in Figure 13, in step S14, the planning function 113 determines the optimization patch parameters and a number of patches corresponding to those optimization patch parameters. Next, the process proceeds to step S301.

[0107] In step S301, the model update function 117 determines whether the optimized patch parameters determined by the planning function 113 and the multiple patches corresponding to those optimized patch parameters meet the clinical requirements. If the model update function 117 determines that the optimized patch parameters determined by the planning function 113 and the multiple patches corresponding to those optimized patch parameters meet the clinical requirements, the process proceeds to step S15, where the inference function 114 performs an inference task based on the multiple patches determined by the planning function 113 to perform inference.

[0108] If the model update function 117 determines that the optimized patch parameters determined by the planning function 113 and the multiple patches corresponding to those optimized patch parameters do not meet the clinical requirements, it proceeds to step S302.

[0109] In step S302, the model update function 117 presents the user with optimization patch parameters and corresponding patches that do not meet clinical requirements, and allows the user to edit them. The model update function 117 then stores the user's edited results in, for example, the medical data storage device 5. Next, the process proceeds to step S303.

[0110] In step S303, the model update function 117 trains a deep learning model based on the user's editing results stored in the medical data storage device 5 and updates the deep learning model.

[0111] In the flowchart of the third embodiment shown in Figure 13, steps S14 and S15 are the same as in the first embodiment, so their explanation will be omitted.

[0112] As a result, according to the image processing method and image processing apparatus 1B of the third embodiment, a deep learning model can be continuously trained and inference results optimized for different inference tasks can be multiplexed.

[0113] In the third embodiment, the processing circuit 11B of the image processing device 1B may further include the clinical data analysis function 116 of the second embodiment.

[0114] (Comparative example) The differences between the multiple patches planned by the comparative example and the multiple patches planned by the image processing method according to this embodiment will be explained below with reference to Figures 14A to 14C. Figures 14A to 14C are schematic diagrams illustrating the differences between the patch plan related to the comparative example and the patch plan according to the image processing method according to this embodiment.

[0115] Figures 14A and 14B show multiple patches planned by the comparative example, and Figure 14C shows multiple patches planned by the image processing method according to this embodiment. Here, the comparative example in Figure 14A plans the patches using the sliding-window method, while the comparative example in Figure 14B plans the patches using a method that combines the sliding-window method and the coarse-to-fine method (coarse-to-fine + sliding-window).

[0116] As shown in Figures 14A to 14C, the multiple patches planned using the sliding window method are all the same size and are uniformly distributed across the entire area of ​​the image data. Multiple patches planned using a combination of the sliding window method and the coarse-to-fine method show improved distribution and concentration of the patches near the region of interest in the image data compared to those planned using the sliding window method alone. However, the size of the multiple patches remains the same, and their distribution is still uniform.

[0117] Compared to the comparative example, the image processing method according to this embodiment dynamically generates appropriate patches by setting appropriate patch parameters for different inference tasks. Therefore, the size and position of the multiple patches are adapted to the region of interest in the inference task. In other words, the multiple patches are distributed with appropriate sizes and in appropriate positions that can cover the region of interest.

[0118] Therefore, inference based on appropriate patches can accelerate the inference process of deep learning models and can be applied to a variety of different inference tasks.

[0119] Furthermore, in the embodiments described above, the processing circuit is not limited to being implemented by a single processor, but may be composed of a combination of multiple independent processors, with each processor executing a program to realize each processing function. Also, each processing function of the processing circuit may be implemented by appropriately distributing or integrating it across one or more processing circuits. Moreover, each processing function of the processing circuit may be implemented by a mixture of hardware such as circuits and software. In addition, although an example in which the programs corresponding to each processing function are stored in a single memory circuit has been described here, the embodiments are not limited to this. For example, the programs corresponding to each processing function may be distributed and stored in multiple memory circuits, and the processing circuit may read and execute each program from each memory circuit.

[0120] Furthermore, while the embodiments described above illustrate examples where each functional unit in this specification is implemented by a processing circuit, the embodiments are not limited to these. For example, each functional unit in this specification may be implemented not only by the processing circuits described in the embodiments, but also by hardware alone, software alone, or a combination of hardware and software.

[0121] Furthermore, the term "processor" used in the above-described embodiments refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Here, instead of storing the computer program in a memory circuit, the computer program may be directly incorporated into the processor's circuit. In this case, the processor realizes its function by reading and executing the program incorporated into the circuit. Moreover, each processor in this embodiment is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor and realize its function.

[0122] Furthermore, the "computer program" used in the description of the embodiments described above includes a computer program executed by a processor, and this computer program is provided pre-installed in ROM (Read Only Memory) or a memory circuit. This computer program may also be provided as a file in an installable or executable format on a computer-readable, non-transient storage medium such as a CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), or DVD (Digital Versatile Disk). Alternatively, this computer program may be stored on a computer connected to a network such as the Internet and provided or distributed by downloading it via the network. For example, this computer program consists of modules containing the processing functions described above. In actual hardware, the CPU reads the computer program from a storage medium such as ROM and executes it, loading each module onto the main memory and generating it in the main memory.

[0123] Furthermore, in the embodiments described above, each component of each illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific form of distribution or integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads and usage conditions. Moreover, each processing function performed by each device can be implemented, in whole or in any part, by a CPU and a program that is analyzed and executed by the CPU, or by hardware using wired logic.

[0124] Furthermore, among the processes described in the embodiments described above, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified.

[0125] The various types of data discussed in this specification are typically digital data.

[0126] Although several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0127] 1 Image Processing Device 112 Settings Section 113 Planning Department 114 Reasoning Department

Claims

1. An image processing method that performs an inference task on image data using a pre-trained deep learning model, A setting step of setting multiple patch parameters according to the inference task, A planning step involves generating multiple patches for each of the patch parameters, obtaining inference results based on the generated multiple patches, and determining an optimized patch parameter and multiple patches corresponding to the optimized patch parameter from among the multiple patch parameters based on the inference results. An inference step of obtaining an inference result by performing the inference task using the deep learning model based on a plurality of patches corresponding to the determined optimization patch parameters, Image processing methods including [specific details omitted].

2. The aforementioned planning step further includes: A patch generation step that generates the patch for each of the patch parameters, A patch adjustment step for adjusting the aforementioned patch, The process includes a region of interest update step, which involves subtracting the patch from the region of interest of the image data to update a new region of interest, Multiple patches are generated in the new region of interest by performing the patch generation step, the patch adjustment step, and the region of interest update step until the volume of the updated new region of interest becomes zero. The image processing method according to claim 1.

3. In the aforementioned patch generation step, The patch is generated at the corner point of the region of interest, In the aforementioned patch adjustment step, The length of the patch is set based on the patch parameters, The length of the patch is adjusted according to the length of the region of interest, The adjusted position of the patch is moved such that the area of ​​the region of interest covered by the patch increases, and the area of ​​the region of non-interest covered by the patch decreases. The image processing method according to claim 2.

4. In the setting step, recommended values ​​for a plurality of patch parameters are set based on the parameters associated with the inference task in the history database. In the planning step, generate multiple patches for each patch parameter in the recommended values. The image processing method according to claim 1.

5. In the setting step described above, recommended values ​​for multiple patch parameters are set based on user input. In the planning step, generate multiple patches for each patch parameter in the recommended values. The image processing method according to claim 1.

6. In the setting step, within the preset range of the patch parameters, a corresponding input image is generated for each patch parameter having a fixed interval, the inference result for each input image is obtained, and a recommended value for the patch parameter is determined based on the inference result. In the planning step, generate multiple patches for each patch parameter in the recommended values. The image processing method according to claim 1.

7. The multiple patch parameters include base patch length, patch length ratio, base resolution, and resolution ratio. The image processing method according to any one of claims 4 to 6.

8. In the planning step, the optimization patch parameters and a plurality of patches corresponding to the optimization patch parameters are determined based on the inference accuracy and inference time. The image processing method according to claim 1.

9. The aforementioned planning step further includes an editing step, In the editing step, the inference results and the determined multiple patches are displayed on the user interface in a user-editable format, and the multiple patches are adjusted according to the user's edits. The image processing method according to claim 1.

10. The acquisition step of acquiring the aforementioned image data, A segmentation step of dividing the aforementioned image data and extracting the region of interest, An output step that outputs the aforementioned inference result, The image processing method according to claim 1, further comprising:

11. The method further includes a clinical data analysis step of extracting important information related to the inference task from the clinical data of the subject, In the planning step, the optimization patch parameters and a plurality of patches corresponding to the optimization patch parameters are determined based on the important information. The image processing method according to claim 1.

12. The process further includes a model update step that stores the user's editing results and trains the deep learning model based on those editing results. The image processing method according to claim 1.

13. An image processing device that performs an inference task on image data using a trained deep learning model, A setting unit that sets multiple patch parameters according to the aforementioned inference task, A planning unit that generates multiple patches for each of the patch parameters, obtains inference results based on the multiple generated patches, and determines an optimized patch parameter and multiple patches corresponding to the optimized patch parameter from among the multiple patch parameters based on the inference results, An inference unit that performs the inference task using the deep learning model based on a plurality of patches corresponding to the determined optimization patch parameters and obtains the inference result, An image processing device equipped with the following features.

14. A computer program that, when executed by a processor, performs an inference task on image data using a trained deep learning model, When the aforementioned computer program is executed, A setting step of setting multiple patch parameters according to the inference task, A planning step involves generating multiple patches for each of the patch parameters, obtaining inference results based on the generated multiple patches, and determining an optimized patch parameter and multiple patches corresponding to the optimized patch parameter from among the multiple patch parameters based on the inference results. An inference step of obtaining an inference result by performing the inference task using the deep learning model based on a plurality of patches corresponding to the determined optimization patch parameters, A computer program that includes this.

Images