Image processing method, image processing apparatus, and computer program product
By dynamically adjusting sliding window parameters and generating appropriate sliding windows, the inference process of deep learning models is optimized, solving the problems of computational redundancy and long inference time in existing technologies, and realizing fast and efficient image processing on devices with limited resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2026-03-31
AI Technical Summary
Existing deep learning models struggle to achieve optimal inference results when faced with different tasks or objectives, resulting in computational redundancy and long inference times. This is especially true for achieving fast image processing on embedded or mobile platforms with limited computing resources.
By dynamically adjusting sliding window parameters, appropriate sliding windows are generated based on actual needs and inference results, thus optimizing the inference process of deep learning models. This includes setting multiple sliding window parameters, planning sliding window generation and adjustment, updating regions of interest, and optimizing sliding window generation using historical databases and user input.
It accelerates the inference process of deep learning models, improves inference accuracy, and is applicable to various inference tasks, meeting the needs of rapid image processing in clinical settings.
Smart Images
Figure CN121767375A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to image processing methods, image processing apparatus, and computer program products. Background Technology
[0002] Deep learning is an algorithm based on neural networks that learns representations of data. The goal of representation learning is to obtain better representations of data and build better deep learning models, thereby learning such representations on large-scale unlabeled data. Deep learning has been particularly well-received in image processing tasks, achieving excellent results. In image processing tasks, deep learning can be used for image segmentation, classification, and detection.
[0003] However, existing deep learning systems face various challenges. For example, deep learning models typically require a large number of parameters and have complex computational graphs. When performing segmentation tasks on high-resolution 3D images, they require significant computational resources, leading to increased inference time. Furthermore, in clinical settings, diagnostic and treatment decisions often require rapid image processing results, placing demands on the response speed of deep learning models. Additionally, hardware devices such as embedded or mobile platforms in medical equipment typically have limited computational resources. For instance, running complex deep learning-based segmentation models on mobile medical devices or handheld ultrasound equipment is difficult.
[0004] To address the aforementioned problems, the following existing technologies are proposed.
[0005] In a prior art, a sliding-window method is used to improve deep learning models. This method divides the complete image into fixed-size patches, allowing the deep learning model to perform inference within each patch and then fusing the inference results from each patch into a final inference result. This approach avoids the problem of insufficient computational resources required for inference on the complete image.
[0006] Another existing technique improves deep learning models using a coarse-to-fine approach. This method first uses a coarse model to locate key target regions, then uses a fine model to perform inference within those regions. This reduces processing in the background areas, thereby increasing inference speed.
[0007] In addition, there are techniques called model compression, which reduces the computational requirements of deep learning models by decreasing the number of parameters, pruning, quantizing, or using lightweight model architectures. There are also techniques that distill complex deep learning models into simplified models, thereby reducing computational complexity and improving inference speed while maintaining the performance of deep learning models.
[0008] Furthermore, there is a technique that combines the sliding window method with a coarse-to-fine approach (Coarse-to-fine + Sliding-Window), which uses pre-defined sliding window parameters such as aspect ratio and length to generate the same sliding window for different tasks or objectives. Therefore, the problem with this technique is that it is difficult to obtain optimal inference results when faced with different requirements, shapes, and volumes of tasks or objectives, and it leads to computational redundancy and longer inference times. Summary of the Invention
[0009] Problems in the prior art
[0010] In the aforementioned prior art, the same sliding window is usually generated for different tasks or targets using pre-set aspect ratio, length and other sliding window parameters. Therefore, when faced with different requirements, shapes and volumes of tasks or targets, it is difficult to obtain the optimal reasoning results, and it will lead to computational redundancy and long reasoning time.
[0011] Means for solving technical problems
[0012] The present invention was made in view of the above-mentioned technical problems, and aims to provide an image processing method, an image processing device, and a computer program product that can dynamically adjust the sliding window to accelerate the inference process of deep learning models and is applicable to different inference tasks.
[0013] Furthermore, the image processing method, image processing apparatus, and computer program product according to the present invention can dynamically adjust the inference process of the deep learning model according to actual needs, and can train the deep learning model based on the user's selection and editing results.
[0014] Furthermore, the image processing method, image processing apparatus, and computer program product according to the present invention enable deep learning models to continuously learn and reuse optimized inference results for different inference tasks.
[0015] According to the image processing method of the present invention, an inference task is performed on image data using a trained deep learning model, characterized by comprising: a setting step, wherein multiple sliding window parameters are set in accordance with the inference task; a planning step, wherein multiple sliding windows are generated for each of the sliding window parameters, and inference results are obtained based on the generated multiple sliding windows, and optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters are determined from the multiple sliding window parameters according to the inference results; and an inference step, wherein the inference task is performed using the deep learning model based on the determined multiple sliding windows corresponding to the optimized sliding window parameters to obtain inference results.
[0016] Therefore, compared to existing technologies that use the same pre-set sliding window parameters to generate the same sliding window for different inference tasks, the image processing method of this invention dynamically generates appropriate sliding windows by setting appropriate sliding window parameters for different inference tasks. Thus, inference based on appropriate sliding windows can accelerate the inference process of deep learning models and is applicable to a wide variety of inference tasks.
[0017] Furthermore, the image processing method of the present invention dynamically generates an appropriate sliding window by determining optimized sliding window parameters based on the inference results. Therefore, it can further accelerate the inference process of deep learning models and improve inference accuracy.
[0018] According to the image processing method of the present invention, the planning step further includes: a sliding window generation step, generating the sliding window for each sliding window parameter; a sliding window adjustment step, adjusting the sliding window; and a region of interest (ROI) update step, subtracting the sliding window from the ROI of the image data to update a new ROI, performing the sliding window generation step, the sliding window adjustment step, and the ROI update step in the new ROI until the volume of the updated new ROI is zero, thereby generating multiple sliding windows.
[0019] According to the image processing method of the present invention, in the sliding window generation step, the sliding window is generated at a non-empty corner position of the region of interest; in the sliding window adjustment step, the length of the sliding window is set according to the sliding window parameters, and the length of the sliding window is adjusted accordingly with the length of the region of interest, so that the position of the adjusted sliding window is moved in such a way that the range of the region of interest covered by the sliding window increases and the range of the non-region of interest covered decreases.
[0020] Therefore, compared to existing technologies that use the same pre-set sliding window parameters to generate the same sliding window for different inference tasks, the image processing method of this invention generates multiple sliding windows based on the region of interest (ROI) and adjusts the length and position of these multiple sliding windows according to the ROI. Thus, it is possible to obtain multiple sliding windows with appropriate positions, lengths, and numbers while ensuring coverage of the entire ROI, thereby enabling dynamic adjustment of the sliding windows to accelerate the inference process of deep learning models.
[0021] According to the image processing method of the present invention, in the setting step, recommended values for multiple sliding window parameters are set based on parameters corresponding to the inference task in the historical database, and in the planning step, multiple sliding windows are generated according to each of the recommended sliding window parameters.
[0022] According to the image processing method of the present invention, in the setting step, recommended values for a plurality of sliding window parameters are set based on user input, and in the planning step, a plurality of sliding windows are generated according to each of the recommended sliding window parameters.
[0023] According to the image processing method of the present invention, in the setting step, within a pre-set range of the sliding window parameters, a corresponding input image is generated for each sliding window parameter having a certain interval, an inference result of each input image is obtained, a recommended value of the sliding window parameter is determined based on the inference result, and in the planning step, multiple sliding windows are generated according to each sliding window parameter in the recommended value.
[0024] According to the image processing method of the present invention, the plurality of sliding window parameters include a basic sliding window length, a sliding window length ratio, a basic resolution, and a resolution ratio.
[0025] Therefore, compared to existing technologies that use the same pre-set sliding window parameters for different inference tasks, the image processing method of the present invention can set appropriate sliding window parameters for different inference tasks. Thus, it provides a universal image processing method applicable to various inference tasks.
[0026] Furthermore, the image processing method of the present invention can determine optimized sliding window parameters based on the inference results. Therefore, the sliding window can be dynamically adjusted to accelerate the inference process of the deep learning model.
[0027] According to the image processing method of the present invention, in the planning step, the optimized sliding window parameters and a plurality of sliding windows corresponding to the optimized sliding window parameters are determined based on the inference accuracy and the inference time.
[0028] Therefore, the image processing method of the present invention can determine and optimize sliding window parameters and generate multiple sliding windows based on inference accuracy and inference time. Thus, it can further dynamically adjust the sliding windows to accelerate the inference process of deep learning models.
[0029] According to the image processing method of the present invention, the planning step further includes an interactive editing step, in which the inference result and the determined plurality of sliding windows are displayed in a user interface in a manner that the user can edit, and the plurality of sliding windows are adjusted according to the user's editing.
[0030] Therefore, the image processing method of the present invention allows users to select multiple sliding windows corresponding to the required inference accuracy and inference speed, and allows users to directly edit and adjust multiple sliding windows. Thus, the inference process of the deep learning model can be dynamically adjusted according to actual needs, and the deep learning model can be trained based on the user's selections and editing results, thereby further accelerating the inference process of the deep learning model.
[0031] The image processing method according to the present invention is characterized by further comprising: an acquisition step, acquiring the image data; a segmentation step, segmenting the region of interest in the image data; and an output step, outputting the inference result.
[0032] The image processing method according to the present invention is characterized by further including a clinical data analysis step, wherein key information related to the reasoning processing task is extracted from the clinical data of the subject, and in the planning step, the optimized sliding window parameters and a plurality of sliding windows corresponding to the optimized sliding window parameters are determined based on the key information.
[0033] Therefore, compared to existing technologies that use the same pre-set sliding window parameters to generate the same sliding window for different inference tasks, the image processing method of this invention determines and optimizes the sliding window parameters based on key information in clinical data and generates multiple corresponding sliding windows. Thus, it can optimize the sliding window parameters based on clinical data, further accelerating the inference process of deep learning models.
[0034] The image processing method according to the present invention is characterized by further including a model update step, storing the user's editing results, and enabling the deep learning model to learn based on the editing results.
[0035] Therefore, the image processing method of the present invention can train a deep learning model based on user selections and editing results. This enables the deep learning model to continuously learn, reuse optimized inference results for different inference tasks, and further accelerates the inference process of the deep learning model.
[0036] According to the image processing apparatus of the present invention, an inference task is performed on image data using a trained deep learning model. The apparatus comprises: a setting unit that sets a plurality of sliding window parameters corresponding to the inference task; a planning unit that generates a plurality of sliding windows for each of the sliding window parameters, obtains inference results based on the generated plurality of sliding windows, and determines optimized sliding window parameters and a plurality of sliding windows corresponding to the optimized sliding window parameters from the plurality of sliding window parameters based on the inference results; and an inference unit that performs the inference task using the deep learning model based on the determined plurality of sliding windows corresponding to the optimized sliding window parameters to obtain an inference result.
[0037] The computer program product according to the present invention includes a computer program that, when executed by a processor, performs an inference task on image data using a trained deep learning model. The computer program, when executed, includes the following steps: a setting step, setting a plurality of sliding window parameters corresponding to the inference task; a planning step, generating a plurality of sliding windows for each of the sliding window parameters, obtaining inference results based on the generated plurality of sliding windows, determining optimized sliding window parameters and a plurality of sliding windows corresponding to the optimized sliding window parameters from the plurality of sliding window parameters based on the inference results; and an inference step, using the deep learning model to perform the inference task based on the determined plurality of sliding windows corresponding to the optimized sliding window parameters to obtain an inference result.
[0038] Therefore, compared to existing technologies that use the same pre-set sliding window parameters to generate the same sliding window for different inference tasks, the image processing apparatus and computer program product of the present invention dynamically generate appropriate sliding windows by setting appropriate sliding window parameters for different inference tasks. Thus, inference based on appropriate sliding windows can accelerate the inference process of deep learning models and is applicable to a wide variety of inference tasks.
[0039] Furthermore, the image processing apparatus and computer program product of the present invention dynamically generate an appropriate sliding window by determining optimized sliding window parameters based on the inference results. Therefore, the inference process of deep learning models can be further accelerated, and inference accuracy can be improved.
[0040] Invention Effects
[0041] The image processing method, image processing apparatus, and computer program product of the present invention can dynamically adjust the sliding window to accelerate the inference process of deep learning models and are applicable to different inference tasks.
[0042] Furthermore, the image processing method, image processing apparatus, and computer program product according to the present invention can dynamically adjust the inference process of the deep learning model according to actual needs, and can train the deep learning model based on the user's selection and editing results.
[0043] Furthermore, the image processing method, image processing apparatus, and computer program product according to the present invention enable deep learning models to continuously learn and reuse optimized inference results for different inference tasks. Attached Figure Description
[0044] Figure 1 This is a block diagram illustrating a configuration example of the image processing system according to the first embodiment.
[0045] Figure 2 This is a block diagram illustrating a configuration example of the image processing apparatus according to the first embodiment.
[0046] Figure 3 This is a schematic diagram representing the input image with different sliding window parameters.
[0047] Figure 4 It is a graph showing the relationship between the sliding window length ratio and the accuracy of the inference result.
[0048] Figure 5 This is a schematic diagram showing the layout of multiple sliding windows.
[0049] Figure 6 It is a graph showing the relationship between the time and accuracy of the reasoning result.
[0050] Figure 7 This is a schematic diagram representing the user interface for editing via a sliding window in the interactive editing section.
[0051] Figure 8 This is a flowchart illustrating the image processing method of the first embodiment.
[0052] Figure 9 This is a flowchart illustrating the planning of multiple sliding windows in the first embodiment.
[0053] Figure 10 This is a block diagram illustrating a configuration example of the image processing apparatus according to the second embodiment.
[0054] Figure 11 This is a diagram illustrating examples of clinical data and key information.
[0055] Figure 12 This is a block diagram illustrating a configuration example of the image processing apparatus according to the third embodiment.
[0056] Figure 13 This is a flowchart representing the updating of the model based on the model updating unit.
[0057] Figure 14 This is a schematic diagram illustrating the design of a sliding window in a comparative example.
[0058] Explanation of reference numerals in the attached figures
[0059] 1 Image processing device, 11 Processing circuit, 111 Acquisition unit, 112 Setting unit, 113 Planning unit, 113A Sliding window generation unit, 113B Sliding window adjustment unit, 113C Region of interest updating unit, 113D Interactive editing unit, 114 Inference unit, 115 Output unit, 12 Memory, 13 Communication IF, 2 Operation terminal, 3 Image data generation device, 4 Image data preprocessing device, 5 Medical data storage device, 1A Image processing device, 11A Processing circuit, 116 Clinical data analysis unit, 1B Image processing device, 11B Processing circuit, 117 Model updating unit. Detailed Implementation
[0060] Hereinafter, embodiments of the image processing method, image processing apparatus, and computer program product of the present invention will be described with reference to the accompanying drawings.
[0061] (First Embodiment)
[0062] Reference Figure 1 An example of the configuration of the image processing system 100 will be described. Figure 1 This is a block diagram illustrating a configuration example of the image processing system 100 according to the first embodiment.
[0063] The image processing system 100 includes an image processing device 1, an operating terminal 2, an image data generation device 3, an image data preprocessing device 4, and a medical data storage device 5. The devices are connected to each other via a shared signal transmission line, such as a bus or network, in a manner enabling mutual communication.
[0064] Image processing apparatus 1 is an apparatus that performs inference tasks on image data of medical images generated by image data generation apparatus 3 using a trained deep learning model. Image processing apparatus 1 may function as a server of image processing system 100, for example, or it may be a workstation capable of performing high-speed information processing.
[0065] The operating terminal 2 is a terminal operated by an operator (such as a doctor or technician). The operating terminal 2 functions as a client of the image processing system 100. The operating terminal 2 can be a computer, laptop, smartphone, tablet computer, wearable device, mobile terminal, or handheld terminal, etc.
[0066] The operating terminal 2 may also display the user interface (UI) described later in a manner that allows the user to operate, input, or edit.
[0067] Additionally, the operating terminal 2 may also have an input interface for receiving various operations from the user. The input interface converts these operations into electrical signals and transmits them to the processing circuitry of the operating terminal 2. The input interface of the operating terminal 2 can be a mouse, keyboard, buttons, panel switches, slider switches, trackball, operating panel, or touch panel.
[0068] Additionally, the operating terminal 2 may also have a display for showing various data or information. The display may be a liquid crystal display, a plasma display, an organic EL display, or an LED display. The display may also be a touch panel display that also functions as an input interface.
[0069] Image data generation device 3 is a device for generating image data (hereinafter referred to as "image data") for medical images. Image data generation device 3 may be a medical image diagnostic device (e.g., X-ray diagnostic device, X-ray CT device, MRI device, ultrasound diagnostic device, nuclear medicine examination device, etc.). Image data generation device 3 sends the generated image data to medical data storage device 5.
[0070] Image data preprocessing apparatus 4 is an apparatus for preprocessing image data. Image data preprocessing apparatus 4 includes a segmentation unit 41. Segmentation unit 41 distinguishes and segments regions of interest (ROI) and non-ROI regions in the image data. As a result of segmentation, for example, segmentation unit 41 segments skeletal regions into ROI regions and other regions, including blood vessels, into non-ROI regions. The positions of the segmented regions can be represented as positional information in the subject coordinate system of the image data. The segmentation performed by segmentation unit 41 can apply known image recognition techniques; for example, a thresholding algorithm can be used to segment regions with a CT value greater than 150 HU into ROI regions, or a coarse segmentation or detection model (e.g., the nnUNet Coarse model) can be used to segment the ROI regions. Image data preprocessing apparatus 4 sends the preprocessing results obtained by segmentation unit 41 to medical data storage apparatus 5.
[0071] The medical data storage device 5 is a device for storing medical data. For example, it stores image data, clinical data (described later), inference results based on deep learning models, and user-edited results. The medical data storage device 5 can be a storage medium (e.g., magnetic storage medium, electromagnetic storage medium, optical storage medium, semiconductor memory) or a drive device for reading and writing information between itself and the storage medium. The medical data storage device 5 stores image data sent from the image data generation device 3 and preprocessing results sent from the image data preprocessing device 4.
[0072] Reference Figure 2An example of the configuration of the image processing apparatus 1 will be described. Figure 2 This is a block diagram illustrating a configuration example of the image processing apparatus 1 according to the first embodiment.
[0073] The image processing apparatus 1 includes a processing circuit 11, a memory 12, and a communication IF 13. The components are communicatively connected to each other via a bus that serves as a shared signal transmission line.
[0074] The memory 12 is a device for storing various data or information. The memory 12 can be a storage medium that can be read by a processor (e.g., magnetic storage medium, electromagnetic storage medium, optical storage medium, semiconductor memory), or a drive device for reading and writing data or information between itself and the storage medium. The memory 12 stores computer programs that enable the processing circuit 11 to perform its various functions.
[0075] Communication IF13 is an interface for communicating various data or information with the various components included in the image processing system 100. Communication IF13 communicates various data or information with the operating terminal 2 or the medical data storage device 5.
[0076] The processing circuit 11 is a circuit that controls the overall operation of the image processing device 1. The processing circuit 11 includes at least one processor. A processor can be a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (such as a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA)). When the processor is a CPU, the CPU implements each function by reading and executing the computer programs stored in the memory 12. When the processor is an ASIC, each function is directly embedded as a logic circuit within the ASIC's circuitry. The processor can be configured as a single circuit or by combining multiple independent circuits. The processing circuit 11 implements the functions of the functional units described later.
[0077] In this embodiment, the processing circuit 11 includes an acquisition unit 111, a setting unit 110, a planning unit 113, an inference unit 114, and an output unit 115. Hereinafter, each functional unit of the processing circuit 11 will be described.
[0078] The acquisition unit 111 acquires various data or information. For example, the acquisition unit 111 acquires image data and obtains the region of interest and non-region of interest obtained by segmenting the image data by the segmentation unit 41. The acquisition unit 111 can acquire various data or information by accessing the medical data storage device 5.
[0079] The following is for reference Figures 3-4 The recommended values for the sliding window parameters based on the setting unit 112 will be explained. Figure 3 This is a schematic diagram representing the input image with different sliding window parameters. Figure 4 It is a graph showing the relationship between the sliding window length ratio and the accuracy of the inference result.
[0080] The setting unit 112 sets multiple sliding window parameters in accordance with the inference task performed by the deep learning model. Here, sliding window parameters include, for example, the base sliding window length, the sliding window length ratio, the base resolution, and the resolution ratio (Base patch size, Patchsize ratio, Base resolution, Resolution ratio). Alternatively, sliding window parameters can also be other parameters that determine the position, length, and resolution of the sliding window.
[0081] Furthermore, the setting unit 112 sets recommended values for each of the multiple sliding window parameters. Here, when the multiple sliding window parameters include, for example, a basic sliding window length, a sliding window length ratio, a basic resolution, and a resolution ratio, the recommended values for each of the multiple sliding window parameters are, for example, a basic sliding window length of (128, 128, 128), a sliding window length ratio of 0.9 to 1.35, a basic resolution of (1.0, 1.0, 1.0), and a resolution ratio of 0.9 to 1.3. In the first embodiment, the example given is that the recommended values for the sliding window parameters are a range of values for the sliding window parameters; however, the recommended values for the sliding window parameters can also be a specific value of the sliding window parameters.
[0082] Specifically, the setting unit 112 sets recommended values for multiple sliding window parameters based on parameters corresponding to the inference task performed by the deep learning model, stored in the historical database DB in the medical data storage device 5. For example, the setting unit 112 sets recommended values for each of the multiple sliding window parameters based on parameters of organ segmentation data included in the historical database DB stored in the medical data storage device 5. Here, the organ segmentation data is the inference result of a segmentation inference task performed by the deep learning model, and the inference task performed by the deep learning model can also be a tumor detection task, a lung lesion classification task, etc.
[0083] Alternatively, the setting unit 112 may set recommended values for multiple sliding window parameters based on user input to the operation terminal 2. For example, the user may specify multiple sliding window parameters, such as base sliding window length, sliding window length ratio, base resolution, and resolution ratio, in the operation terminal 2, and input their recommended values through the operation terminal 2. The setting unit 112 then sets the recommended values for the multiple sliding window parameters based on the user's input.
[0084] Alternatively, the setting unit 112 generates a corresponding input image for each sliding window parameter with a certain interval within a preset range of sliding window parameters, obtains the inference result of each input image, and determines the recommended value of the sliding window parameter based on the inference result.
[0085] Specifically, let's take determining the recommended value for the sliding window length ratio as an example. For instance, assuming the base sliding window length = (128, 128, 128), within a pre-set range of 0.6 to 1.6 for the sliding window length ratio, a series of corresponding input images are generated for each sliding window length ratio with intervals of 0.05, and the inference result for each input image is obtained. In this case, the length of the input image = base sliding window length * sliding window length ratio. When the sliding window length ratio = 1.0, the length of the input image = (128, 128, 128), and the accuracy of the inference result is 0.972. When the sliding window length ratio = 1.25, the length of the input image = (160, 160, 160), and the accuracy of the inference result is 0.971. Therefore, we can obtain... Figure 3 The image shown represents a series of corresponding input images generated for each sliding window parameter with a certain interval within a pre-defined range.
[0086] In addition, we can also obtain such as Figure 4 The sliding window parameters shown include the relationship between the sliding window length ratio and the accuracy of the inference results. According to... Figure 4 The relationships shown indicate that the range of sliding window length ratios that have a minimal impact on the accuracy of the inference results is 0.9 to 1.35. Therefore, the setting unit 112 determines the recommended values for the sliding window length ratio as 0.9 to 1.35. Similarly, the setting unit 112 determines recommended values for the base sliding window length, base resolution, and resolution ratio.
[0087] In addition, here, the preset range of the sliding window length ratio is determined to be 0.6 to 1.6 based on prior experience. However, the preset range of the sliding window parameters can also be determined by other methods. A certain interval of the sliding window length ratio is an interval of 0.05, but it can also be other intervals. The "preset range" and "certain interval" of the sliding window parameters are not limited, as long as they can be used to determine the recommended value of the sliding window parameters.
[0088] Reference Figure 5 The planning of the sliding window based on Planning Department 113 is explained. Figure 5This is a schematic diagram showing the layout of multiple sliding windows.
[0089] The planning unit 113 determines the sliding window parameters and plans the sliding windows. Specifically, the planning unit 113 generates multiple sliding windows for each sliding window parameter. The planning unit 113 includes a sliding window generation unit 113A, a sliding window adjustment unit 113B, and a region of interest updating unit 113C.
[0090] The sliding window generation unit 113A generates a sliding window for each sliding window parameter.
[0091] Specifically, such as Figure 5 As shown in Figure A, the sliding window generation unit 113A generates the sliding window ( Figure 5 The dashed bounding box in the image is generated from the region of interest output by the image data preprocessing unit 4. Figure 5 The non-empty corner position of the diagonal region in the middle. Figure 5 (The location indicated by the circle in A). Here, a "non-empty corner point" of the region of interest refers to any one of the eight endpoints of the rectangle circumscribed by the region of interest. The sliding window generation unit 113 obtains these eight endpoints and generates a sliding window at any one of them. Here, the sliding window generation unit 113, for example, selects the endpoint with the smallest coordinate value in a mutually orthogonal XYZ coordinate system to generate the sliding window.
[0092] Next, the sliding window adjustment unit 113B adjusts the sliding window.
[0093] Specifically, such as Figure 5 As shown in Figure B, the sliding window adjustment unit 113B sets the length of the sliding window according to the sliding window parameters. For ease of explanation, an example is given with a base resolution of (1.0, 1.0, 1.0), a resolution ratio of 1.25, and a base sliding window length * sliding window ratio of (128, 128, 128). To reduce the number of sliding windows and accelerate the inference process of the deep learning model, the sliding window adjustment unit 113B adjusts the length of the sliding window accordingly to the length of the region of interest (ROI). In other words, the sliding window adjustment unit 113B optimizes the sliding window length based on the length of the ROI. Here, let the lengths on the mutually orthogonal XYZ axes of the ROI be (Lx, Ly, Lz), and the lengths (x, y, z) of the sliding window on the XYZ axes be adjusted based on the following mathematical formula 1.
[0094] Mathematical formula 1:
[0095] x:int(Lx / ceil(Lx / (base patch size*base resolution*resolution ratio)))
[0096] y:int(Ly / ceil(Ly / (base patch size*base resolution*resolution ratio)))
[0097] z:int(Lx / ceil(Lz / (base patch size*base resolution*resolution ratio)))
[0098] …(1)
[0099] Here, `int` represents rounding down, and `ceil` represents rounding up. Therefore, the length of the sliding window is adjusted to (110, 96, 148). This allows the length of the sliding window to adapt to the length of the region of interest.
[0100] like Figure 5 As shown in Figure C, the sliding window adjustment unit 113B moves the adjusted position of the sliding window in a manner that increases the range of the region of interest covered by the sliding window and decreases the range of the non-region of interest covered. Here, firstly, it is ensured that the range of the region of interest covered by the sliding window from the beginning does not decrease; secondly, the position of the sliding window is adjusted so that the sliding window covers more of the region of interest. Specifically, in Figure 5 In B, the upper left corner of the sliding window covers an unnecessary area of non-interest. Therefore, in Figure 5 In option C, moving the sliding window to the lower right increases the area of interest it covers while decreasing the area of non-interest. Alternatively, moving the left and top edges of the sliding window to the lower right decreases its length and reduces the area of non-interest it covers.
[0101] Next, the region of interest update unit 113C updates the new region of interest by subtracting a sliding window from the region of interest in the image data.
[0102] Specifically, such as Figure 5 As shown in D, the region of interest update unit 113C starts from... Figure 5 In C, the area represented by the diagonal lines is the region of interest. Subtracting the dashed box (representing a sliding window) updates the region of interest. Figure 5 The slanted area in D represents the new region of interest.
[0103] Next, as Figure 5 E, Figure 5As shown in F, the planning unit 113 performs the functions of the sliding window generation unit 113A, sliding window adjustment unit 113B, and region of interest update unit 113C described above in the updated new region of interest until the volume of the updated new region of interest is zero, thereby generating multiple sliding windows. That is, the planning unit 113 cyclically performs the steps of generating sliding windows through the sliding window generation unit 113A, adjusting sliding windows through the sliding window adjustment unit 113B, and updating the new region of interest through the region of interest update unit 113C in the updated new region of interest until the generated sliding windows cover the entire region of interest and the volume of the updated new region of interest is zero.
[0104] like Figure 5 D~ Figure 5 As shown in F, in the first loop, Planning Department 113 generates... Figure 5 The dashed box in D represents the sliding window, which is further generated by Planning Department 113 in the second loop. Figure 5 The dashed box on the right side of E represents the sliding window, which is further generated by Planning Department 113 in the third loop. Figure 5 The dashed box in the middle of F represents a sliding window.
[0105] It should be noted that in subsequent loops, the sliding window generation unit 113A selects the endpoints of the rectangle tangent to the region of interest that are different from the endpoints used in previous loops to generate the sliding window.
[0106] Furthermore, the planning unit 113 generates multiple sliding windows for each sliding window parameter among the recommended values of the sliding window parameters set by the setting unit 112. Thus, for each sliding window parameter, the planning unit 113 generates a corresponding... Figure 5 The multiple sliding windows shown.
[0107] In addition, the sliding window parameters, sliding window length and position, number of cycles, number of sliding windows, etc. mentioned above are all examples and are not limited to the values mentioned in this manual.
[0108] Reference Figure 6 The curves representing the relationship between inference time (seconds) and accuracy (%) are explained. Figure 6 It is a graph showing the relationship between the time and accuracy of the reasoning result.
[0109] Next, the planning department 113 obtains the inference results based on the generated multiple sliding windows, and determines the optimized sliding window parameters and the multiple sliding windows corresponding to the optimized sliding window parameters from the multiple sliding window parameters according to the inference results.
[0110] Specifically, by generating multiple sliding windows for each sliding window parameter based on the planning unit 113, a series of sliding windows can be obtained under each sliding window parameter. The planning unit 113 obtains the inference results based on the multiple sliding windows. Here, the inference results include the inference time and accuracy based on the multiple sliding windows. Therefore, it is possible to obtain... Figure 6 The figure shown is a trade-off curve representing the relationship between time and accuracy of the inference results.
[0111] Figure 6 The inference result for each point in the graph shown corresponds to a specific sliding window parameter and multiple sliding windows generated and planned based on that parameter. For example, Figure 6 The reasoning result of the points represented by the circles in the diagram corresponds to... Figure 5 The parameters of the sliding window and multiple sliding windows are shown.
[0112] In addition, in the first embodiment, the planning unit 113 determines the optimized sliding window parameter and the multiple sliding windows corresponding to the optimized sliding window parameter from multiple sliding window parameters based on the reasoning accuracy and reasoning time of the reasoning result.
[0113] Specifically, the planning department 113 pre-sets thresholds for inference accuracy and inference time. For example, the planning department 113 sets the threshold for inference accuracy to 97.2%. Therefore, the planning department 113... Figure 6 In the curves shown, among the inference results with an inference accuracy exceeding the threshold of 97.2%, the inference result with the shortest inference time is selected, i.e., the inference result shown by the circle in the figure is selected. Therefore, the planning unit 113 determines the sliding window parameters corresponding to this inference result as the optimized sliding window parameters, and determines multiple sliding windows corresponding to these optimized sliding window parameters. The multiple sliding windows determined by the planning unit 113 are as follows: Figure 6 As shown in the right figure.
[0114] Alternatively, Planning Department 113 may also include Interactive Editing Department 113D.
[0115] Reference Figure 7 The UI of the sliding window editing based on the interactive editing department 113D is described. Figure 7 This is a schematic diagram representing the user interface (UI) for editing based on the sliding window of the interactive editing department 113D.
[0116] like Figure 7 As shown, the interactive editing unit 113D displays the reasoning results and multiple sliding windows determined by the planning unit in a user-editable manner within the user interface (UI). Furthermore, the interactive editing unit 113D adjusts the multiple sliding windows based on the user's edits.
[0117] For example, after a user selects the "Edit" button in the user interface (UI), the user can use a "drag" operation to move the central slide of the three slides to the lower right a certain distance, resulting in... Figure 7 The image below shows the three slides after editing. Furthermore, user edits are not limited to moving slides; they can also change the length of any slide, or add or delete a certain number of slides.
[0118] Therefore, the planning department 113 determined the optimized sliding window parameters and multiple sliding windows corresponding to these optimized sliding window parameters.
[0119] The inference unit 114 uses a deep learning model to perform inference on image data to complete the inference task. Specifically, based on the optimized sliding window parameters determined by the planning unit 113 and multiple sliding windows corresponding to these optimized sliding window parameters, the inference unit 114 uses a deep learning model to perform the inference task and obtain the inference result. The inference task is, for example, image processing tasks such as image segmentation, classification, and detection based on the deep learning model. In addition, the inference unit 114 can also combine the process executed by the image processing device 1 with methods such as model pruning and model compression to complete the inference task.
[0120] The output unit 115 outputs various data and information. For example, the output unit 115 outputs the reasoning result based on the reasoning unit 114.
[0121] (Image processing methods)
[0122] Reference Figure 8 The flow of the image processing method of the first embodiment will be described. Figure 8 This is a flowchart illustrating the image processing method of the first embodiment.
[0123] In step S11, the acquisition unit 111 acquires image data. Then, the process proceeds to step S12.
[0124] In step S12, the segmentation unit 41 of the image data preprocessing apparatus 4 segments the region of interest and the region of non-interest from the image data acquired by the acquisition unit 111 in step S11. Thus, the region of interest is segmented from the image data. Next, the process proceeds to step S13.
[0125] In step S13, the setting unit 112 sets multiple sliding window parameters corresponding to the inference task performed by the deep learning model in the image data. Specifically, the setting unit 112 sets recommended values for the multiple sliding window parameters based on parameters corresponding to the inference task stored in the historical database DB in the medical data storage device 5. Alternatively, the setting unit 112 sets recommended values for the multiple sliding window parameters based on user input. Alternatively, within a pre-set range of sliding window parameters, the setting unit 112 generates corresponding input images for each sliding window parameter with a certain interval, obtains the inference result for each input image, and determines the recommended value of the sliding window parameter based on the inference result. Thus, recommended values for multiple sliding window parameters are set corresponding to the inference task. Next, the process proceeds to step S14.
[0126] In step S14, the planning unit 113 generates multiple sliding windows for each sliding window parameter set by the setting unit 112 in step S13, obtains inference results based on the generated multiple sliding windows, and plans the sliding windows by determining optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters from the multiple sliding window parameters according to the inference results. A flowchart of the steps performed by the planning unit 113 will be provided below. Figure 9 This will be discussed later. Therefore, based on the reasoning results, optimized sliding window parameters and multiple sliding windows corresponding to these optimized parameters are determined. Next, proceed to step S15.
[0127] In step S15, the inference unit 114 performs an inference task using a deep learning model based on the optimized sliding window parameters determined by the planning unit 113 in step S14 and the multiple sliding windows corresponding to these optimized sliding window parameters, thereby obtaining an inference result. Thus, an inference result based on the multiple sliding windows corresponding to the optimized sliding window parameters is obtained. Next, the process proceeds to step S16.
[0128] In step S16, the output unit 115 outputs the reasoning result based on the reasoning unit 114 obtained in step S15. The process then ends.
[0129] Reference Figure 9 The planning process of the sliding window in the first embodiment will be explained. Figure 9 This is a flowchart illustrating the planning of multiple sliding windows in the first embodiment.
[0130] In step S141, the sliding window generation unit 113A selects any one of the reference values of the plurality of sliding window parameters set by the setting unit 112, and generates a sliding window according to the selected sliding window parameter. Then, the process proceeds to step S142.
[0131] In step S142, the sliding window generation unit 113A places the generated sliding window at a non-empty corner position of the region of interest. Then, the process proceeds to step S143.
[0132] In step S143, the sliding window adjustment unit 113B adjusts the sliding window generated by the sliding window generation unit 113A. Furthermore, the sliding window adjustment unit 113B sets the length of the sliding window generated by the sliding window generation unit 113A according to the sliding window parameters, and adjusts the length of the sliding window accordingly to the length of the segmented region of interest, thereby moving the position of the adjusted sliding window in a manner that increases the range of the region of interest covered by the sliding window and decreases the range of the non-region of interest covered. Thus, the length and position of the sliding window generated by the sliding window generation unit 113A are adjusted. Next, the process proceeds to step S144.
[0133] In step S144, the region of interest update unit 113C updates the new region of interest by subtracting the adjusted sliding window from the region of interest segmented from the image data by the sliding window adjustment unit 113B. Then, the process proceeds to step S145.
[0134] In step S145, the planning unit 113 determines whether the volume of the new region of interest (ROI) updated by the region of interest update unit 113C is zero. If the volume of the updated ROI is determined to be zero, the process proceeds to step S146. If the volume of the updated ROI is determined to be non-zero, the planning unit 113 returns the process to step S142, and the sliding window generation unit 113A configures sliding windows at other non-empty corner positions of the updated ROI. Steps S142 to S144 are repeated until the volume of the updated ROI is zero. Thus, the planning unit 113 generates and plans multiple sliding windows for each sliding window parameter.
[0135] In step S146, the planning unit 113 determines whether the planned multiple sliding windows require user editing. If user editing is required, the process proceeds to step S147. If user editing is not required, the process proceeds to step S148.
[0136] In step S147, the interactive editing unit 113D displays multiple sliding windows planned by the planning unit 113 in the user interface (UI) in a user-editable manner. Furthermore, the interactive editing unit 113D adjusts the multiple sliding windows according to the user's edits. Next, the process proceeds to step S148.
[0137] In step S148, the planning unit 113 obtains the inference results of multiple sliding windows based on the planning, and determines optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters from the multiple sliding window parameters set by the setting unit 112 according to the inference results. Then, the sliding window planning process ends and proceeds to step S15.
[0138] The planning unit 113 ends the process of planning the sliding windows and proceeds to step S15, which will be described later, where the reasoning unit 114 performs reasoning tasks based on the planned sliding windows.
[0139] Therefore, the image processing apparatus 1 and image processing method according to the first embodiment can dynamically adjust the sliding window to accelerate the inference process of the deep learning model and are applicable to different inference tasks.
[0140] Furthermore, the image processing apparatus 1 and image processing method according to the first embodiment can dynamically adjust the inference process of the deep learning model according to actual needs, and can train the deep learning model based on the user's selection and editing results.
[0141] (Second Implementation)
[0142] The following is for reference Figure 10 , 11 The image processing apparatus 1A of the second embodiment will be described. Figure 10 This is a block diagram illustrating a configuration example of the image processing apparatus 1A according to the second embodiment. Figure 11 This is a schematic diagram illustrating examples of clinical data and key information. In the following description, only the differences between the second embodiment and the first embodiment will be explained, while the similarities will be omitted.
[0143] The difference between the second embodiment and the first embodiment is that the processing circuit 11A of the image processing device 1A further includes a clinical data analysis unit 116.
[0144] The clinical data analysis unit 116 extracts key information related to the inference task performed by the deep learning model from the clinical data of the examinee. Furthermore, the planning unit 113 determines optimization parameters and multiple sliding windows corresponding to these optimization parameters based on the key information extracted from the clinical data of the examinee by the clinical data analysis unit 116.
[0145] Specifically, the clinical data analysis unit 116 obtains, for example, the clinical data of the examinee from the medical data storage device 5. The clinical data of the examinee includes, for example... Figure 11 The above diagram shows information including the examinee's personal information, examination purpose, examination department, examination site, and imaging technology. The Clinical Data Analysis Department 116 extracts key information related to the inference task performed by the deep learning model from the examinee's clinical data, such as... Figure 11 As shown in the figure below, the key information extracted from this clinical data is that the purpose of the examination was preoperative assessment for lumbar spine surgery and the examination site was vertebral artery CTA.
[0146] Next, the planning department 113 determines the inference objectives in the inference task based on the key information extracted by the clinical data analysis department 116, and determines the algorithm and accuracy requirements for each inference objective based on the current clinical progress. The planning department 113 then determines the optimized sliding window parameters based on these accuracy requirements.
[0147] For example, in routine thoracolumbar spine surgery, the assessment of the spinal arteries, especially their branch, the AKA artery, is crucial, as damage to this branch during surgery can lead to spinal cord ischemia. To assess this branch, osteotomy is required in the image data. However, the diameter of the spinal arteries is only 0.1 mm to 1.2 mm; therefore, the accuracy of bone detection during osteotomy is extremely high.
[0148] Therefore, the planning department 113 determines the optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters based on the key information extracted by the clinical data analysis department 116.
[0149] Furthermore, the clinical data and key information mentioned above are merely examples and are not limited to this; as long as the optimization parameters can be determined based on this key information, it is acceptable.
[0150] Therefore, according to the image processing method and image processing apparatus 1A of the second embodiment, by determining the optimized sliding window parameters and generating multiple corresponding sliding windows based on key information in clinical data, the sliding window parameters can be optimized based on clinical data, thereby further accelerating the inference process of the deep learning model.
[0151] (Third Implementation)
[0152] The following is for reference Figure 12 The image processing apparatus 1B of the third embodiment will be described. Figure 12 This is a block diagram illustrating a configuration example of the image processing apparatus 1B according to the third embodiment. In the following description, only the differences between the third embodiment and the first embodiment will be described, while the similarities will be omitted.
[0153] The difference between the third embodiment and the first embodiment is that the processing circuit 11B of the image processing apparatus 1B further includes a model update unit 117.
[0154] The model update unit 117 stores the user's edit results of the optimized sliding window parameters determined by the planning unit 113 and the multiple sliding windows corresponding to the optimized sliding window parameters, and enables the deep learning model to learn based on the edit results.
[0155] Reference Figure 13 The process of model updating based on model updating unit 117 is explained. Figure 13 This is a flowchart illustrating the model update based on the model update unit 117.
[0156] like Figure 13 As shown, in step S14, the planning unit 113 determines the optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters to plan the sliding window. Next, the process proceeds to step S301.
[0157] In step S301, the model update unit 117 determines whether the optimized sliding window parameters determined by the planning unit 113 and the multiple sliding windows corresponding to the optimized sliding window parameters meet clinical requirements. If the model update unit 117 determines that the optimized sliding window parameters determined by the planning unit 113 and the multiple sliding windows corresponding to the optimized sliding window parameters meet clinical requirements, the process proceeds to step S15, where the inference unit 114 performs inference based on the multiple sliding windows determined by the planning unit 113.
[0158] If the model update unit 117 determines that the optimized sliding window parameters determined by the planning unit 113 and the multiple sliding windows corresponding to the optimized sliding window parameters do not meet clinical needs, it proceeds to step S302.
[0159] In step S302, the model update unit 117 displays optimized sliding window parameters that do not meet clinical requirements, along with corresponding multiple sliding window prompts, to the user, allowing the user to edit them. Furthermore, the model update unit 117 stores the user's edits, for example, in the medical data storage device 5. Next, the process proceeds to step S303.
[0160] In step S303, the model update unit 117 updates the deep learning model by learning from the user's editing results stored in the medical data storage device 5.
[0161] In addition, Figure 13 In the flowchart of the third embodiment shown, steps S14 and S15 are the same as in the first embodiment, so their description is omitted.
[0162] Therefore, according to the image processing method and image processing apparatus 1B of the third embodiment, the deep learning model can continuously learn and reuse the optimized inference results for different inference tasks.
[0163] (Comparative Example)
[0164] The following is for reference Figure 14 The differences between the multiple sliding windows planned in the comparative example and the multiple sliding windows planned by the image processing method of the present invention will be explained. Figure 14 This is a schematic diagram illustrating the difference between the sliding window design of the comparative example and the sliding window design of the image processing method of the present invention.
[0165] Figure 14 The upper and middle figures show multiple sliding windows in the comparative example plan. Figure 14 The figure below illustrates multiple sliding windows planned using the image processing method of the present invention. Among them, Figure 14 The comparison example in the image above uses the sliding-window method to plan the sliding window. Figure 14The comparative example in the middle figure uses a method that combines the sliding window approach with the coarse-to-fine approach (Coarse-to-fine + Sliding-Window) to plan the sliding window.
[0166] like Figure 14 As shown, multiple sliding windows planned using the sliding window method are of the same size and are evenly distributed across the entire area of the image data. Compared to multiple sliding windows planned solely by the sliding window method, multiple sliding windows planned using a combination of the sliding window method and a coarse-to-fine approach exhibit improved distribution, with the windows concentrated near the region of interest in the image data. However, the window sizes remain the same, and the distribution is still even.
[0167] Compared to the comparative example, the image processing method of the present invention dynamically generates appropriate sliding windows by setting appropriate sliding window parameters for different inference tasks. Therefore, the size and position of multiple sliding windows are adapted to the region of interest in the inference task. That is, multiple sliding windows are distributed at appropriate sizes and positions that can cover the region of interest.
[0168] Therefore, reasoning based on appropriate sliding windows can accelerate the reasoning process of deep learning models and can be applied to various different reasoning tasks.
[0169] Furthermore, in the above embodiments, the processing circuit is not limited to being implemented by a single processor; it can also be constructed by combining multiple independent processors, with each processor executing a program to implement each processing function. Additionally, the processing functions of the processing circuit can be appropriately distributed or integrated into one or more processing circuits. Furthermore, the processing functions of the processing circuit can also be implemented through a combination of hardware (such as circuits) and software. While an example has been described where the program corresponding to each processing function is stored in a single storage circuit, the implementation is not limited to this. For example, it is also possible to configure the program corresponding to each processing function to be distributed and stored in multiple storage circuits, with the processing circuit reading and executing each program from each storage circuit.
[0170] Furthermore, while the above embodiments illustrate examples where each functional part of this specification is implemented by processing circuitry, the embodiments are not limited to these examples. For instance, each functional part of this specification may be implemented not only by the processing circuitry described in the embodiments, but also by hardware alone, software alone, or a combination of hardware and software to achieve the same function.
[0171] Furthermore, the term "processor" used in the above description of the embodiments can refer to circuits such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), Application Specific Integrated Circuit (ASIC), or programmable logic device (e.g., Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), and Field Programmable Gate Array (FPGA)). Alternatively, instead of storing the computer program in a memory circuit, the computer program can be directly loaded into the processor circuit. In this case, the processor performs its function by reading and executing the program loaded into the circuit. Furthermore, the processors in this embodiment are not limited to being configured as a single circuit; multiple independent circuits can be combined to form a single processor and perform its functions.
[0172] Furthermore, the "computer program product" used in the above description of the embodiments includes a computer program executed by a processor, which is provided pre-loaded into ROM (Read Only Memory) or storage circuitry, etc. Additionally, the computer program can also be provided as a file that can be installed on or executed on computer-readable non-volatile storage media such as CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), DVD (Digital Versatile Disk). Furthermore, the computer program can also be stored on a computer connected to a network such as the Internet and provided or distributed via network download. For example, the computer program is composed of modules including the aforementioned processing functions. As actual hardware, the CPU reads and executes the computer program from a storage medium such as ROM, thereby loading each module onto the main storage device and generating it on the main storage device.
[0173] Furthermore, in the above embodiments, the constituent elements of each device illustrated are functional conceptual elements and do not necessarily need to be physically configured as shown in the figures. That is, the specific manner in which the devices are distributed or concentrated is not limited to that illustrated, and can be configured in any unit, functionally or physically, in accordance with various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can also be implemented by a CPU and the program parsed and executed by the CPU, or as hardware implementation based on wiring logic.
[0174] Furthermore, among the processes described in the above embodiments, all or part of the processes described as automatically performed can be performed manually, or all or part of the processes described as manually performed can be performed automatically using known methods. In addition, the information regarding the processing procedures, control procedures, specific names, various data, or parameters shown in the above documents or drawings can be arbitrarily changed, except where specifically stated.
[0175] Furthermore, the various types of data processed in this specification are typically digital data.
[0176] Several embodiments have been described above, but these embodiments are given as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope or spirit of the invention, and are included in the scope of the invention as described in the claims and its equivalents.
Claims
1. An image processing method, which uses a trained deep learning model to perform inference tasks on image data, characterized in that, include: The steps involve setting multiple sliding window parameters corresponding to the inference task. The planning steps include generating multiple sliding windows for each sliding window parameter, obtaining inference results based on the generated multiple sliding windows, and determining optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters from the multiple sliding window parameters according to the inference results; as well as The inference step involves using the deep learning model to perform the inference task based on a plurality of sliding windows corresponding to the optimized sliding window parameters, thereby obtaining the inference result.
2. The image processing method according to claim 1, characterized in that, The planning steps also include: The sliding window generation step involves generating the sliding window for each sliding window parameter. The sliding window adjustment step involves adjusting the sliding window; and The region of interest (ROI) update step involves subtracting the sliding window from the ROI of the image data to update the new ROI. In the new region of interest, the sliding window generation step, the sliding window adjustment step, and the region of interest update step are performed until the volume of the updated new region of interest is zero, thereby generating multiple sliding windows.
3. The image processing method according to claim 2, characterized in that, In the sliding window generation step, The sliding window is generated at a non-empty corner position within the region of interest. In the sliding window adjustment step, The length of the sliding window is set according to the sliding window parameters. The length of the sliding window is adjusted accordingly to the length of the region of interest. The position of the adjusted sliding window is moved in such a way that the range of the region of interest covered by the sliding window is increased and the range of the region of non-interest covered is decreased.
4. The image processing method according to claim 1, characterized in that, In the setting step, recommended values for multiple sliding window parameters are set based on parameters corresponding to the inference task established in the historical database. In the planning step, multiple sliding windows are generated according to each of the sliding window parameters in the recommended values.
5. The image processing method according to claim 1, characterized in that, In the setting step, recommended values for multiple sliding window parameters are set based on user input. In the planning step, multiple sliding windows are generated according to each of the sliding window parameters in the recommended values.
6. The image processing method according to claim 1, characterized in that, In the setting step, within the preset range of the sliding window parameters, a corresponding input image is generated for each sliding window parameter with a certain interval, the inference result of each input image is obtained, and the recommended value of the sliding window parameter is determined based on the inference result. In the planning step, multiple sliding windows are generated according to each of the sliding window parameters in the recommended values.
7. The image processing method according to any one of claims 4 to 6, characterized in that, The sliding window parameters include the base sliding window length, the sliding window length ratio, the base resolution, and the resolution ratio.
8. The image processing method according to any one of claims 1 to 7, characterized in that, In the planning step, the optimized sliding window parameters and the multiple sliding windows corresponding to the optimized sliding window parameters are determined based on the inference accuracy and inference time.
9. The image processing method according to any one of claims 1 to 7, characterized in that, The planning steps also include an interactive editing step. In the interactive editing step, the reasoning result and the determined multiple sliding windows are displayed in the user interface in a manner that the user can edit, and the multiple sliding windows are adjusted according to the user's editing.
10. The image processing method according to any one of claims 1 to 7, characterized in that, Also includes: The step involves acquiring the image data. The segmentation step involves segmenting the region of interest in the image data. as well as Output the steps and output the reasoning result.
11. The image processing method according to any one of claims 1 to 7, characterized in that, It also includes a clinical data analysis step, extracting key information relevant to the reasoning processing task from the subject's clinical data. In the planning step, the optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters are determined based on the key information.
12. The image processing method according to any one of claims 1 to 7, characterized in that, It also includes a model update step, which stores the user's editing results and enables the deep learning model to learn based on the editing results.
13. An image processing apparatus, which uses a trained deep learning model to perform inference tasks on image data, characterized in that, have: The setting unit sets multiple sliding window parameters corresponding to the inference task; The planning department generates multiple sliding windows for each sliding window parameter, obtains inference results based on the generated multiple sliding windows, and determines optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters from the multiple sliding window parameters according to the inference results; as well as The inference unit, based on a plurality of sliding windows corresponding to the optimized sliding window parameters, uses the deep learning model to perform the inference task and obtain the inference result.
14. A computer program product comprising a computer program that, when executed by a processor, performs an inference task on image data using a trained deep learning model, characterized in that, When the computer program is executed, it includes the following steps: The steps involve setting multiple sliding window parameters corresponding to the inference task. The planning steps include generating multiple sliding windows for each sliding window parameter, obtaining inference results based on the generated multiple sliding windows, and determining optimized sliding window parameters and multiple sliding windows corresponding to the optimized sliding window parameters from the multiple sliding window parameters according to the inference results; as well as The inference step involves using the deep learning model to perform the inference task based on a plurality of sliding windows corresponding to the optimized sliding window parameters, thereby obtaining the inference result.