Image processing method and image processing system

Abstract

The present application provides an image processing method and system, which uses a trained deep learning model to infer image data. The image processing method comprises the following steps: receiving an image; determining a first part and a second part in the received image; using the deep learning model to perform first inference processing on the determined first part and second inference processing on the determined second part, wherein the first inference processing and the second inference processing are different inference processing; and integrating the results of the first inference processing and the second inference processing and outputting the integrated inference result.

Description

Technical Field

[0001] This invention relates to image processing methods and systems. More particularly, it relates to image processing methods and systems that, in deep learning-based image processing, can balance the accuracy and speed of the inference process in deep learning. Background Technology

[0002] Deep learning is currently a crucial technique for image processing, including segmentation, classification, and detection. A complete deep learning framework comprises two main parts: training and inference. The training process inputs a labeled training dataset (or ground truth) into the model, calculates the objective function (loss function) between the output detection results and the ground truth, and uses methods such as gradient descent and stochastic gradient descent to correct the network parameters and minimize the loss function. This process is repeated until the error between the network's output detection results and the ground truth meets the specified accuracy, indicating that the model has converged and the error in the model's predictions has been reduced, thus completing the model training. The inference process involves inputting unlabeled live data into the trained model to obtain the actual detection values.

[0003] In the inference process, inaccurate results may occur when using a pre-trained model to infer images. This inaccuracy is particularly noticeable in certain parts of the image, such as the edges of organs, tumors, and the boundaries of organ defects caused by partial organ resection.

[0004] Existing technologies have proposed several methods to improve the inference accuracy in deep learning. For example, Patent Document 1 (US2019 / 0205606A) discloses an artificial intelligence-based medical image segmentation method and system that can adaptively apply appropriate algorithms for inference processing on the same set of images. Specifically, the technology in Patent Document 1 automatically selects one or more suitable deep learning models for segmentation based on the image's scanning protocol, organ features, and other characteristics. When multiple models are selected, a model concatenation approach is used, concatenating the inference result of the previous model with the original image as the input to the next model.

[0005] Patent document 2 (US10825168B2) discloses an automatic organ segmentation machine learning method and system, which focuses on organ segmentation, uses image patches of different sizes for multi-channel inference, and then concatenates the results of multi-channel inference as the input to the last layer of a deep learning network.

[0006] Patent document 3 (Japanese Patent Application Publication No. 2022-002027) discloses a learning device and an inference device that use a machine learning model. The device determines whether the object data has a predictable region that can easily predict the inference result of the inference process, performs a prescribed data processing on the data region that is a predictable region, and outputs the data required for inference of the region that is not a predictable region to the machine learning model, thereby shortening the average time of data processing. Summary of the Invention

[0007] Problems in the prior art

[0008] To improve the accuracy of the inference process, existing technologies have attempted to perform more refined processing through improved inference algorithms. These improved inference processes are called "enhanced inference." Specific examples of enhanced inference include: increasing the overlap ratio between image patches used in inference during image segmentation; performing inference on each image during image classification; and flipping or scaling images before performing inference during image detection.

[0009] In the prior art, when performing the above-mentioned enhanced inference, these enhanced inference processes are applied to all images that are the objects of processing, which inevitably increases the processing time of the algorithm. Therefore, the prior art has a technical problem that accuracy and processing speed cannot be balanced in image processing using enhanced inference.

[0010] Means for solving technical problems

[0011] This invention addresses the problems of the prior art. It proposes a novel method that balances accuracy and speed in inference processing within deep learning algorithms. The image to be processed is divided into a first part and a second part. The first part refers to the portion of the image where deep learning algorithms are prone to inaccurate results, such as areas near where the inference results show abrupt changes. The second part is the portion of the image excluding the first part. Different inference processes are applied to the first and second parts respectively—namely, a first inference process and a second inference process. For example, a refined enhanced inference is used for the first part, while a faster, standard inference is used for the second part. The results of the enhanced and standard inference processes are then combined to form the final processed image.

[0012] Specifically, according to one aspect of the present invention, an image processing method is provided, which uses a trained deep learning model to perform inference processing on image data, characterized by comprising the following steps: a receiving step, receiving an image; a determining step, determining a first part and a second part in the received image; an inference step, using the deep learning model to perform a first inference processing on the determined first part and a second inference processing on the determined second part, wherein the first inference processing and the second inference processing are different inference processing; and an integration step, integrating the results of the first inference processing and the second inference processing, and outputting the integrated inference result.

[0013] According to another aspect of the present invention, an image processing system is provided, which performs inference processing on image data using a trained deep learning model, characterized in that it comprises: a receiving device for receiving an image; a determining device for determining a first part and a second part in the received image; an inference step for performing a first inference process on the determined first part and a second inference process on the determined second part using the deep learning model, wherein the first inference process and the second inference process are different inference processes; and an integration device for integrating the results of the first inference process and the second inference process and outputting the integrated inference result.

[0014] According to the present invention, the image to be processed is divided into a first part and a second part, wherein the first part refers to the portion of the image near the location where the result of the inference processing has a jump. Different inference processing is performed on the first part and the second part respectively. Specifically, refined enhanced inference is used for the first part, while faster normal inference is used for the second part. The processing results of enhanced inference and normal inference are then integrated as the processing result of the entire image, thereby achieving a balance between accuracy and speed in deep learning model inference and obtaining a more ideal inference result. Attached Figure Description

[0015] Figure 1 This is a block diagram illustrating an example of the structure of the image processing system according to the present invention.

[0016] Figure 2 This is a comparative diagram used to illustrate the features of the processing performed by the image processing system of the present invention.

[0017] Figure 3 This is a flowchart illustrating an example of the processing performed by the image processing system of the present invention.

[0018] Figure 4 This is a schematic diagram of the preprocessing steps for determining the first and second parts of the image processing system according to the first embodiment of the present invention.

[0019] Figure 5A and Figure 5B This is a schematic diagram illustrating the process by which an image processing system according to a first embodiment of the present invention determines the first part and the second part.

[0020] Figure 6 is a schematic diagram illustrating the principle and effect of the image processing system according to the first embodiment of the present invention in determining the first part and the second part.

[0021] Figure 7 This is a schematic diagram of the processing status of step S200 performed by the image processing system according to the first embodiment of the present invention.

[0022] Figure 8 This is a schematic diagram of the processing status of step S300 performed by the image processing system according to the first embodiment of the present invention.

[0023] Figure 9A and Figure 9B This is a schematic diagram of the processing status of step S500 performed by the image processing system according to the first embodiment of the present invention.

[0024] Figure 10 This is a schematic diagram illustrating the process by which the image processing system according to the second embodiment of the present invention determines the first part and the second part.

[0025] Figure 11 This is a schematic diagram of the processing status of the deep learning model of the image processing system according to the second embodiment of the present invention in step S100.

[0026] Figure 12 This is a schematic diagram of the processing status of step S200 performed by the image processing system according to the second embodiment of the present invention.

[0027] Figure 13This is a schematic diagram of the processing status of step S300 performed by the image processing system according to the second embodiment of the present invention.

[0028] Figure 14 This is a schematic diagram of the processing status of step S500 performed by the image processing system according to the second embodiment of the present invention. Detailed Implementation

[0029] Hereinafter, embodiments of the image processing system, image processing method, and storage medium relating to this application will be described in detail with reference to the accompanying drawings. However, the image processing system, image processing method, and storage medium relating to this application are not limited to the embodiments shown below. Furthermore, in the following description, the same constituent elements will be given common reference numerals and repeated descriptions will be omitted.

[0030] First, a general overview of the image processing system of the present invention will be described. The image processing system of this application can exist in the form of a medical image diagnostic device such as an ultrasound diagnostic device, a CT imaging device, or an MRI imaging device, or it can exist independently in the form of a workstation or the like.

[0031] Figure 1 This is a block diagram illustrating an example of the structure of the image processing system according to the present invention. The image processing system 1 of this embodiment utilizes a deep learning neural network to perform inference processing on the input image, such as... Figure 1 As shown, the image processing system 1 mainly includes a receiving device 10, a determining device 20, an inference device 30, and an integration device 40. The receiving device 10 is used to receive an image of the object to be processed. The determining device 20 is used to determine a first part and a second part other than the first part in the image received by the receiving device 10. In this invention, the first part refers to the part of the image in which the deep learning model algorithm is prone to inaccurate results, such as the part near the location where the inference processing result has a jump. Details about the first part will be described later. The inference device 30 uses a trained deep learning model to perform different first inference processing and second inference processing on the first part and the second part determined by the determining device 20, respectively. The integration device 40 integrates the inference results obtained from the different inference processing and outputs the integrated inference result.

[0032] Image processing system 1 can be equipped, for example, in an image processing system such as an ultrasound diagnostic device. In this case, image processing system 1 also includes a control unit (omitted from the illustrations), an ultrasound probe, a display, an input / output interface, and a device body. The receiving device 10, determining device 20, inference device 30, and integration device 40 are mounted in the control unit and are communicatively connected to these ultrasound probes, displays, input / output interfaces, and device body. Since the structure and function of the control unit, ultrasound probe, display, input / output interface, and device body are well known to those skilled in the art, detailed descriptions are omitted.

[0033] The processing performed by the image processing system 1 of this embodiment will now be described in detail.

[0034] Figure 2 This is a comparative diagram used to illustrate the features of the processing performed by the image processing system of the present invention.

[0035] Figure 2 The features of the present invention are illustrated by comparing it with existing technology processes. The left side is a flowchart of existing image processing, and the right side is a flowchart of the processing performed by the image processing system of the present invention. Figure 2 As shown, in the processing performed by the image processing system according to the present invention, steps S100, S200, and S300 are characteristic steps of the present invention. Unlike the prior art which employs enhanced reasoning on the entire image in step S20', the present invention first determines a first part and a second part of the image to be processed in step S100. Then, in step S200, the present invention performs different reasoning processes on the determined first part and second part, namely, a first reasoning process and a second reasoning process. Specifically, as will be discussed later... Figure 3 As described in detail, sophisticated enhanced inference is used for the first part, while faster normal inference is used for the second part. Subsequently, in step S300, the present invention integrates the processing results of enhanced inference and normal inference as the processing result for the entire image. The image processing system of the present invention distinguishes between the first and second parts of the image to be processed, and performs different first and second inference processes respectively. Therefore, unlike existing technologies that perform enhanced inference processing on the entire image of the object being processed, the present invention can achieve a balance between accuracy and speed during deep learning model inference, obtaining a more ideal inference result.

[0036] The following uses Figure 3 The details of the processing performed by the image processing system of the present invention will be described.

[0037] Figure 3 This is a flowchart illustrating an example of the processing performed by the image processing system of the present invention.

[0038] First, in step S10, the image processing system 1 receives the medical image data of the object to be processed through the receiving device 10.

[0039] The subsequent steps S100 to S300, and the processing of step S30, are combined with the preceding text. Figure 2 This has already been explained, so I will not repeat it here.

[0040] The image processing system according to the present invention may further include a judgment step of S400 and a correction step of S500 between steps S300 and S30. In step S400, the image processing system 1 judges whether the integrated algorithm result is satisfactory. This step can be performed by pre-setting a predetermined threshold for the technical indicator representing the result of the reasoning process and comparing the indicator of the integrated result with the threshold, or by human judgment by the user of the image processing system 1.

[0041] If the judgment result in step S400 is satisfactory (step S400, Yes), the process proceeds to step S30, the algorithm result is output, and the process ends. Conversely, if the judgment result in step S400 is unsatisfactory (step S400, No), the process proceeds to step S500, where the first part determined in step S100 is corrected, and the processing of steps S200 and S300 continues for the corrected first part and the other second part. The judgment in step S400 is executed again for the integrated algorithm result until the judgment result in step S400 is "Yes". The processing in step S500 can be automatically corrected by the correction function of image processing system 1 according to preset rules, or it can be manually corrected by the user through the user interface of image processing system 1.

[0042] Steps S400 and S500 are not mandatory. For example... Figure 2 The method shown, which only includes steps S10, S100 to S300, and step S30, can also achieve the purpose and effect of the present invention.

[0043] The above provides a general overview of the processing performed by the image processing system of the present invention. The present invention will now be described in detail using segmentation and classification processing as examples.

[0044] (First Embodiment)

[0045] The first embodiment applies the present invention to the case of image segmentation.

[0046] The following example uses liver segmentation, combined with... Figure 4 Figure 9 illustrates the first embodiment of the present invention in detail.

[0047] First, combine Figure 4 Figure 6 illustrates the process of determining the first and second parts of step S100 in the first embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of the preprocessing steps for determining the first and second parts of the image processing system according to the first embodiment of the present invention. Figure 4 The diagram illustrates an example of segmentation processing on an image. The large box represents the entire image, the four large squares within the box represent the image patches targeted by the segmentation process, and the small, dark squares within each patch represent individual pixels. The numbers in the small squares represent the inferred probability of that pixel. The size and number of boxes and squares shown are for illustrative purposes only and may not reflect actual scale; the actual number of pixels may also be greater than those shown. This applies to all accompanying figures.

[0049] like Figure 4 As shown, when the image processing system of the first embodiment performs liver segmentation processing on medical image data, in step S100, the determining device 20 first performs general inference on the entire image of the object to be processed using a deep learning model. In this invention, "general inference" refers to light inference, which has lower accuracy, lower processing load, and faster speed compared to enhanced inference. Specifically, the general inference of this invention can be, for example, as follows: Figure 4 The inference shown uses a large image patch. After performing light inference on the entire processing object image, the determining device 20 obtains a matrix composed of the inference probabilities of each pixel in the entire image. The inference probability is an example of the "inference result index" in this invention, which refers to the probability that a pixel belongs to the target region (liver) when segmenting a target region (e.g., liver) in the processing object image. Figure 4 In the image, the dark area represents the target region (liver), and the light background represents the area outside the target region (liver).

[0050] like Figure 4 As shown, within the dark liver region, after mild inference processing, the segmentation result for each pixel is that the pixel belongs to the liver region, and the corresponding inference probability for each pixel is 1.0. In the light-colored region, the segmentation result for each pixel is that the pixel does not belong to the liver region, and the corresponding inference probability for each pixel is 0.0. Near the boundary between the dark and light-colored regions, the segmentation result for each pixel is that it belongs to the liver region with a probability between 0.0 and 1.0. Figure 4 The example shows, for instance, four pixels located on the boundary, with inference probabilities of 0.52, 0.50, 0.50, and 0.48, respectively. Thus, in... Figure 4 In the image of the object to be processed shown, there are regions near the boundary between the dark and light regions where the result of the inference processing exists, that is, the inference probability in this embodiment jumps from 0.0 to a value greater than 0.0 and less than 1.0, or from 1.0 to a value greater than 0.0 and less than 1.0. In this embodiment, the region near such a region in the image of the object to be processed where the result of the inference processing jumps is defined as the first part.

[0051] In the first embodiment of image segmentation, the first part and the second part of the image in this invention refer to different regions in an image. Specifically, the first part and the second part refer to a key region that is important for segmentation and a non-key region that is not important for segmentation, respectively. The process of determining the first part and the second part according to the image processing system of the first embodiment is described in detail below.

[0052] Figure 5A and Figure 5B This is a schematic diagram illustrating the process by which an image processing system according to a first embodiment of the present invention determines the first part and the second part.

[0053] In such Figure 4 After obtaining the inference probability matrix of each pixel contained in the entire image of the object being processed, the determining device 20, as shown, Figure 5A and Figure 5B The first and second parts are determined based on the inference probability matrix.

[0054] First, such as Figure 5A As shown, in this embodiment, the determining device 20 sets a key area inference probability range with a lower limit greater than 0 and an upper limit less than 1 based on the obtained inference probability matrix. The image area composed of pixels whose inference probability values ​​are within the set key area inference probability range is determined as the first part, and the image area other than that is determined as the second part.

[0055] Figure 5A As an example, the lower limit is set to 0.05, the upper limit is set to 0.95, and the probability range for inference of the key area corresponding to the first part is set as follows:

[0056] 0.05 ≤ inference probability ≤ 0.95

[0057] That is, according to the first embodiment, the present invention defines the image region composed of pixels with inference probabilities between 0.05 and 0.95. Figure 5BThe dark, somewhat wide outer contour area of ​​the liver was identified as the key area, and all other pixels that met the criteria were excluded.

[0058] 0 ≤ inference probability < 0.05

[0059] or

[0060] 0.95 < inference probability ≤ 1.0

[0061] The image region composed of pixels, that is Figure 5B The light-colored areas are defined as non-critical areas. In this embodiment, the critical areas and non-critical areas are examples of the first part and the second part, respectively.

[0062] The principle and effect of the method for determining the first and second parts in step S100 are explained below with reference to Figure 6.

[0063] Figure 6 illustrates the principle and effect of the image processing system according to the first embodiment of the present invention in determining the first and second parts, using the segmentation of the liver, tumor, and liver defect as examples. Figure 6A It is a schematic diagram representing the real image that is being processed. Figure 6B This is a diagram illustrating the principle and effect of setting the probability range for inference in key areas. Figure 6C This is a schematic diagram showing the segmented image displaying the first and second parts.

[0064] like Figure 6A As shown, in the real medical image that is the object of processing, there is a liver that is the object of segmentation. Figure 6A (the oblique part in the text) and tumors in the liver or defects caused by surgical removal, etc. Figure 6A The grid line portion (as described above) is processed by the determining device 20 using mild reasoning. Figure 6A After segmenting the objects in the image—the liver, tumor, and defect—we obtain... Figure 6B The inference probability distribution is shown below. Figure 6B The horizontal axis represents the inference probability of each pixel, and the vertical axis represents the number of pixels corresponding to the normalized inference probability. From Figure 6B As can be seen, in the inference processing results, the number of pixels near inference probabilities 0 and 1 is the highest. The number of pixels is significantly lower in the range slightly away from 0 but greater than 0 (e.g., 0.05) and slightly away from 1 but less than 1 (e.g., 0.95). Therefore, this invention selects such a range of inference probabilities with significantly fewer pixels as the key region inference probability range, thereby determining the key region (first part) corresponding to the key region inference probability range and the non-key region (second part) elsewhere.

[0065] In this embodiment, the inference probability range for the key region is set to be between 0.05 and 0.95. However, this is merely an example, and the inference probability range for the key region in this invention is not limited to this. Furthermore, in this embodiment, the inference probability range for the key region is set such that the difference between the lower limit and 0 and the difference between the upper limit and 1 are 0.05-1 = 0.05 and 1-0.95 = 0.05, respectively. The difference between the lower limit and 0 and the difference between the upper limit and 1 are equal. However, this invention is not limited to this; the inference probability range for the key region can also be set such that the difference between the lower limit and 0 and the difference between the upper limit and 1 are not equal. According to this invention, the inference probability range where the number of pixels near inference probabilities 0 and 1 is significantly low can be selected as the inference probability range for the key region. In this invention, the number of pixels is normalized, and the range where the number of pixels is significantly low is determined by the normalized pixel distribution. However, this invention is obviously not limited to this; the definition and determination method of a significantly low pixel distribution can adopt other solutions well-known to those skilled in the art in the prior art. This is not the focus of this invention and will not be described in detail here.

[0066] After setting the inference probability range for key regions to between 0.05 and 0.95, the image processing system of the present invention according to the first embodiment determines, as follows: Figure 6C The key area shown (Part 1) is from... Figure 6C As can be seen, pixels with inference probabilities between 0.05 and 0.95 constitute the outer contour of the liver and the contour of the tumor and / or defect with a certain width. The outer contour of the liver and the contour of the tumor / defect are the key targets for segmentation of the medical image of the object being processed. Through the determination step S100 of the present invention, the above-mentioned outer contour of the liver and the contour region of the tumor / defect are determined as the key region (first part), and the other regions are determined as non-key regions (second part).

[0067] The above is a description of the determination of the first and second parts of step S100 in the first embodiment of the present invention.

[0068] After completing the process of determining the first part and the second part, the present invention proceeds to the reasoning process in step S200.

[0069] The following is combined Figure 7 The reasoning process of step S200 in the first embodiment of the present invention is explained.

[0070] Figure 7 This is a schematic diagram of the processing status of step S200 performed by the image processing system according to the first embodiment of the present invention.

[0071] After determining the first part (key area) and the second part (non-key area) in step S100, step S200 applies different reasoning processes to the first part (key area) and the second part (non-key area), namely, first reasoning process and second reasoning process. Specifically, the reasoning device 30 uses enhanced reasoning as the first reasoning process for the determined first part. As enhanced reasoning, it can be as follows: Figure 7 The image is re-divided into blocks centered on the first part for inference.

[0072] In addition, as an enhancement to inference, other test-time augmentation methods include flipping image patches containing key regions and performing inference again, or increasing the patch overlap ratio when inferring key regions. This invention is not limited to these methods; other segmentation enhancement inference schemes well-known to those skilled in the art can also be employed. The inference device 30 uses a lighter, faster inference process for the second part as the second inference process, which has a lower processing load. Here, the inference device 30 can directly use the results of the lighter inference process performed on the second part in step S100 when determining the first and second parts. Directly using the results of the lighter inference process in step S100 saves processing time, reduces processing load, and further accelerates processing speed, making it preferred. This invention is not limited to these methods; other lighter inference processes can also be performed on the second part as the second inference process.

[0073] After the reasoning process is completed, the present invention proceeds to the integration process in step S300.

[0074] Next, combined Figure 8 The integration process of step S300 in the first embodiment of the present invention is explained.

[0075] Figure 8 This is a schematic diagram of the processing status of step S300 performed by the image processing system according to the first embodiment of the present invention.

[0076] like Figure 8 As shown, in step S300, the processing results of the enhanced reasoning of the first part and the mild reasoning of the second part obtained in step S200 are integrated, and the integrated processing result is used as the complete reasoning result.

[0077] Specifically, this invention integrates the reasoning results by directly superimposing the results of the enhanced reasoning of the first part and the mild reasoning of the second part. In step S200, if the preferred method of directly using the mild reasoning processing results performed on the second part when determining the first and second parts in step S100 is used, the enhanced reasoning processing results obtained in step S200 for the first part can be directly overlaid on the mild reasoning results obtained in step S100 for the second part. Alternatively, the enhanced reasoning results of the first part and the mild reasoning results of the second part can be fused according to a certain weight. In the case of segmenting the liver, the integration processing result in step S300 is as follows: Figure 8 The complete liver segmentation result is shown.

[0078] Thus, the present invention can output the processing result of step S300 as the final result of the reasoning process.

[0079] Furthermore, as described above, in a preferred embodiment, the image processing system 1 according to the present invention may further include a judgment step S400 and a correction step S500 between steps S300 and S30. In step S400, the image processing system 1 determines whether the integrated algorithm result is satisfactory. If the judgment result in step S400 is "yes", the processing proceeds to step S30, the algorithm result is output, and the processing ends. Conversely, if the judgment result in step S400 is "no", the processing proceeds to step S500, the first part and the second part determined in step S100 are corrected, and the processing returns to step S200.

[0080] The details of the judgment and processing in step S400 are combined with the above text. Figure 3 The process of correcting step S500 has already been explained. The following section, with reference to Figure 9, will explain the process of correcting step S500.

[0081] Figure 9A and Figure 9B This is a schematic diagram of the processing status of step S500 performed by the image processing system according to the first embodiment of the present invention.

[0082] like Figure 9A As shown, the processing in step S500 can be achieved by adjusting the lower and upper limits of the inference probability range for the key area. This adjustment can be automatically performed by the correction function of the image processing system 1 according to preset rules, or it can be manually adjusted by the user. Additionally, as... Figure 9B As shown, the processing in step S500 can also be performed in, for example... Figure 6CThe corrections can be made directly through the user interface in the image of the key area (first part) and the non-key area (second part). Such corrections can be made by the user clicking on the corresponding area in the image and dragging and dropping with the mouse, or they can be done automatically by the system.

[0083] After executing step S500, the image processing system of the present invention re-processes steps S200 and S300 for the corrected first part and the other second part, and performs the judgment of step S400 again on the integrated algorithm result until the judgment result of step S400 is "yes".

[0084] As mentioned above, steps S400 and S500 are not mandatory. The objectives and effects of the present invention can be achieved even without performing steps S400 and S500.

[0085] Summary of the first implementation method

[0086] According to the first embodiment, the inference processing of the present invention is used for image segmentation. The first part is the key region in the image segmentation, and the second part is the non-key region in the image segmentation. The determination step uses a deep learning model to segment the image, obtaining an inference result index representing each pixel in the segmented image. The region consisting of pixels whose inference result indexes are between a specified lower limit and a specified upper limit is determined as the key region, and the remaining regions are determined as non-key regions. The inference step performs enhanced segmentation based on enhanced inference on the key regions, using the result as the inference result for the key regions, and directly uses the segmentation result for the non-key regions from the determination step as the inference result for the non-key regions. Thus, according to this embodiment, since the image to be processed is divided into a first part and a second part, different inference processing is performed on the first part and the second part respectively—namely, a first inference processing and a second inference processing. Specifically, a refined enhanced inference processing is used on the first part, while a faster, conventional inference processing is used on the second part. The results of the enhanced inference processing and the conventional inference processing are then integrated as the processing result for the entire image, thereby automatically achieving a balance between accuracy and speed during deep learning model inference, and obtaining a more ideal inference result.

[0087] Details about deep learning and methods for segmenting images using deep learning models are well known in the art and are not the focus of this invention, so they will not be described in detail here.

[0088] In the above description, the inference result index of the first embodiment was explained using inference probability as an example. However, the present invention is not limited to this. In addition to using inference probability, for example, the inference uncertainty of the deep learning model can also be used as the inference result index, or the differences between image pixels can be found using traditional image processing methods as the inference result index to find the first part of the image. Details about inference uncertainty are described in detail in, for example, the paper "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?" by Alex Kendall and Yarin Gal (31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA), and are omitted here.

[0089] (Second Implementation)

[0090] In the first embodiment described above, the image processing system of the present invention was explained using segmentation as an example. However, the implementation is not limited thereto. For example, the image processing system of the present invention can also be used for image classification.

[0091] The second embodiment applies the present invention to image classification. The following example uses classification processing for organ localization, such as the liver, in conjunction with... Figures 10-14 The second embodiment of the present invention will be described in detail.

[0092] In the description of the second embodiment, the differences from the first embodiment described above will be mainly explained. Furthermore, in the description of the second embodiment, structures identical to those in the first embodiment will be labeled with the same reference numerals and their descriptions will be omitted.

[0093] First, combine Figures 10-11 This describes the process of determining the first and second parts in step S100 of the second embodiment of the present invention.

[0094] In the second embodiment of the image classification processing for organ localization, the first part and the second part of the image in the present invention refer to different ranges of the number of images in multiple images. Specifically, the first part and the second part refer to the range of the number of key images that are important for classification and the range of the number of non-key images that are not important for classification, respectively.

[0095] Figure 10This is a schematic diagram illustrating the process by which the image processing system according to the second embodiment of the present invention determines the first part and the second part.

[0096] Figure 10 The image shows an example of performing classification processing, such as liver localization, on a series of object images. Figure 10 As shown, the image processing system according to the second embodiment of the present invention, for example, a set of multiple consecutive 2D images of the upper body of a human body, uses a multi-channel deep learning model to classify and detect the multiple images. A predetermined range of images near which the classification and detection results show abrupt changes is determined as a key range (first part), and the remaining range is determined as a non-key range (second part). Specifically, the image processing system according to the second embodiment of the present invention uses a multi-channel input deep learning model to classify and detect each group of images, and determines the group containing images with abrupt changes in the classification and detection results as the key range. 。 The "each set of images" referred to here is Figure 10 The numbers are ①~ A collection of multiple consecutive 2D images within a range shown by a long, flat rectangle extending along the left and right sides of the human body. Figure 10 The image range for each group is shown only (the individual images within each group are not shown). The number of images in each group is equal to the number of input channels of the model.

[0097] exist Figure 10 In the example shown, classification detection identified image groups numbered ④ to ⑥ (the darker areas in the image) as containing liver images. ， The others are numbered ①~③ and ⑦~ The image group is the image group that does not contain the liver. Furthermore, in image groups ④ to ⑥ that contain the liver, image groups ④ and ⑥ respectively contain transitions from images without the liver to images with the liver, and from images with the liver to images without the liver. Image group ⑤, however, contains the liver in every image and does not contain transitions. Therefore, according to the determining device 20 of the second embodiment, image groups ④ and ⑥ are determined as the key number range (first part), while image groups ① to ③, image group ⑤, and ⑦ to ⑥ are excluded. The number of sheets identified as non-priority (Part Two).

[0098] Figure 11 This is a schematic diagram illustrating the processing status of the deep learning model in step S100 of the image processing system according to the second embodiment of the present invention. Figure 11As shown, in step S100, a multi-channel deep learning model can infer the classification results of a set of images at once. For example, when the number of channels is set to 10, 10 2D images are used as input to the model, and after one inference, the result of whether a liver exists in these 10 images can be obtained. In this invention, as long as the inference result of a set of images, such as 10 images, is that at least one image contains a liver, it is considered that a liver exists in that set of images. The number of images in each group and the corresponding number of groups can be adjusted by adjusting the number of channels.

[0099] Inference based on multi-channel deep learning models is faster and has a lower processing load, but its classification accuracy is also lower. Inference performed solely using multi-channel deep learning models may produce inaccurate results. According to the determination device 20 of the second embodiment, after performing inference based on a multi-channel deep learning model on a series of target images, it obtains results indicating whether the classification results of each group of images in the entire image contain a jump portion. The image groups whose classification results contain jump portions are defined as the key image range (first part), and the other image groups are defined as the non-key image range (second part). The image processing system of this embodiment then performs more refined enhanced inference on the key image range (first part).

[0100] The following is combined Figure 12 The processing of step S200 in the second embodiment of the present invention will be explained.

[0101] Figure 12 This is a schematic diagram of the processing status of step S200 performed by the image processing system according to the second embodiment of the present invention.

[0102] After determining the first part (the range of key sheets) and the second part (the range of non-key sheets) in step S100, step S200 applies different reasoning processes to the first part (the range of key sheets) and the second part (the range of non-key sheets), namely, the first reasoning process and the second reasoning process. The reasoning device 30 is as follows: Figure 12 The first part, as shown, is processed using enhanced reasoning as the first reasoning method. Specifically, as enhanced reasoning, it can be as follows: Figure 12 The diagram shows that inference is performed on each image in the first part based on a single-channel deep learning model. By using a single-channel deep learning model to perform more refined inference on each image in the first part, a more accurate classification result can be obtained for each image.

[0103] Figure 12 The example shown is an example of augmented reasoning based on a single-channel deep learning model. However, the invention is not limited to this. The augmented reasoning in the second embodiment can also employ other augmented reasoning schemes for classification that are well known to those skilled in the art.

[0104] The inference device 30 employs a lighter, faster inference process for the second part as the second inference process. Here, similar to the first embodiment, the inference device 30 of the second embodiment can directly use the processing results of the lighter inference process performed on the second part in step S100 when determining the first and second parts. However, the invention is not limited to this; other lighter inference processes can also be performed again for non-critical page count ranges as the second inference process.

[0105] After the reasoning process is completed, the present invention proceeds to the integration process in step S300.

[0106] Next, combined Figure 13 The integration process of step S300 in the second embodiment of the present invention is explained.

[0107] Figure 13 This is a schematic diagram of the processing status of step S300 performed by the image processing system according to the second embodiment of the present invention.

[0108] like Figure 13 As shown, in step S300, the processing results of the enhanced reasoning of the first part and the mild reasoning of the second part obtained in step S200 are integrated to form a complete reasoning result.

[0109] The details of step S300 are the same as in the first embodiment, and will not be repeated here.

[0110] like Figure 13 As shown, the image processing system according to the second embodiment of the present invention can further obtain more accurate classification results in the range of key images (first part) (groups ④ and ⑥), such as images where no liver is present at the location represented by the dashed line and images where a liver is present at the location represented by the solid line.

[0111] Thus, the present invention can output the processing result of step S300 as the result of the reasoning process.

[0112] In addition, as described above, similar to the first embodiment, as a preferred embodiment, the image processing system 1 according to the second embodiment may also include a determination step S400 and a correction step S500 between step S300 and step S30.

[0113] The details of the judgment process in step S400 are combined with those in the first embodiment. Figure 3 The explanations are the same. The following is in conjunction with... Figure 14 The processing of the correction step S500 in the second embodiment is explained.

[0114] Figure 14 This is a schematic diagram of the processing status of step S500 performed by the image processing system according to the second embodiment of the present invention.

[0115] like Figure 14 As shown, the processing in step S500 can be achieved by adjusting the boundary of the range of key images (first part). This adjustment can be performed automatically by the correction function of the image processing system 1 according to the preset rules, just like in the first embodiment. Of course, it can also be achieved by the user clicking on the corresponding area in the image with the mouse and dragging and dropping.

[0116] After executing step S500, the image processing system of this embodiment re-processes steps S200 and S300 for the corrected range of key images (first part) and the range of non-key images (second part), and performs the judgment of step S400 again on the integrated algorithm result until the judgment result of step S400 is "yes".

[0117] Similar to the first embodiment, steps S400 and S500 are not mandatory in this embodiment. The objective and effect of the present invention can be achieved even without performing steps S400 and S500.

[0118] Summary of the second implementation method

[0119] According to the second embodiment, the inference processing of the present invention is used to locate organs contained in an image based on image classification. The receiving step receives multiple consecutive images. A first part is the range of images representing the most important elements in the classification of the multiple images, i.e., the key image range; a second part is the range of images representing the least important elements in the classification of the multiple images, i.e., the non-key image range. The determining step uses a multi-channel deep learning model to perform classification detection on the multiple images, determining a specified range of images near images with abrupt changes in classification detection results as the key image range, and determining the remaining ranges as the non-key image ranges. The inference step uses a single-channel deep learning model to perform classification detection on the key image range, using the result as the inference result for the key image range, and directly using the classification detection result for the non-key image range from the determining step as the inference result for the non-key image range. Thus, similar to the first embodiment, according to this embodiment, since the image of the object to be processed is divided into a first part and a second part, different inference processes are performed on the first part and the second part respectively, namely, first inference processing and second inference processing. Specifically, a fine-grained enhanced inference process is used on the first part, while a faster normal inference process is used on the second part. Then, the results of the enhanced inference process and the normal inference process are integrated as the processing result of the entire image, thereby automatically achieving a balance between accuracy and speed during deep learning model inference and obtaining a more ideal inference result.

[0120] The image processing system according to the present invention achieves a balance between accuracy and speed during deep learning network inference. In a first embodiment for image segmentation, the number of image blocks for inference according to the present invention can be reduced to less than half that of conventional methods without changing the inference accuracy. In a second embodiment for image classification, without changing the inference accuracy, taking 1000 whole-body CT images as an example, when light inference processing infers once for every 10 images, while enhanced inference processing infers for every image, the number of image inference processes according to the present invention can be reduced to less than 20% of that of conventional methods.

[0121] (Other implementation methods)

[0122] The above embodiments use liver segmentation and classification as an example for illustration, but the present invention can of course be applied to the segmentation and classification of other organs and tissue structures other than the liver, as well as other image processing besides segmentation and classification.

[0123] The image processing, segmentation, classification, deep learning model and neural network training and inference described in the above embodiments can all be implemented using various methods in the prior art, and detailed descriptions are omitted here.

[0124] This invention can be implemented not only as the aforementioned image processing system, but also as an image processing method, a program, or a medium storing an image processing program.

[0125] The image processing system described in this application can be installed in a medical image diagnostic device, or it can be a standalone image processing system. In the latter case, the image processing system has a processing circuit that performs the same processes as described above, and a memory that stores programs corresponding to each function and various information. Furthermore, the processing circuit acquires two-dimensional or three-dimensional medical image data from a medical image diagnostic device or image storage device, such as an ultrasound diagnostic device, via a network, and performs the aforementioned processing using the acquired medical image data. Here, the processing circuit is a processor that implements the functions corresponding to each program by reading and executing programs from memory.

[0126] The term "processor" used in the above description refers to circuits such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), Application Specific Integrated Circuits (ASICs), and programmable logic devices (e.g., simple programmable logic devices (SPLDs), complex programmable logic devices (CPLDs), and field-programmable gate arrays (FPGAs)). The processor performs its function by reading and executing a program stored in memory. Alternatively, instead of storing the program in memory, the program can be directly loaded into the processor's circuitry. In this case, the processor performs its function by reading and executing the program loaded into the circuitry. Furthermore, the processors in this embodiment are not limited to a single circuit configuration; multiple independent circuits can be combined to form a single processor to perform its function.

[0127] Furthermore, the constituent elements of the devices illustrated in the above embodiments are functional concepts and do not necessarily need to be physically configured as shown in the illustrations. That is, the specific form of the distribution / combination of the devices is not limited to the form shown in the illustrations, and they can be configured, in any unit, functionally or physically, in a distributed / combined manner depending on various loads or usage conditions. Moreover, the processing functions performed by each device can be implemented, in whole or in part, by a CPU and the program parsed and executed by that CPU, or implemented as hardware as interconnect logic.

[0128] Furthermore, the processing method described in the above embodiments can be implemented by executing a pre-prepared processing program by a computer such as a personal computer or workstation. This processing program can be distributed via a network such as the Internet. Alternatively, the processing program can be recorded on a non-transitory recording medium that can be read by a computer, such as a hard disk, floppy disk (FD), CD-ROM, MO, DVD, USB memory, or SD card memory, and executed by the computer after reading it from the non-transitory recording medium.

[0129] Several embodiments of the present invention have been described, but these embodiments are merely illustrative and not intended to limit the scope of the invention. These embodiments can be implemented in a wide variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the present invention and its equivalents.

Claims

1. An image processing method, which uses a trained deep learning model to perform inference processing on image data, characterized in that, Includes the following steps: Receiving steps: Receiving images; The steps are defined to determine the first and second portions of the received image; The reasoning steps involve using the deep learning model to perform a first reasoning process on the determined first part and a second reasoning process on the determined second part, wherein the first reasoning process and the second reasoning process are different reasoning processes. as well as The integration step combines the results of the first and second reasoning processes and outputs the integrated reasoning result. The inference process is used to segment the image; The determination step uses the deep learning model to segment the image, obtains the inference result index of each pixel in the image representing the segmentation result, and determines the region formed by pixels whose inference result index is between a specified lower limit and a specified upper limit as the first part, and determines the other regions as the second part. The reasoning step performs enhanced segmentation on the first part based on enhanced reasoning, uses the result as the reasoning result of the first part, and directly uses the segmentation result of the second part in the determination step as the reasoning result of the second part.

2. The image processing method as described in claim 1, characterized in that, The first part is the key region in the segmentation of the image, and the second part is the non-key region in the segmentation of the image.

3. The image processing method as described in claim 2, characterized in that, The inference result index is the inference probability.

4. The image processing method as described in claim 2, characterized in that, The inference result indicator is the inference uncertainty.

5. The image processing method as described in claim 2, characterized in that, The enhanced reasoning employs at least one of the following methods: The image blocks used in the inference process are divided with the key region as the center. The image patch containing the key region is inverted and inference is performed again. as well as Increase the overlap rate between image blocks when performing inference processing on the key region.

6. The image processing method according to any one of claims 2 to 5, characterized in that, It also includes a correction step, which corrects the integrated reasoning result by adjusting the key areas.

7. An image processing system that uses a trained deep learning model to perform inference processing on image data, characterized in that, have: Receiving device, receiving images; The determining device determines a first portion and a second portion of the received image; The inference device uses the deep learning model to perform a first inference process on the determined first part and a second inference process on the determined second part, wherein the first inference process and the second inference process are different inference processes. as well as The integration device combines the results of the first and second inference processes and outputs the integrated inference result. The inference process is used to segment the image; The determining device uses the deep learning model to segment the image, obtains an inference result index for each pixel in the image representing the segmentation result, and determines the region formed by pixels whose inference result index is between a specified lower limit and a specified upper limit as the first part, and determines the other regions as the second part. The inference device performs enhanced segmentation based on enhanced reasoning on the first part, uses the result as the inference result of the first part, and directly uses the segmentation result of the second part in the determining device as the inference result of the second part.

8. A computer-readable storage medium storing an image processing program that uses a trained deep learning model to perform inference processing on image data, characterized in that, The program causes the computer to perform the following steps: Receiving steps: Receiving images; The steps are defined to determine the first and second portions of the received image; The reasoning steps involve using the deep learning model to perform a first reasoning process on the determined first part and a second reasoning process on the determined second part, wherein the first reasoning process and the second reasoning process are different reasoning processes. as well as The integration step combines the results of the first and second reasoning processes and outputs the integrated reasoning result. The inference process is used to segment the image; The determination step uses the deep learning model to segment the image, obtains the inference result index of each pixel in the image representing the segmentation result, and determines the region formed by pixels whose inference result index is between a specified lower limit and a specified upper limit as the first part, and determines the other regions as the second part. The reasoning step performs enhanced segmentation on the first part based on enhanced reasoning, uses the result as the reasoning result of the first part, and directly uses the segmentation result of the second part in the determination step as the reasoning result of the second part.

Images