Image segmentation device and image segmentation method

Abstract

The invention provides an image segmentation apparatus and method for segmenting a plurality of segmentation targets included in an image. The image segmentation device is provided with: a variable field-of-view function calculation means for calculating a variable field-of-view function capable of adaptively generating a field-of-view range of a corresponding size with respect to the plurality of segmentation targets; an image block generation means for generating image blocks of corresponding sizes by using the variable field function calculated by the variable field function calculation means for the plurality of segmentation targets; and an inference mechanism for inferring the image by using a segmentation model trained by using the image blocks generated by the image block generation mechanism to obtain segmentation results of the plurality of segmentation targets.

Description

Technical Field

[0001] The present invention relates to an image segmentation device and an image segmentation method, and in particular to an image segmentation device and a method based on an adaptive FOV method suitable for multi-scale target segmentation. Background Art

[0002] In medical image analysis, human organs and lesions often exhibit multi-scale characteristics, specifically, the target size distribution is wide. For example, the pulmonary trachea has about 24 levels of branches from the first level of bronchus to the final level of alveoli. The scale distribution of lung nodules may range from 0 to 30 mm or even larger, while the size of kidney tumors can range from less than 3 cm to about 20 to 30 cm in diameter. How to improve the segmentation effect of targets of different scales has always been an important topic in medical image segmentation processing.

[0003] Medical image segmentation can be divided into two methods: fully automatic and semi-automatic. Fully automatic medical image segmentation is a technology that automatically obtains segmentation results (such as tumors, organs, tissues, etc.) directly from the model after inputting a specified image. In the task of fully automatic medical image segmentation, the segmentation targets often have large differences in shape, scale, and position. It is difficult to obtain accurate segmentation results for these segmentation targets with large differences in shape, scale, and position.

[0004] On the other hand, semi-automatic medical image segmentation is more commonly used. Semi-automatic medical image segmentation refers to a technology in which, for example, the user first manually specifies the segmentation position or range, and then the model segments the specified position or range. In the task of semi-automatic medical image segmentation, specifying the segmentation range is the key. If a too large field of view (FOV) is specified, the segmentation accuracy of small targets is insufficient, resulting in a waste of computing resources; on the contrary, if a too small FOV is specified, it is easy to have incomplete target coverage.

[0005] In the prior art, some measures have been taken to try to solve the above problems for image segmentation. These measures include, for example, the sliding window technology, which divides the entire image into fixed-size patches in the form of a sliding window, performs reasoning, and integrates the reasoning results of each patch to obtain the segmentation results of all targets. As other measures, hierarchical segmentation can also be listed, that is, using models with different segmentation accuracies from coarse to fine. For example, two images with different resolutions are usually used to train different models. The coarse model generally has a lower resolution and a larger FOV, which is better for segmenting large targets. The fine model generally has a higher resolution and a smaller FOV, which is better for segmenting small targets.

[0006] Prior art literature:

[0007] Patent document: CN109598728A

[0008] Non-patent literature:

[0009] Non-patent literature 1: "A3D Coarse-to-Fine Framework for Volumetric Medical Image Segmentation," 2018 International Conference on 3D Vision (3DV), 2018, pp. 682-690, doi: 10.1109 / 3DV.2018.00083.

[0010] Non-patent literature 2: "nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation" Nat Methods 18, 203–211 (2021). (https: / / doi.org / 10.1038 / s41592-020-01008-z)

[0011] Non-patent literature 3: "L-SNet: from Region Localization to Scale Invariant Medical Image Segmentation" Jiahao Xie, Sheng Zhang, Jianwei Lu, Ye Luo (https: / / arxiv.org / abs / 2102.05971) Summary of the invention

[0012] Problems in the prior art

[0013] As mentioned above, no matter which method is used, the existing technology usually adopts the same processing method for all targets when performing image segmentation, without considering the scale differences of the segmented targets. This approach will bring the following technical problems, namely: for large segmentation targets, the FOV is prone to be insufficient, resulting in incomplete segmentation; and for small segmentation targets, the FOV is prone to be too large, resulting in insufficient segmentation accuracy.

[0014] On the other hand, in traditional medical imaging technologies, such as X-ray diagnosis, endoscopic photography and other related technologies, the size of the imaging area can be changed by moving the diagnostic bed or the photographic device to change the zoom processing of the photographic distance. In the doctor's daily reading of films, the appropriate observation field of view can also be selected by zooming in or out of the image.

[0015] Therefore, it is hoped that there will be such a technology, that is, in the processing of medical image segmentation based on machine learning, the FOV can be adaptively selected according to the scale of different segmentation targets, so as to automatically enlarge small targets and reduce large targets in the observation field of view, just like the behavior of doctors reading films, so as to obtain high-precision segmentation results.

[0016] The object of the present invention is to provide an image segmentation device and method based on an adaptive FOV method for segmenting multi-scale targets, which can adaptively select a suitable FOV for segmentation according to the scale of the segmented target.

[0017] Means for solving technical problems

[0018] The present invention is made to solve the problems of the above-mentioned prior art. The present invention proposes an adaptive FOV device and method for multi-scale target segmentation, including: an adaptive FOV calculation unit: an adaptive FOV function is obtained by calculating and fitting target features of different scales. An image block generation unit generates image blocks of different FOVs for targets of different scales. A model training unit uses image blocks of different FOVs to train models in an adaptive manner for targets of different scales. A model inference unit obtains an initial segmentation result through a coarse model, calculates the optimal FOV for the initial segmentation result, and then uses a fine model to further infer the optimal FOV to obtain a final inference result.

[0019] Specifically, according to one aspect of the present invention, there is provided an image segmentation device for segmenting a plurality of segmentation targets contained in an image, and characterized in that it comprises: a variable field of view function calculation mechanism for calculating a variable field of view function, wherein the variable field of view function can adaptively generate a field of view range of corresponding sizes for the plurality of segmentation targets; an image block generation mechanism for generating image blocks of corresponding sizes for the plurality of segmentation targets using the variable field of view function calculated by the variable field of view function calculation mechanism; and an inference mechanism for inferring the image using a segmentation model trained by using the image blocks generated by the image block generation mechanism, so as to obtain segmentation results for the plurality of segmentation targets.

[0020] According to another aspect of the present invention, there is provided an image segmentation method for segmenting a plurality of segmentation targets contained in an image, and the method is characterized in that it comprises: a variable field of view function calculation step, for calculating a variable field of view function, wherein the variable field of view function can adaptively generate a field of view range of corresponding sizes for the plurality of segmentation targets; an image block generation step, for generating image blocks of corresponding sizes for the plurality of segmentation targets using the variable field of view function calculated in the variable field of view function calculation step; and an inference step, for inferring the image using a segmentation model trained by using the image blocks generated in the image block generation step, to obtain segmentation results for the plurality of segmentation targets.

[0021] According to the present invention, a variable field of view function is calculated for segmentation targets of different sizes, and the generated variable field of view function is used to generate image blocks with sizes corresponding to the scales of each segmentation target, and the image is segmented on this basis. Therefore, the present invention is suitable for segmentation tasks with large differences in the scales and shapes of segmentation targets, and can solve the technical problems of insufficient segmentation accuracy for small targets and incomplete segmentation for large targets, and obtain high-precision segmentation results.

[0022] In addition, the present invention can be applied to semi-automatic segmentation algorithms and fully automatic segmentation algorithms, and the variable field of view function calculation mechanism of the present invention can be applied separately to other detection and segmentation models. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a block diagram showing an example of the structure of the image segmentation device related to the present invention.

[0024] Figure 2 It is a comparison diagram for explaining the characteristics of the processing executed by the image segmentation device related to the present invention.

[0025] Figure 3 : is a flowchart showing the image segmentation process executed by the image segmentation device according to the first embodiment of the present invention.

[0026] Figure 4A 1 is a flowchart showing a process of calculating a variable field of view function by the image segmentation device according to the first embodiment of the present invention.

[0027] Figure 4B FIG. 1 is a schematic diagram showing statistics used when the image segmentation device according to the first embodiment of the present invention calculates a variable field of view function.

[0028] Figure 5A FIG. 1 is a schematic diagram showing a first mode of processing for calculating a variable field of view function by the image segmentation device according to the first embodiment of the present invention.

[0029] Figure 5B FIG. 1 is a schematic diagram showing a second aspect of the process of calculating the variable field of view function by the image segmentation device according to the first embodiment of the present invention.

[0030] Figure 6 FIG. 1 is a schematic diagram showing a process of generating image blocks by the image segmentation device according to the first embodiment of the present invention.

[0031] Figure 7 FIG. 1 is a schematic diagram showing an inference process in image segmentation processing according to the first embodiment of the present invention.

[0032] Figure 8: is a flowchart showing the image segmentation process executed by the image segmentation device according to the second embodiment of the present invention. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the image segmentation device, image segmentation method, and storage medium of the present invention will be described in detail with reference to the accompanying drawings. In addition, the image segmentation device, image segmentation method, and storage medium of the present invention are not limited to the embodiments shown below. In the following description, common reference numerals are given to the same components and repeated descriptions are omitted.

[0034] First, the image segmentation device of the present invention is briefly described. The image segmentation device of the present invention can exist in the form of a medical image diagnosis device such as an ultrasonic diagnosis device, a CT imaging device, an MRI imaging device, or can exist independently in the form of a workstation or the like.

[0035] Figure 1 1 is a block diagram showing an example of the structure of the image segmentation device of the present invention. The image segmentation device 1 of this embodiment uses a deep learning neural network to segment the segmentation target contained in the image. Figure 1 As shown, the image segmentation device 1 mainly comprises a variable field of view function calculation mechanism 10, an image block generation mechanism 20, a training mechanism 30 and an inference mechanism 40. The variable field of view function calculation mechanism 10 is used to receive an image of a processing object, and calculate a variable field of view function with a statistical value obtained by statistically analyzing the shape features of multiple segmentation targets of different sizes as a variable, and the variable field of view function is used to adaptively generate a field of view range of corresponding sizes for the above-mentioned multiple segmentation targets. The image block generation mechanism 20 generates image blocks with sizes corresponding to the scales of the segmentation targets using the variable field of view function calculated by the variable field of view function calculation mechanism 10 for the segmentation targets. The training mechanism 30 trains the segmentation model using the image blocks generated by the image block generation mechanism 20. The inference mechanism 40 uses the segmentation model trained by the training mechanism 30 to infer the image and obtain the final segmentation result of the segmentation target.

[0036] The image segmentation device 1 of the present invention can be equipped in an image segmentation device such as an ultrasonic diagnostic device, for example. In this case, the image segmentation device 1 also includes a control unit, an ultrasonic probe, a display, an input / output interface, and a device body, which are omitted from the figure. The variable field of view function calculation mechanism 10, the image block generation mechanism 20, the training mechanism 30, and the reasoning mechanism 40 are equipped in the control unit and are communicatively connected to these ultrasonic probes, displays, input / output interfaces, and the device body, etc. Since the structures and functions of the control unit, ultrasonic probe, display, input / output interface, and the device body are already well known to those skilled in the art, their detailed description is omitted.

[0037] Next, the processing executed by the image segmentation device 1 according to the present embodiment will be described in detail.

[0038] Figure 2 It is a comparison diagram for explaining the characteristics of the processing executed by the image segmentation device related to the present invention.

[0039] Figure 2 The features of the processing of the present invention are explained by comparing it with the flow of the prior art. The left side is a flow chart of the image processing of the prior art, and the right side is a flow chart of the processing performed by the image segmentation device of the present invention.

[0040] like Figure 2 As shown, in the process performed by the image segmentation device of the present invention, steps S300 and S400 in the training process, and steps S700 and S800 in the reasoning process are characteristic steps of the present invention, wherein, unlike the prior art in which the training of the fine model and the coarse model are also generated by randomly selecting and cutting the training image patches, the present invention adds a step S300 of calculating the adaptive FOV function (variable field of view function) in the training of the fine model, and then in step S400, the adaptive FOV function calculated in step S300 is used to generate an image patch with a size suitable for the segmentation target scale for training. Similarly, in the reasoning process, the segmentation process of the present invention adds step S700 in the fine model reasoning, in which the variable field of view function is also used to obtain an image patch of a suitable size based on the coarse reasoning result, and then the image patch of the suitable size is used to perform reasoning based on the fine model. The processing of steps S700 and S800 in the reasoning process is not necessary for the present invention and can be appropriately omitted. The specific situation will be described in detail later.

[0041] Through the processing in the above-mentioned characteristic steps, the image segmentation device of the present invention calculates a variable field of view function with the statistics of the shape features of multiple segmentation targets as variables for the multi-scale segmentation targets in the image, and uses the generated variable field of view function to generate image blocks of corresponding sizes, and performs image segmentation on this basis. Therefore, the present invention is suitable for segmentation tasks with large differences in the scale and shape of segmentation targets, and can solve the technical problems of insufficient segmentation accuracy of small targets and incomplete segmentation of large targets, and obtain high-precision segmentation results.

[0042] The above is an overview of the processing performed by the image segmentation device of the present invention. The present invention will be described in detail below using semi-automatic segmentation and fully automatic segmentation as examples.

[0043] (First embodiment)

[0044] The first embodiment is a case where the present invention is applied to semi-automatic segmentation.

[0045] Figure 3 1 is a flowchart showing a first embodiment of the image segmentation process performed by the image segmentation device of the present invention. The image segmentation process can be divided into two main parts: a training process and an inference process. First, the training process is described with reference to the accompanying drawings.

[0046] During the training process, first in step S10 , the image segmentation device 1 receives an image to be segmented and a label image of a target contained in the image for training.

[0047] Subsequently, the image segmentation device 1 performs training of a coarse model through steps S100 and S200.

[0048] Specifically, in step S100, the image segmentation device 1 may, for example, first select a foreground / background area to be cropped at a certain ratio, and then crop the selected area to be cropped at a fixed resolution to obtain a patch image and a patch label. Next, the image segmentation device performs preprocessing including normalization and other operations to generate training image blocks.

[0049] Then, in step S200, the image segmentation device 1 trains the coarse segmentation model using the patch images and patch labels obtained in step S100.

[0050] In this embodiment, the coarse segmentation model may be, for example, a 3D U-Net model.

[0051] On the other hand, the image segmentation device 1 of the present invention performs training of a detailed model through steps S300 to S500.

[0052] Specifically, in step S300, the image segmentation device 1 calculates a variable field of view function through a variable field of view function calculation mechanism 10. The variable field of view function is a function that uses a statistical value obtained by performing statistics on shape features of multiple segmentation targets of different scales as a variable, and is used to adaptively generate a field of view range of corresponding size for multiple segmentation targets of different scales.

[0053] The following combination Figure 4A , Figure 4B , Figure 5A , Figure 5B The details of step S300 will be described in detail.

[0054] Figure 4A and Figure 4B is a schematic diagram for illustrating the process of calculating an adaptive FOV function, wherein Figure 4A is a flowchart showing a process of calculating a variable field of view function by the image segmentation device according to the first embodiment of the present invention. Figure 4BFIG. 1 is a schematic diagram showing statistics used when the image segmentation device according to the first embodiment of the present invention calculates a variable field of view function.

[0055] like Figure 4A As shown, step S300 may include step S301 and step S302, and the variable field of view function calculation mechanism 10 calculates the variable field of view function through steps S301 and S302.

[0056] Specifically, the variable field of view function calculation mechanism 10 first obtains statistics on the shape features of each segmentation target in the training set image in step S301, where the shape features are features related to the shape of each segmentation target, preferably scale features related to the scale of each segmentation target. Specifically, the scale features may include the length of the bounding box of the segmentation target in each direction in the three-dimensional space, the volume size of the segmentation target, the radius of the segmentation target, the major and minor diameters, etc. Figure 4B The distribution of scale characteristics of each target is shown as an example of statistics, where Figure 4B The left side shows the distribution examples of the lengths of the bounding boxes of each target in the z-axis, y-axis, and x-axis directions, namely, len_z, len_y, and len_x. Figure 4B An example of the distribution of the volume of each target is shown on the right side.

[0057] The scale feature of the segmented target is an example of the shape feature. The length of the segmented target's bounding box in each direction in the three-dimensional space, the volume of the segmented target, the radius of the segmented target, the major and minor diameters, etc. are examples of the scale feature. The variable field of view function of the present invention is generated based on the statistics obtained by statistically analyzing the shape features of the segmented target, and the statistics are used as variables. In the generation of the variable field of view function, it is only necessary to use the shape features related to the shape of each segmented target and analyze the shape features as follows: Figure 4B The required statistics can be obtained by analyzing the data distribution statistics shown.

[0058] The details of the shape feature statistics in step S301 are described below in conjunction with Figure 5A and Figure 5B The calculation process of the adaptive FOV function will be clearer if explained.

[0059] Return to Figure 4A Description.

[0060] After step S301 , the variable viewing function calculation mechanism 10 calculates the variable viewing function according to the statistics of the shape features obtained in step S301 in step S302 .

[0061] The variable field of view function calculation mechanism 10 calculates the variable field of view function in step S302, which may include at least two methods: directly fitting the variable field of view function from the statistics of the shape features of the segmented target in the data set, and obtaining the variable field of view function by learning the statistics of the shape features.

[0062] The following combination Figure 5A and Figure 5B The processing details of step S302 in the above two modes are described in detail.

[0063] Figure 5A FIG. 1 is a schematic diagram of a first mode of processing for calculating a variable field of view function by the image segmentation device according to the first embodiment of the present invention.

[0064] like Figure 5A As shown, when the image segmentation device according to the first embodiment of the present invention calculates the variable field of view function, as a first method, for example, the variable field of view function calculation mechanism 10 designs a variable field of view function by fitting in step S302, with the statistics of the shape features of each target contained in the data set obtained in step S301 as variables and the FOV size as the function value, and the parameters of the variable field of view function are set according to the statistics.

[0065] The variable field of view function calculation mechanism 10 preferably uses a monotonically increasing linear function as the variable field of view function, that is, the variable field of view function is designed so that the FOV size increases as the statistics of the shape characteristics of the target increase. More preferably, the variable field of view function calculation mechanism 10 uses Figure 5A The piecewise linear function shown in FIG. 1 remains unchanged when the variable is extremely small or extremely large, and remains monotonically increasing when the variable is between the extremely small and extremely large values ​​as a variable field of view function.

[0066] Figure 5A In the first embodiment of the present invention shown, as the statistic used in generating the piecewise linear function, the average value mean(len_z, len_y, len_x) of the lengths of the bounding box of each target in the z-axis, y-axis, and x-axis directions, i.e., len_z, len_y, len_x, is used. That is, as an example, Figure 5AThe piecewise linear function shown in is a function FOV with the average value mean(len_z,len_y,len_x) of len_z, len_y, and len_x of each target as a variable and the size of the field of view as a function value, wherein the parameters of the function FOV are set according to statistics so that it has the following characteristics, namely: near the two ends of the horizontal axis, that is, when the average value mean(len_z,len_y,len_x) of len_z, len_y, and len_x is smaller than the minimum value or larger than the maximum value, the size of the field of view remains unchanged, and in the middle part of the horizontal axis, that is, when the average value mean(len_z,len_y,len_x) of len_z, len_y, and len_x is between the minimum value and the maximum value, the size of the field of view increases monotonically and linearly with the increase of the average values ​​of len_z, len_y, and len_x.

[0067] The piecewise linear function FOV with the above characteristics can be expressed by the following formula (1):

[0068] FOV=max(a,min(b,mean(len_z,len_y,len_x))) / c×d (1)

[0069] Among them, FOV is the size of the field of view, len_z, len_y, and len_x are the lengths of the segmented target in the z direction, y direction, and x direction in the three-dimensional space, respectively, and the parameters a, b, c, and d are adjustable hyperparameters related to the statistics of the scale feature (mean(len_z, len_y, len_x) in this example).

[0070] In formula (1), a and b represent the minimum and maximum values ​​of the statistic variable at the two inflection points of the piecewise linear function, respectively, c represents the middle value of the statistic, b>c>a, and parameter d is used to ensure that there is an appropriate space between the field of view FOV and the segmentation target, and is named as the space adjustment parameter in the present invention.

[0071] The following describes in detail how the piecewise linear function illustrated by equation (1) can adaptively generate a field of view range of corresponding size for segmentation targets of different scales.

[0072] like Figure 5A As shown, according to formula (1), when mean(len_z,len_y,len_x)>b, that is, when the average value of len_z, len_y, and len_x is greater than the maximum value b, min(b,mean(len_z,len_y,len_x)) in formula (1), that is, taking the minimum value of b and mean(len_z,len_y,len_x), is:

[0073] min(b,mean(len_z,len_y,len_x))=b

[0074] At this time, since b>a, the result of max(a,min(b,mean(len_z,len_y,len_x))) in formula (1), that is, taking the maximum value of a and min(b,mean(len_z,len_y,len_x))=b, is:

[0075] max(a,min(b,mean(len_z,len_y,len_x)))=max(a,b)=b

[0076] That is, when mean(len_z,len_y,len_x)>b, that is, when the average value of len_z, len_y, and len_x is greater than the maximum value b, equation (1) becomes:

[0077] FOV=max(a,min(b,mean(len_z,len_y,len_x))) / c×d

[0078] =b / c×d

[0079] On the other hand, when mean(len_z,len_y,len_x)<a, that is, the average value of len_z, len_y, and len_x is smaller than the minimum value a, since b>a, it is inevitable that mean(len_z,len_y,len_x)<b. Therefore, min(b,mean(len_z,len_y,len_x)) in formula (1) becomes:

[0080] min(b,mean(len_z,len_y,len_x))=mean(len_z,len_y,len_x)

[0081] Since mean(len_z,len_y,len_x)<a, therefore,

[0082] max(a,min(b,mean(len_z,len_y,len_x)))=a

[0083] That is to say, when mean(len_z,len_y,len_x)<a, that is, when the average value of len_z, len_y, and len_x is smaller than the minimum value a, equation (1) will become:

[0084] FOV=max(a,min(b,mean(len_z,len_y,len_x))) / c×d

[0085] =a / c×d

[0086] When mean(len_z,len_y,len_x) is between a and b, that is, when the average value of len_z, len_y, and len_x is between the minimum value a and the maximum value b, since mean(len_z,len_y,len_x)≤b, therefore,

[0087] min(b,mean(len_z,len_y,len_x))=mean(len_z,len_y,len_x)

[0088] Since mean(len_z,len_y,len_x)≥a, therefore,

[0089] max(a,min(b,mean(len_z,len_y,len_x)))=mean(len_z,len_y,len_x)

[0090] That is, when a≤mean(len_z,len_y,len_x)≤b, that is, when the average values ​​of len_z, len_y, and len_x are between the minimum value a and the maximum value b, equation (1) becomes:

[0091] FOV=max(a,min(b,mean(len_z,len_y,len_x))) / c×d

[0092] =mean(len_z,len_y,len_x) / c×d

[0093] Therefore, formula (1) can be simplified to the following piecewise function:

[0094]

[0095] That is, the variable field of view function FOV illustrated by formula (1) is a piecewise linear function with the average value mean(len_z,len_y,len_x) of len_z, len_y, and len_x as a variable. When the variable mean(len_z,len_y,len_x) is smaller than the minimum value a or larger than the maximum value b, the size of the field of view is fixed at a / c×d and b / c×d, respectively. When the variable mean(len_z,len_y,len_x) is between the minimum value a and the maximum value b, the size of the field of view increases linearly with the increase of mean(len_z,len_y,len_x).

[0096] The following describes in detail the method for setting the hyperparameters a, b, c, and d in the first embodiment of the present invention.

[0097] Figure 5A In the first embodiment of the present invention shown, the minimum value a is, for example, the length value at the 10% quantile of the length distribution of all segmented targets in the training set, the maximum value b is, for example, the length value at the 90% quantile of the length distribution of all segmented targets in the training set, and the middle value c is, for example, the length value at the median of the length distribution of all segmented targets in the training set. Corresponding to the middle value c being the median, the spatial adjustment parameter d is the product of the image median resolution IR and the matrix size patch_size of the deep learning model input, that is, d = IR × patch_size.

[0098] Taking the training data set with a length distribution of 1mm to 100mm with a total of 100 consecutive integer lengths, IR of 1mm / pixel, and patch_size of 96 pixels as an example, at this time, a=10mm, b=90mm, c=50mm, then according to the above analysis, when the length of the target to be segmented is small (1-10mm), FOV=a / c×d=a / c×(IR×patch_size)=10mm / 50mm×(1mm / pixel×96 pixels)=19.2mm, when the length of the target to be segmented is large (90-100mm), FOV=b / c×d=b / c×(IR×patch_size)=90mm / 50mm×(1mm / pixel×96 pixels)=172.8mm, when the length of the target to be segmented is between 10 and 90mm, FOV is a value between 19.2mm and 172.8mm that varies proportionally with the length of the target to be segmented. Thus, the adaptive FOV function of the present invention becomes as follows Figure 5A The piecewise linear function shown in the middle below has a smaller FOV when the object to be segmented is smaller ( Figure 5A Left side below), when the size of the target to be segmented is large, the corresponding FOV is also large ( Figure 5A There is a suitable space between the target to be segmented and the corresponding FOV. By generating such an adaptive FOV function, a FOV of a suitable size that is adapted to the scale of the target to be segmented can be obtained.

[0099] The specific settings of the above hyperparameters a, b, c, and d are only examples, and the present invention is not limited thereto.

[0100] For example, in the above example, a selects the length value at the 10% quantile of the length distribution of all segmented targets in the training set, b selects the length value at the 90% quantile of the length distribution of all segmented targets in the training set, and c selects the length value at the median of the length distribution of all segmented targets in the training set. However, a and b, which represent the minimum and maximum values ​​of the horizontal coordinates at the two inflection points of the piecewise linear function, can obviously take other values. For example, a can take the length value at the 15% quantile, 20% quantile, etc. of the length distribution of all segmented targets in the training set, and b can take the length value at the 85% quantile, 80% quantile, etc. of the length distribution of all segmented targets in the training set. Similarly, in addition to the median, parameter c can also take the average value of the length distribution of all segmented targets in the training set or other values ​​that reflect the overall average value of the scale feature. In the above example, d selects the product of the image median resolution IR (for example, 1 mm / pixel) and the matrix size patch_size (for example, 96 pixels) of the deep learning model input. The parameter d is used to ensure that there is an appropriate space between the field of view and the segmentation target. In the above example, the image median resolution IR is set to 1 mm / pixel, and the matrix size patch_size of the deep learning model input is set to 96 pixels, but the present invention is obviously not limited to this. The image median resolution IR and the matrix size patch_size of the deep learning model input can be set to other values ​​as needed. In addition, in the first embodiment of the present invention, the scale feature of the target to be segmented is set to the length of the bounding box, and the parameter d correspondingly selects d=IR×patch_size, but when the FOV function is set with other scale features other than the length of the bounding box as variables, the parameter d can take other corresponding values. Hyperparameters a, b, c, and d can be applicable to the present invention as long as they are adjustable hyperparameters related to the statistics of the scale feature.

[0101] In addition, in the above example of the first embodiment of the present invention, the average value mean(len_z, len_y, len_x) of the lengths of the bounding box of each target in the z-axis, y-axis, and x-axis directions, i.e., len_z, len_y, and len_x, is used as a statistic, and the average value is used as a variable and the variable field of view function is designed based on the average value, but the present invention is not limited to this. In addition to the lengths of the bounding box of each target in the z-axis, y-axis, and x-axis directions, the radius of each target can also be added to calculate the average value, that is, the average value mean(len_z, len_y, len_x, r) of the lengths of the bounding box of each target in the z-axis, y-axis, and x-axis directions and the radius, i.e., len_z, len_y, len_x, and r, is used as a statistic to design the variable field of view function. At this time, the piecewise linear function of the variable field of view function of the present invention can be expressed by the following formula (2):

[0102] FOV=max(a,min(b,mean(len_z,len_y,len_x,r) / c.))×d (2)

[0103] FOV, len_z, len_y, len_x, a, b, c, and d in formula (2) are the same as those in the implementation described above for formula (1), and r is the radius of the segmented target.

[0104] In addition, in addition to the mean value, the median, mode, quartile, etc. can be used as statistics as appropriate according to the situation. Similarly, the parameters and function form of the variable field of view function itself can also have various deformations. As long as the increasing linear function is designed by fitting with the statistics of the shape characteristics of the target to be segmented as the variable, it can be applied to the present invention.

[0105] The above is a description of the first method of calculating the variable field of view function by the image segmentation device according to the first embodiment of the present invention. The following describes a second method of calculating the variable field of view function by the image segmentation device according to the first embodiment of the present invention.

[0106] Figure 5B FIG. 1 is a schematic diagram of a second embodiment of the process of calculating the variable field of view function by the image segmentation device according to the first embodiment of the present invention.

[0107] like Figure 5B As shown, when calculating the variable field of view function, the image segmentation device according to the first embodiment of the present invention uses a neural network as a second method to obtain the variable field of view function by automatic fitting.

[0108] For example, in step S302, the variable field of view function calculation mechanism 10 can use the statistics of the shape features obtained in step S301 as the input of the neural network, the optimal FOV as the prediction target, and the difference between the predicted value and the optimal FOV as the loss function for optimization, thereby obtaining the FOV function.

[0109] In the second method, the statistic of the shape feature used as the input of the neural network can be the average value of len_z, len_y, and len_x, or the average value of len_z, len_y, len_x, and r, etc., as in the first method. In the second method, the loss function Loss used as the input of the neural network can be L1 loss or L2 loss.

[0110] Return to the following Figure 3 Description of the process.

[0111] After step S300 , the process proceeds to step S400 . In step S400 , the image segmentation device 1 generates a training image block having a size adapted to the scale of the object to be segmented using the variable field of view function calculated in step S300 .

[0112] The following combination Figure 6 The details of step S400 will be described in detail.

[0113] Figure 6 FIG. 1 is a schematic diagram of a process of generating image blocks of appropriate sizes by the image segmentation device according to the first embodiment of the present invention.

[0114] like Figure 6 As shown, step S400 includes steps S401 to S403, and the training mechanism 30 generates training image blocks of appropriate sizes through steps S401 to S403. Specifically, the training mechanism 30 first randomly selects n objects to be segmented in the training set in step 401, Figure 6 The example shows three targets to be segmented with obvious differences in scale: target 1 to be segmented, target 2 to be segmented, and target n to be segmented. Then, in step S402, the training institution 30 uses the variable field of view function calculated in step S300 to substitute the shape features of each target to be segmented as a variable into the variable field of view function for target 1 to be segmented, target 2 to be segmented...target n to be segmented, and calculates the field of view range FOV1, FOV2...FOVn corresponding to each target to be segmented. The details and effects of using the variable field of view function to calculate the field of view range of appropriate size using the shape features of each target to be segmented as a variable in step S402 are described in combination with the above. Figure 5A and Figure 5B This has been detailed in the description and will not be repeated here.

[0115] Subsequently, in step S403, the training mechanism 30 generates image blocks 1, image blocks 2, ... image blocks n for training, respectively, for each target to be segmented 1, target to be segmented 2, ... target to be segmented n, using the field of view ranges FOV1, FOV2, ... FOVn of appropriate sizes calculated in step S402. The processing of generating image blocks in step S403 can adopt any method known in the art, for example, the image blocks can be cut with the center of gravity of the target to be segmented as the center and the field of view ranges FOV1, FOV2, ... FOVn as the target size, so as to obtain training image blocks with a size suitable for the scale of the target to be segmented. The specific details of the generation of training image blocks can be implemented in various ways in the prior art, and further detailed description is omitted here.

[0116] Return to the following Figure 3 Description of the process.

[0117] After step S400, the process proceeds to step S500. In step S500, the image segmentation device 1 uses the training image blocks of a suitable size adapted to the scale of the target to be segmented and the corresponding target label image obtained in step S400 to train a fine segmentation model. In the first embodiment of the present invention, 3D U-Net is preferably used as the fine segmentation model.

[0118] The deep learning model and neural network training can be implemented in various ways in the prior art, and the detailed description is omitted here.

[0119] The above is the description of the training process in the image segmentation process according to the first embodiment of the present invention. The following is a description of the inference process in the image segmentation process.

[0120] In the inference process of the image segmentation processing according to the first embodiment of the present invention, the inference unit 40 uses Figure 3 In steps S600 to S800 shown in FIG. 1 , the image is inferred by the segmentation model trained by the training mechanism 30 to obtain the final segmentation result.

[0121] As a preferred embodiment, the inference mechanism 40 may include a first segmentation mechanism and a second segmentation mechanism, wherein the first segmentation mechanism uses a coarse model (first segmentation model) to infer the image in step S600 to obtain a coarse inference result (first segmentation result), and the second segmentation mechanism uses a variable field of view function calculated by a variable field of view function calculation mechanism according to the coarse inference result (first segmentation result) in step S700 to generate a first segmentation result image block with a suitable size. Furthermore, the second segmentation mechanism uses a fine segmentation model (second segmentation model) to infer the first segmentation result image block in step S800 to obtain a final segmentation result.

[0122] Combine the following Figure 7 The inference process of the image segmentation processing according to the first embodiment of the present invention will be described in detail. Figure 7 It is a schematic diagram showing the detailed processing of the inference process in the image segmentation processing.

[0123] like Figure 7 As shown, in the reasoning process, the reasoning mechanism 40 first performs reasoning of the coarse model in step S600 to obtain a coarse reasoning result. The coarse reasoning process can be, for example, inputting the original image and the seed point into the coarse reasoning model for reasoning. Specifically, for example, the user can specify the target seed point to be segmented, cut a fixed-size image block with the seed point as the center and input it into the coarse segmentation model, and the coarse segmentation model performs prediction to obtain the coarse reasoning result.

[0124] The reasoning of the coarse model can be implemented in various ways in the prior art, and further detailed description is omitted here.

[0125] Subsequently, in step S700, the inference mechanism 40 uses the variable field of view function to calculate the field of view FOV of a suitable size for each rough inference result obtained in step S600, and generates an image block of a suitable size. The processing of step S700 is similar to the processing of step S400, especially steps S402 to S403 in the training process, and a detailed description is omitted here.

[0126] Then, in step S800, the inference mechanism 40 inputs the image block of appropriate size obtained in step S700 into the fine segmentation model, and the fine segmentation model performs prediction to obtain the final segmentation result.

[0127] As described above, the image segmentation device of the present invention aims at multi-scale segmentation targets in the image. First, during the training process, a variable field of view function with the statistics of the shape features of multiple segmentation targets as variables is calculated. During the inference process, the generated variable field of view function is used to generate image blocks of corresponding sizes. On this basis, the image is segmented using the model trained during the training process. Therefore, the present invention adaptively segments multi-scale targets to be segmented, which can solve the technical problems of insufficient segmentation accuracy of small targets and incomplete segmentation of large targets, and obtain high-precision segmentation results.

[0128] The above describes the inference process of the first embodiment of the present invention, but the present invention is not limited thereto, and the inference process of the image segmentation processing of this embodiment can be modified in various ways.

[0129] For example, a judgment step may be added after step S700 to judge whether the FOV calculation result in step S700 is similar to the FOV of the coarse model reasoning result in step S600. When the FOV calculation result in step S700 is similar to the FOV of the coarse model reasoning result in step S600, the fine model segmentation in step S800 may be skipped, and the coarse model segmentation result in step S600 may be directly used as the final segmentation result. At this time, the calculation result based on the adaptive FOV method of the present invention is used as a judgment reference in the judgment step, so that a high-precision segmentation result can also be obtained, and the beneficial effects of improving processing speed and saving computing resources can be achieved.

[0130] In addition, in some special cases, such as due to the special shape, scale or contrast of the segmentation target, the FOV calculation result in step S700 may not be the optimal result. In this case, if the segmentation result of step S800 is still used as the final segmentation result, good segmentation accuracy may not be obtained. Therefore, the present invention can also provide a mode that supports the selection algorithm, that is, adding a selection step after step S800 to select the better segmentation result of the coarse model reasoning result of step S600 and the fine model reasoning result of step S800 as the final segmentation result. The selection step can be performed, for example, by pre-setting a specified threshold for the technical indicator representing the result of the reasoning process, and comparing the indicator of the coarse model reasoning result and the indicator of the fine model reasoning with the threshold respectively, or it can be manually selected by the user of the image segmentation device 1 according to the specified standard.

[0131] The above-described judgment of whether the FOVs are similar and the judgment criteria, as well as the segmentation result selection algorithm and selection criteria can be implemented in various ways in the prior art, and are not the focus of the present invention, so detailed descriptions are omitted here.

[0132] According to the first embodiment of the present invention, the variable field of view function calculation mechanism calculates the adaptive FOV function by fitting or learning based on the statistics obtained by statistically analyzing the shape features of all targets in the entire data set, and generates the corresponding FOV based on the adaptive FOV function, rather than simply expanding the FOV of the coarse model outward by a certain size as in the prior art. Therefore, the FOV accuracy of the present invention does not depend on the accuracy of the coarse model. When the segmentation accuracy of the coarse model is low or fails, the FOV function obtained according to the present invention is still capable of processing and obtaining a suitable FOV. In addition, the shape features used in the present invention are not unique, and can be a single feature or multiple features, such as the length of the bounding box of the segmented target in each direction in three-dimensional space, the volume of the segmented target, the radius of the segmented target, the long and short diameters and other scale features, or other shape features that can characterize the scale or shape of the target.

[0133] According to the first embodiment of the present invention described above, multi-scale segmentation targets can be segmented adaptively, thereby solving the technical problems of insufficient segmentation accuracy for small targets and incomplete segmentation for large targets, and obtaining high-precision segmentation results.

[0134] Based on the technical solution of the first embodiment described above, the inventor tested the segmentation of medical images of advanced lung tumors and evaluated the segmentation results. When the Dice similarity coefficient (DSC) was used as the evaluation standard, the DSC of the segmentation result according to the first embodiment of the present invention was 0.774, compared with the DSC of the prior art of 0.706. The segmentation accuracy of the present invention is significantly better than that of the prior art.

[0135] (Second embodiment)

[0136] The above describes the first semi-automatic implementation method in which the original image and seed points are input into the inference model to perform inference in step S600, but the present invention is not limited thereto. For example, the present invention may also be used for fully automatic image segmentation.

[0137] The second embodiment is to apply the present invention to the case of fully automatic segmentation. Figure 8 The second embodiment of the present invention will be described in detail.

[0138] In the description of the second embodiment, the differences from the above-mentioned first embodiment will be mainly described. In the description of the second embodiment, the same reference numerals are given to the same configurations as those of the above-mentioned first embodiment, and the description thereof will be omitted.

[0139] Figure 8 : is a flowchart showing the image segmentation process executed by the image segmentation device according to the second embodiment of the present invention.

[0140] like Figure 8 As shown, in the reasoning process of the second embodiment of the present invention, in the processing of step S600', different from the processing of step S600 of the first embodiment, the second embodiment performs coarse model reasoning in a fully automatic manner through a sliding window form, that is, the sliding window traverses the entire original image to obtain image blocks, and reasoning is performed on each image block to obtain the reasoning result of each image block, and the reasoning results of each image block are integrated to obtain the coarse reasoning result of the entire image.

[0141] The other steps of the image segmentation process in the second embodiment are the same as those in the first embodiment, and detailed description thereof will be omitted here.

[0142] According to the second embodiment of the present invention, the present invention is applied to fully automatic segmentation, and can adaptively segment multi-scale segmentation targets, thereby solving the technical problems of insufficient segmentation accuracy for small targets and incomplete segmentation for large targets. As with the first embodiment, high-precision segmentation results can be obtained.

[0143] Based on the technical solution of the second embodiment described above, the inventor tested the segmentation of renal tumor medical images and evaluated the segmentation results. When the evaluation standard uses DSC, the DSC of the segmentation results of the prior art using a single coarse segmentation and a hierarchical segmentation from coarse to fine is 0.825 and 0.855 respectively. In comparison, the DSC of the segmentation result according to the second embodiment of the present invention is 0.867. In addition, for segmentation targets of different sizes, the DSC of the segmentation results of the prior art from coarse to fine hierarchical segmentation when the radius of the renal tumor is less than 20 mm and less than 40 mm is 0.768 and 0.838 respectively. In comparison, the DSC of the segmentation results according to the second embodiment of the present invention is 0.811 and 0.860 respectively. The segmentation accuracy of the present invention under different segmentation methods and different segmentation target scales is better than that of the prior art.

[0144] (Other embodiments)

[0145] In the reasoning process of the above-mentioned first and second embodiments, the processing based on the coarse model and the fine model of steps S600 (S600') to S800 is a preferred embodiment of the present invention, but the processing of steps S700 and S800 is not necessary, that is, in step S600 (S600'), the image can be inferred using the model trained by the training process of the present invention, and the obtained segmentation result can be directly used as the final reasoning result.

[0146] In addition, the above describes an implementation method of reasoning an image using a model trained by the training mechanism 30, but the coarse segmentation model in the reasoning process of the present invention can also be applied to any existing segmentation model, as long as it is a model that can obtain the scale information of the target to be segmented, such as a target detection model, a model based on a traditional threshold segmentation method, etc., and the coarse model segmentation can also be manual segmentation. The coarse segmentation model of the present invention can be any one of a deep learning segmentation model, a landmark model, a target detection model, and a graph cut segmentation model.

[0147] The above embodiment has been described by taking segmentation as an example, but the present invention can of course be applied to other image processing other than segmentation, such as detection.

[0148] The present invention can be applied to semi-automatic segmentation algorithms and fully automatic segmentation algorithms. Moreover, the variable field of view function calculation mechanism 10 of the present invention can be applied separately to other existing detection and segmentation models.

[0149] The image processing, segmentation, deep learning model and neural network training and reasoning described in the above embodiments can be implemented in various ways in the prior art, and detailed description is omitted here.

[0150] The present invention can be implemented not only as the above-mentioned image segmentation device, but also as an image segmentation method, a program, and a medium storing the image segmentation program.

[0151] The image segmentation device of the present application can be installed in a medical image diagnosis device, or the image segmentation device can perform processing alone. In such a case, the image segmentation device has a processing circuit that performs the same processing as the above steps, and a memory that stores programs and various information corresponding to each function. In addition, the processing circuit obtains two-dimensional or three-dimensional medical image data from a medical image diagnosis device such as an ultrasonic diagnostic device or an image storage device via a network, and uses the obtained medical image data to perform the above processing. Here, the processing circuit is a processor that realizes the functions corresponding to each program by reading and executing the program from the memory.

[0152] The term "processor" used in the above description refers to, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an Application Specific Integrated Circuit (ASIC), a programmable logic device (for example, a simple programmable logic device (SPLD), a complex programmable logic device (CPLD), and a field programmable gate array (FPGA)) and other circuits. The processor implements its functions by reading and executing a program stored in a memory. In addition, instead of storing the program in the memory, the program is directly loaded into the circuit of the processor. In this case, the processor implements its functions by reading and executing the program loaded into the circuit. In addition, each processor of the present embodiment is not limited to the case where the processor is configured as a single circuit, and a plurality of independent circuits are combined to form one processor to implement its functions.

[0153] In addition, the components of each device illustrated in the above-mentioned embodiments are conceptual functions and do not necessarily need to be physically configured as shown. That is, the specific form of the dispersion / integration of each device is not limited to the form shown in the figure, and all or part of it can be functionally or physically dispersed / integrated in arbitrary units according to various loads or usage conditions. Furthermore, the processing functions performed by each device can be implemented in whole or in part by the CPU and the program analyzed and executed by the CPU, or as hardware of wiring logic.

[0154] In addition, the processing method described in the above embodiment can be implemented by executing a pre-prepared processing program by a computer such as a personal computer or a workstation. The processing program can be distributed via a network such as the Internet. In addition, the processing program can also be recorded in a non-temporary recording medium that can be read by a computer, such as a hard disk, a floppy disk (FD), a CD-ROM, MO, a DVD, a USB memory, and a Flash memory such as an SD card memory, and executed by a computer reading it from the non-temporary recording medium.

[0155] Several embodiments of the present invention have been described, but these embodiments are provided as examples and do not limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the scope of the invention. These embodiments and their variations are included in the scope and purpose of the invention, and are also included in the scope of the present invention and its equivalents.

Claims

1. An image segmentation device for segmenting a plurality of segmentation targets contained in an image, characterized in that: have: A variable field of view function calculation mechanism is used to calculate a variable field of view function, wherein the variable field of view function can adaptively generate a field of view range of corresponding size for the plurality of segmentation targets; An image block generating mechanism, for the plurality of segmentation targets, using the variable field of view function calculated by the variable field of view function calculating mechanism to generate image blocks of corresponding sizes; as well as The inference unit uses a segmentation model trained by using the image blocks generated by the image block generation unit to infer the image, thereby obtaining segmentation results of the multiple segmentation targets.

2. The image segmentation device according to claim 1, characterized in that: The reasoning mechanism comprises: A first segmentation mechanism, which segments the image using a first segmentation model to obtain a first segmentation result; The second segmentation mechanism generates a first segmentation result image block using the variable field of view function according to the first segmentation result, and segments the first segmentation result image block using a second segmentation model to obtain the segmentation result.

3. The image segmentation device according to claim 1, characterized in that: The variable field of view function uses a statistic obtained by performing statistics on shape features of the plurality of segmentation targets of different sizes as a variable.

4. The image segmentation device according to claim 3, characterized in that: The shape feature is a scale feature related to the scale of the segmented object.

5. The image segmentation device according to claim 4, characterized in that: The scale feature includes at least one selected from the group consisting of the length of the segmented object in each direction in the three-dimensional space of the boundary box, the volume of the segmented object, and the major and minor axes of the segmented object.

6. The image segmentation device according to claim 5, characterized in that: The variable field of view function calculation mechanism uses a monotonically increasing linear function set according to the scale feature as the variable field of view function.

7. The image segmentation device according to claim 6, characterized in that: The variable field of view function calculation mechanism uses a piecewise linear function represented by the following formula as the variable field of view function: FOV=max(a,min(b,mean(len_z,len_y,len_x))) / c×d Wherein, FOV is the field of view; len_z, len_y, len_x are the lengths of the segmented target in the z direction, y direction, and x direction in the three-dimensional space respectively; a, b, c, and d are adjustable hyperparameters related to the statistics of the scale features.

8. The image segmentation device according to claim 3, characterized in that: The variable field of view function calculation mechanism calculates the variable field of view function by automatic fitting of a neural network. The neural network takes the shape features of the multiple segmented targets as input and the field of view range as output. The variable field of view function calculation mechanism calculates the variable field of view function by adjusting the loss function of the neural network to obtain the optimal field of view range.

9. The image segmentation device according to claim 2, characterized in that: The first segmentation model is a coarse segmentation model; The second segmentation model is a fine segmentation model.

10. The image segmentation device according to claim 9, characterized in that: The coarse segmentation model is any one of a deep learning segmentation model, a landmark model, a target detection model, and a graph cut segmentation model.

11. An image segmentation method for segmenting a plurality of segmentation targets contained in an image, characterized in that: include: A variable field of view function calculation step, calculating a variable field of view function, wherein the variable field of view function can adaptively generate a field of view range of corresponding size for the plurality of segmentation targets; An image block generating step, for the plurality of segmentation targets, using the variable field of view function calculated in the variable field of view function calculating step to generate image blocks of corresponding sizes; as well as The inference step uses a segmentation model trained by using the image blocks generated by the image block generation step to infer the image, so as to obtain segmentation results of the multiple segmentation targets.

Images