Method and apparatus for training instance segmentation model for image
The method trains an instance segmentation model using feature maps and label images to update its segmentation based on proportions and loss functions, effectively distinguishing similar anatomical structures in X-ray images, enhancing diagnostic accuracy in medical imaging.
Patent Information
- Application Number
- PCT/KR2024/007229
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-05
AI Technical Summary
Conventional instance segmentation models struggle to accurately distinguish objects with similar shapes in X-ray images, particularly in medical diagnostics, where differentiating between similar anatomical structures like vertebrae is challenging.
The method involves training an instance segmentation model using feature maps and label images to identify individual and group instance areas, updating the model based on the proportions of these areas, and employing loss functions like Dice loss and Intersection over Union to enhance segmentation accuracy by considering connection relationships between instances.
This approach enables accurate segmentation of similar instances in X-ray images by learning the characteristics and connection relationships of individual and group instances, improving diagnostic accuracy in medical imaging.
Smart Images

Figure KR2024007229_05122024_PF_FP_ABST
Abstract
Description
Method and device for training an instance segmentation model for images
[0001] Embodiments of the present invention relate to a method and apparatus for training an instance segmentation model for an image, and a method and apparatus for instance segmentation based on an instance segmentation model.
[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.
[0003] Recent advances in computer vision technology have led to the advancement of image processing for diagnosis in the medical field. In particular, X-ray image-based diagnostic technology is being used to diagnose various medical conditions in both humans and animals.
[0004] Specifically, a Computer-Aided Diagnostic (CAD) system provides diagnostic aids to physicians based on anatomical structures and pathology, such as animal species and size differences. The physician then makes a diagnosis based on these aids. As diagnostic aids, the CAD system can provide boundaries and ID values for anatomical structures, such as bones and organs, within X-ray images.
[0005] For this purpose, instance segmentation, a computer vision technique, is utilized. Instance segmentation involves identifying objects of interest within an image at the pixel level using a deep learning-based instance segmentation model and then assigning ID values to them. This type of instance segmentation is distinguished from semantic segmentation in that it can independently detect objects of the same type.
[0006] Doctors can make accurate diagnoses by identifying the exact boundaries of objects of interest from X-ray images using instance segmentation models.
[0007] However, conventional instance segmentation models have a problem in accurately distinguishing similarly shaped objects within X-ray images. For example, conventional instance segmentation models have limitations in training the differences between vertebrae that repeatedly have similar appearances.
[0008] Therefore, research is needed on how to train an instance segmentation model so that it can accurately distinguish instances that have similar shapes.
[0009] The main purpose of embodiments of the present invention is to provide a method and device for training an instance segmentation model to accurately segment instances by considering the connection relationship between instances even if the instances to be segmented have similar shapes.
[0010] Another embodiment of the present invention aims to provide a method and apparatus for accurately segmenting similar instances in an input image by using a pre-trained instance segmentation model.
[0011] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.
[0012] According to one aspect of the present invention, a computer-implemented method for training an instance segmentation model is provided, comprising: inputting a training image including pixels into an instance segmentation model to obtain a plurality of feature maps, each feature map representing predicted values of the pixels for one of classes; identifying individual instance regions corresponding to each class based on label images for the training image, and a group instance region combining at least two of the individual instance regions; and updating the instance segmentation model based on weights of the individual instance regions on the feature maps and weights of the group instance regions on at least two feature maps related to the group instance regions.
[0013] According to another aspect of the present invention, there is provided a device for training an instance segmentation model, comprising: a memory for storing commands; and at least one processor, wherein the at least one processor executes the commands to input a training image including pixels into an instance segmentation model to obtain a plurality of feature maps, each feature map representing predicted values of the pixels for one of classes, identify individual instance regions corresponding to each class based on label images for the training image, and a group instance region combining at least two of the individual instance regions, and update the instance segmentation model based on the proportions of the individual instance regions on the feature maps and the proportions of the group instance regions on at least two feature maps related to the group instance regions.
[0014] According to another aspect of the present invention, a device for instance segmentation for an input image is provided, comprising: a memory for storing commands; and at least one processor, wherein the at least one processor executes the commands to segment instances in the input image using a pre-trained instance segmentation model. The instance segmentation model is trained by: inputting a training image including pixels into the instance segmentation model to obtain a plurality of feature maps, each feature map representing predicted values of the pixels for one of classes; identifying individual instance regions corresponding to each class based on label images for the training image, and a group instance region combining at least two of the individual instance regions; and updating the instance segmentation model based on proportions of the individual instance regions on the feature maps and proportions of the group instance regions on at least two feature maps related to the group instance regions.
[0015] As described above, according to one embodiment of the present invention, an instance segmentation model can be trained to accurately segment instances by considering the connection relationship between instances even if the instances to be segmented have similar shapes.
[0016] According to another embodiment of the present invention, similar instances within an input image can be accurately segmented by using a pre-trained instance segmentation model.
[0017] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.
[0018] FIG. 1 is a configuration diagram of an instance splitting device according to one embodiment of the present invention.
[0019] FIG. 2a and FIG. 2b are drawings showing a spinal skeleton and label image of an animal according to one embodiment of the present invention.
[0020] FIG. 3 is a diagram schematically illustrating a first loss function and a second loss function according to one embodiment of the present invention.
[0021] FIG. 4 is an exemplary diagram illustrating calculation of a first loss function according to one embodiment of the present invention.
[0022] FIG. 5 is an exemplary diagram illustrating calculation of a second loss function according to one embodiment of the present invention.
[0023] FIG. 6 is an exemplary diagram illustrating calculation of a first loss function according to another embodiment of the present invention.
[0024] Figure 7 is a flowchart of a training method for an instance segmentation model according to one embodiment of the present invention.
[0025] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.
[0026] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.
[0027] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.
[0028] FIG. 1 is a configuration diagram of an instance splitting device according to one embodiment of the present invention.
[0029] Referring to FIG. 1, the instance segmentation device (100) segments instances for an input image and provides boundary information and meta information for the segmented instances.
[0030] To this end, the instance splitter (100) includes an input unit (110), a splitter (120), and an output unit (130). The instance splitter (100) includes at least one processor and a memory including at least one instruction, and can perform the functions of the input unit (110), the splitter (120), and the output unit (130) through execution of instructions by at least one processor. Furthermore, the instance splitter (100) may further include a communication module for communicating each piece of information.
[0031] The input unit (110) is a component that acquires input images in various formats.
[0032] Here, the input image may be one of various images used in the medical field, such as an X-ray image, a magnetic resonance image, an ultrasound image, or a thermal image.
[0033] Such input images contain at least one object. For example, if the input image is an X-ray image, the input image may contain objects such as animal bones, organs, etc.
[0034] The segmentation unit (120) is a component that segments objects within an input image into instance units. To this end, the segmentation unit (120) includes a preprocessing module (121) and an instance segmentation model (123).
[0035] The preprocessing module (121) converts the input image into byte-format data that can be processed by the instance segmentation model (123).
[0036] Furthermore, the preprocessing module (121) can filter out input images that are unsuitable for instance segmentation. For example, the preprocessing module (121) determines whether the animal's posture in the input image is inappropriate and provides the user with the judgment result. As another example, if the input image has low resolution or poor image quality, the preprocessing module (121) can determine that the input image is unsuitable for instance segmentation.
[0037] The instance segmentation model (123) is a deep learning-based model trained to identify individual instance regions within a training image. The segmentation unit (120) can segment instances within the training image using the pre-trained instance segmentation model (123). Here, the training image may be a type of image utilized in the medical field, similar to the input image.
[0038] For example, when an X-ray image of an animal skeleton is input into an instance segmentation model (123), the instance segmentation model (123) identifies image regions by skeleton type and provides information about which class each of the identified image regions belongs to, along with the boundaries of the identified image regions.
[0039] These instance segmentation models (123) are pre-trained to segment individual instances within training images.
[0040] Specifically, the instance segmentation model (123) extracts feature maps representing predicted values for each class from the training images. The instance segmentation model (123) is updated based on the weights of individual instance regions identified by preset labels on each feature map. As training is repeated, the instance segmentation model (123) learns the characteristics of each individual instance.
[0041] In particular, in order to accurately segment individual instances having similar shapes, the instance segmentation model (123) is trained based on not only individual instances but also group instances combined from the individual instances. The instance segmentation model (123) can understand the connection relationship between individual instances included in the group instance through group instance-based training. To this end, the instance segmentation model (123) is updated based on the proportion of group instance regions on each feature map. The instance segmentation model (123) trained to segment group instances together with individual instances takes into account the connection relationship between the combined individual instances, and therefore can accurately segment individual instances.
[0042] Furthermore, for accurate segmentation, the instance segmentation model (123) can be trained further based on binary cross entropy (BCE) according to the predicted distribution of pixels in the training image and the predicted distribution of the label image.
[0043] In summary, the instance segmentation model (123) is updated based on a first loss function that includes the proportion of individual instance regions within each feature map, and a second loss function that includes the proportion of group instance regions. Furthermore, the instance segmentation model (123) may be further updated based on a third loss function representing binary cross entropy.
[0044] Here, the first loss function and the second loss function can be defined in the form of a Dice loss function or an IoU (Intersection over Union).
[0045] A detailed description of the training of the instance segmentation model (123) is described later with reference to FIGS. 2a, 2b, 3, 4, 5, and 6.
[0046] Meanwhile, the instance segmentation model (123) may be a convolutional neural network (CNN) for image processing. Alternatively, the instance segmentation model (123) may be implemented using various neural networks for image processing, such as CNN-based Mask R-CNN or Swin Transformer using a Transformer structure, depending on hardware performance.
[0047] The instance segmentation model (123) may be trained on the instance segmentation device (100), but may also be trained on a separate computing device for training.
[0048] The output unit (130) is a component that outputs the instance segmentation results to the user based on the processing results of the instance segmentation model (123). To this end, the output unit (130) includes a post-processing module (131) and an output module (133).
[0049] The post-processing module (131) displays instance regions and classes of instance regions within the input image on the input image. As an example, the post-processing module (131) may display different instance regions using different colors and display classes next to the instance regions.
[0050] The output module (133) visually outputs the results of displaying instance areas and class information on an input image through a display device.
[0051] FIG. 2a and FIG. 2b are drawings showing a spinal skeleton and label image of an animal according to one embodiment of the present invention.
[0052] Referring to Figure 2a, an image of the lateral thoracic spinal structure of an animal is shown. Here, the animal may be a dog.
[0053] The animal's spinal structure is the segmentation target of the instance segmentation model. Furthermore, instances such as the heart, keel, and internal organs can be segmented in addition to the spinal structure.
[0054] The animal spine consists of the cervical, thoracic, lumbar, sacrum, and coccygeal vertebrae. Specifically, the animal spine consists of seven cervical vertebrae, thirteen thoracic vertebrae, seven lumbar vertebrae, three sacral vertebrae, and 20 to 23 coccygeal vertebrae. Each bone is assigned a class.
[0055] The instance segmentation model must be able to identify and classify regions of a total of 50 to 53 vertebrae. To train the instance segmentation model to segment these vertebrae, labels for the animal's vertebrae are required.
[0056] Referring to Figure 2b, labeled images of the animal's vertebrae and heart are shown.
[0057] The labeled image shows the instance region representing the shape of the object along with the bounding box for each vertebra. C2 to C7 are labels for the six cervical vertebrae. t1 to t 13 , I1 and I2 are labels for the 13 thoracic vertebrae and the 2 lumbar vertebrae, respectively. Also shown are labels for the heart and the carina.
[0058] Here, the thoracic vertebrae are labeled only for their root area, not their entire area.
[0059] The label image shown in Figure 2b is a representation of labels overlapping on a single image for simplicity. In another example, a single label image may represent a single instance area.
[0060] When an instance segmentation model receives an image as shown in Fig. 2a, it is trained to distinguish instances in the input image and provide classes of the instances as shown in Fig. 2b.
[0061] However, because animal vertebrae have similar shapes, it is difficult to train an instance segmentation model to accurately distinguish each vertebra. For example, an instance segmentation model may have difficulty distinguishing between the 7th and 8th thoracic vertebrae, or between the 13th thoracic vertebrae and the 1st lumbar vertebra. An instance segmentation model may be able to recognize the 3rd thoracic vertebra at the location of the 4th thoracic vertebra. In other words, an instance segmentation model may overfit to each class, resulting in poor segmentation performance.
[0062] To address the issue of instance segmentation models failing to accurately distinguish between vertebrae, some vertebrae can be labeled differently from their neighbors. For example, in Figure 2b, the 1st thoracic vertebra is labeled across the entire area of the actual thoracic vertebrae, making it easier to distinguish from the 7th cervical vertebra and the 2nd thoracic vertebra.
[0063] Nonetheless, the performance of distinguishing instances is improved only around vertebrae that are labeled differently from the surrounding vertebrae.
[0064] According to one embodiment of the present invention, an instance segmentation model can understand the characteristics of individual instances and the connections between individual instances by training not only individual instances for vertebrae but also group instances that combine individual instances. The instance segmentation model can learn the characteristics of individual instances based on a first loss and the connections between individual instances based on a second loss.
[0065] FIG. 3 is a diagram schematically illustrating a first loss function and a second loss function according to one embodiment of the present invention.
[0066] Referring to FIG. 3, a label image (310) showing individual instance areas, first loss items (321) according to a first loss function, and second loss items (323) according to a second loss function are illustrated.
[0067] Within the label image (310), three individual instance regions for the 7th cervical vertebra (C7), the 1st thoracic vertebra (T1), and the 2nd thoracic vertebra (T2) are indicated by solid lines. In Fig. 3, the individual instance regions are indicated within one label image (310), but in other examples, one individual instance region may be indicated in one of multiple label images. Meanwhile, one individual instance region corresponds to one class.
[0068] The first loss items (321) are expressed as the sum of the mismatch rates between the individual instance regions of the label image (310) and the predicted instance regions of the instance segmentation model.
[0069] The second loss items (323) are expressed as the sum of the discrepancies between the combinations of group instance areas, which are two-by-two combinations of three individual instance areas, and the predicted instance areas of the instance segmentation model. As an example, the difference value between a group instance area, which is a combination of individual instance areas for the 7th cervical vertebra and the 1st thoracic vertebra, and the corresponding predicted instance areas is calculated as one of the second loss items (323).
[0070] If we train the instance segmentation model so that T1 is always partitioned with C7 and T2, and T2 is always partitioned with T1 and T3, the instance segmentation model will find C7 in the left adjacent region of T1, and find T2 in the right adjacent region of T1.
[0071] In this way, when the instance segmentation model is trained based on the first loss items (321) and the second loss items (323), the instance segmentation model can segment individual instances corresponding to each class by considering the connection relationship between individual instance regions.
[0072] FIG. 4 is an exemplary diagram illustrating calculation of a first loss function according to one embodiment of the present invention.
[0073] Referring to FIG. 4, the instance segmentation model (123) is updated according to a first loss function that includes the weights of individual instance regions. Here, the first loss function can be defined in the form of a predicted value-based Dice loss function.
[0074] For this purpose, a training image containing four pixels and three label images having the same size as the training image are used as training data for the instance segmentation model (123).
[0075] Here, the label image is a label for the training image, indicating the region of pixels where instances are located within the training image. The label image may be a binary mask for each instance region. One label image has a value of 1 at the positions of pixels corresponding to one class, and a value of 0 at the positions of the remaining pixels. The positions of pixels with a value of 1 in each label image indicate the individual instance region corresponding to each class.
[0076] The instance segmentation model (123) can be trained by a training device including at least one processor and at least one memory. The at least one processor can perform the training method of the instance segmentation model (123) by executing instructions stored in at least one memory. Alternatively, the instance segmentation model (123) can also be trained by an instance segmentation device.
[0077] First, the training device for training the instance segmentation model (123) inputs a training image into the instance segmentation model (123) to obtain three feature maps. Each feature map is a prediction value of pixels for one of the three classes. Includes. represents the probability that the ith pixel belongs to class c. i represents the index of the pixel, and c represents the index of the class. For example, represents the predicted value of pixel 1 for class C1, represents the predicted value of pixel 1 for class C2.
[0078] The training device identifies individual instance regions corresponding to each class based on labeled images. Specifically, the training device identifies pixels 1 and 3 as individual instance regions corresponding to class C1, pixel 2 as an individual instance region corresponding to class C3, and pixel 4 as an individual instance region corresponding to class C2.
[0079] Afterwards, the training device calculates the weight of each instance region on each feature map based on the feature maps and label images. Here, the weight of each instance region means the ratio of the predicted values of each instance region in the feature map to the total predicted values of the feature map. For example, the weight of each instance region for class C1 on the first feature map is the 1st predicted value of the sum of the 4 predicted values of the first feature map. and predicted value number 3 is the consensus ratio.
[0080] The proportion of individual instance areas is an indicator of how well the instance segmentation model (123) learned the characteristics of individual instances within the training image.
[0081] The training device calculates the first loss based on the weights of three individual instance regions. The first loss has an inverse trend to the sum of the weights of the individual instance regions. That is, the larger the weights of the individual instance regions, the smaller the first loss.
[0082] The first loss can be defined as in mathematical expression 1.
[0083]
[0084] In Equation 1, L1 is the first loss, C is the number of classes, c is the index of the class, I is the number of pixels, i is the index of the pixel, is the predicted value of the ith pixel in the feature map corresponding to class c, is the i-th pixel value in the label image corresponding to class c, represents a very small constant that prevents the numerator or denominator from becoming 0. represents the sum of all predicted values of the feature map corresponding to class c. is a feature map masked by the label image corresponding to class c to the extent of each instance area, and represents the predicted values of each instance area within the feature map. represents the sum of the predicted values of individual instance regions in the feature map corresponding to class c. Meanwhile, may be omitted.
[0085] The training device updates the parameters of the instance segmentation model (123) so as to reduce the first loss. That is, for each class, the instance segmentation model (123) is updated so as to increase the proportion of the predicted values of individual instance regions within the feature map relative to the overall predicted values of the feature map.
[0086] FIG. 5 is an exemplary diagram illustrating calculation of a second loss function according to one embodiment of the present invention.
[0087] Referring to FIG. 5, the instance segmentation model (123) is updated according to a second loss function that includes the proportion of the group instance area. Here, the second loss function can be defined in the form of a predicted value-based Dice loss function.
[0088] As training data for the instance segmentation model (123), a combination of a training image containing four pixels and a label image having the same size as the training image is used.
[0089] First, the training device can set class combinations by combining at least two classes among multiple classes. For example, for classes C1, C2, C3, and C4, the training device can set class combinations such as H={(C1, C2), (C3, C4), (C1, C2, C3), (C1, C2, C3, C4)}.
[0090] The training device generates a label image combination corresponding to each class combination from the label images. The label image combination may be a binary mask for a group instance region. A label image combination may be generated through pixel-wise addition between at least two label images for a class combination. If any of the label images to be combined has a value of 1 at a specific pixel location, the label image combination has a value of 1 at the same location. The locations of pixels having a value of 1 in the label image combination represent individual instance regions corresponding to the classes in the class combination. A combination of individual instance regions corresponding to the classes in the class combination is called a group instance region. In particular, the group instance region may be a combination of at least two adjacent individual instance regions among the individual instance regions.
[0091] Although multiple label image combinations can be generated based on the combination of label images, Fig. 5 describes a combination of two label images. In Fig. 5, the label image combination is a combination of the first label image and the second label image illustrated in Fig. 4.
[0092] The training device identifies group instance regions based on a combination of label images. The training device identifies pixels 1, 3, and 4, labeled as class C1, and as group instance regions, respectively.
[0093] Afterwards, the training device calculates the proportion of the group instance region on the two feature maps based on the combination of the two feature maps and the label image. Here, the proportion of the group instance region means the ratio of the predicted values of the group instance region in the two feature maps to the total predicted values of the two feature maps. In Fig. 5, the proportion of the group instance region is the proportion of 6 predicted values out of a total of 8 predicted values. Predicted value 1 of the first feature map , predicted value 3 , predicted value 4 , the first predicted value of the second feature map , predicted value 3 and 4th predicted value The sum of the four predicted values of the first feature map is calculated, and the proportion of the group instance area is the ratio of the calculated sum to the total sum of the four predicted values of the second feature map.
[0094] The proportion of individual instance areas is an indicator of how well the instance segmentation model (123) learned the relationship between individual instances within the training image.
[0095] The training device calculates the second loss based on the weight of the group instance area. The second loss has an inverse trend to the sum of the weights of the group instance areas. That is, the larger the weight of the group instance area, the smaller the second loss.
[0096] The second loss can be defined as in Equation 2.
[0097]
[0098] In Equation 2, L2 is the second loss, H is the number of class combinations, h is the index of the class combination, I is the total number of pixels, i is the index of the pixel, are the predicted values of the ith pixel in the feature maps corresponding to the class combination h, is the i-th pixel value in the label image combination corresponding to the class combination h, represents a very small constant that prevents the numerator or denominator from becoming 0. represents the sum of all predicted values of the feature maps corresponding to the class combination h. are feature maps masked by the group instance area by the label image combination corresponding to the class combination h, and represent the predicted values of the group instance area within the feature maps. represents the sum of the predicted values of the group instance regions in the feature maps corresponding to the class combination h. Meanwhile, may be omitted.
[0099] If we apply Equation 2 to Fig. 5, Is It is the same as, Is It is the same as .
[0100] The training device updates the parameters of the instance segmentation model (123) so as to reduce the second loss. That is, for each class combination, the instance segmentation model (123) is updated so that the proportion of the predicted values of the group instance region within the feature maps increases with respect to the overall predicted values of the feature maps.
[0101] Referring to FIGS. 3 and 4, the training device utilizes both the first loss and the second loss to train the instance segmentation model (123). The training device may also train the instance segmentation model (123) based on the weighted sum of the first loss and the second loss. Accordingly, the instance segmentation model (123) is updated in a direction that reduces the first loss and the second loss.
[0102] Meanwhile, according to another embodiment, the training device can train the instance segmentation model (123) by further utilizing binary cross entropy, i.e., the third loss, which represents the contrast between the predicted values of each individual instance region and the predicted values outside each individual instance region. The third loss is an indicator of how similar the probability distributions of each feature map and each label image are on a pixel-by-pixel basis.
[0103] Referring back to FIG. 4, the training device updates the instance segmentation model (123) so that the size of the prediction values of individual instance regions within each feature map increases and the size of the prediction values outside the individual instance regions decreases. For example, since the individual instance regions corresponding to class C1 within the first feature map are pixels 1 and 3, the instance segmentation model (123) is trained to output large prediction values for pixels 1 and 3 and small prediction values for pixels 2 and 4 for class C1.
[0104] The third loss can be defined as in Equation 3.
[0105]
[0106] In Equation 3, L3 is the third loss, C is the number of classes, c is the index of the class, I is the number of pixels, i is the index of the pixel, is the predicted value of the ith pixel in the feature map corresponding to class c, is the i-th pixel value in the label image corresponding to class c, represents the sigmoid function.
[0107] The training device updates the parameters of the instance splitting model (123) so that the third loss is reduced.
[0108] When the training device trains the instance segmentation model (123) using all of the first loss, the second loss, and the third loss, the overall loss function of the instance segmentation model (123) is defined as in mathematical expression 4.
[0109]
[0110] In mathematical expression 4, L total is the total loss, L1 is the first loss, L2 is the second loss, L3 is the third loss, α is the first smoothing coefficient, and λ is the second smoothing coefficient. The first smoothing coefficient and the second smoothing coefficient have values greater than or equal to 0 and less than or equal to 1.
[0111] The instance segmentation model (123) is updated according to a second loss function that includes the proportion of the group instance area. Here, the second loss function can be defined in the form of a predicted value-based Dice loss function.
[0112] FIG. 6 is an exemplary diagram illustrating calculation of a first loss function according to another embodiment of the present invention.
[0113] Referring to Figure 6, the instance segmentation model (123) is updated according to a first loss function that includes the proportion of individual instance areas. Here, the first loss function can be defined in the form of an area-based Intersection over Union (IoU). IoU refers to the ratio of the overlapping area to the sum of the individual areas.
[0114] The training device can calculate the proportion of the area overlapping between the predicted instance area according to each feature map and the individual instance area according to each label image as the proportion of the individual instance area.
[0115] Specifically, the training device identifies predicted instance regions based on the feature maps output from the instance segmentation model (123). The training device can identify pixels with predicted values greater than a predetermined threshold value as predicted instance regions. For example, pixels 1, 3, and 4 of the first feature map can be identified as predicted instance regions for class C1.
[0116] The training device identifies individual instance regions corresponding to each class based on each label image. The individual instance regions for class C1 are pixels 1 and 3.
[0117] The training device trains an instance segmentation model (123) based on the weight of each individual instance area and the weight of a group instance area that combines individual instance areas.
[0118] Here, the proportion of each individual instance region refers to the overlapping area ratio between the predicted instance region according to each feature map and the individual instance region according to each label image. For example, the number of pixels in the predicted instance region is 3, the number of pixels in the individual instance region is 2, and the number of pixels in the area where the predicted instance region and the individual instance region overlap is 2, so the overlapping area ratio is 2 / (3+2).
[0119] The training device updates the instance segmentation model (123) in a direction that increases the proportion of each individual instance area calculated based on each area ratio. As training is repeated, the instance segmentation model (123) becomes more capable of accurately distinguishing each individual instance area.
[0120] In addition, the training device can update the instance segmentation model (123) further using a second loss function including the proportion of the group instance region. As an example, the training device selects at least two feature maps that share classes with at least two individual instance regions included in the group instance region among the feature maps. The training device identifies predicted instance regions from the at least two feature maps. The training device calculates the area ratio of the predicted instance regions overlapping with the corresponding at least two individual instance regions as the proportion of the group instance region. Thereafter, the training device updates the instance segmentation model (123) based on the proportion of the group instance region.
[0121] In this way, the first loss function is defined as the overlapping area ratio between the predicted instance regions identified according to each feature map and the individual instance regions identified according to each label image, and the second loss function can be defined as the overlapping area ratio between the combination of predicted instance regions according to the feature maps and the combination of individual instance regions according to the label images. The instance segmentation model is updated so that the area ratios increase.
[0122] Figure 7 is a flowchart of a training method for an instance segmentation model according to one embodiment of the present invention.
[0123] Referring to FIG. 7, the training device inputs a training image including pixels into an instance segmentation model to obtain multiple feature maps (S710).
[0124] Here, each feature map represents the predicted values of pixels for one of the classes.
[0125] The training device identifies individual instance regions corresponding to each class based on label images for the training images, and group instance regions that combine at least two of the individual instance regions (S720).
[0126] Here, each label image represents the location of an instance corresponding to one of the above classes within the training image. Meanwhile, a group instance region may be a combination of at least two adjacent individual instance regions among the individual instance regions.
[0127] The training device updates the instance segmentation model based on the weights of individual instance regions on the feature maps and the weights of group instance regions on at least two feature maps related to the group instance regions (S730).
[0128] In one embodiment, the training device can calculate the weight of each individual instance region based on the predicted values of each feature map. As an example, the training device selects a first feature map and a first individual instance region that share the same class among the feature maps and the individual instance regions. The training device calculates the weight of the individual instance region as the ratio of the first predicted values of the first individual instance region in the first feature map to the total predicted values of the first feature map.
[0129] Additionally, the training device can calculate the weight of the group instance region based on the predicted values of each feature map. For example, the training device selects at least two feature maps among the feature maps that share classes with at least two individual instance regions within the group instance region. The training device calculates the weight of the group instance region as the ratio of the predicted values of the group instance region within the at least two feature maps to the total predicted values of the at least two feature maps.
[0130] Furthermore, the training device can update the instance segmentation model further based on the contrast between the predicted values in each individual instance region and the predicted values outside each individual instance region.
[0131] In another embodiment, the training device can calculate the weight of the individual instance region based on the overlapping area ratio between the predicted instance region identified according to the feature map and the individual instance region identified according to the label image.
[0132] Specifically, the training device identifies predicted instance regions from each feature map. The training device calculates the area ratio overlapping each predicted instance region with its corresponding individual instance region as the weight of each individual instance region. The training device then updates the instance segmentation model based on the weight of each individual instance region.
[0133] Additionally, the training device can calculate the proportion of the group instance region based on the overlapping area ratio between the combination of predicted instance regions and the combination of individual instance regions. For example, the training device selects at least two feature maps that share classes with at least two individual instance regions included in the group instance region from among the feature maps. The training device identifies predicted instance regions from the at least two feature maps. The training device calculates the overlapping area ratio of the predicted instance regions with the corresponding at least two individual instance regions as the proportion of the group instance region. Thereafter, the training device updates the instance segmentation model based on the proportion of the group instance region.
[0134] Once the training of the instance segmentation model is completed through the training process of Fig. 7, it can be distributed to instance segmentation devices.
[0135] The instance segmenter segments instances within an input image using a trained instance segmentation model. Specifically, the instance segmenter identifies multiple instance regions from the feature maps output by the instance segmentation model for the input image and identifies the classes of the identified instance regions. The instance segmenter outputs the regions and classes of instances within the input image to the user.
[0136] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."
[0137] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.
[0138] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.
[0139] The above description is merely an example of the technical idea of the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of the present embodiment, but rather to explain it, and the scope of the technical idea of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.
[0140] CROSS-REFERENCE TO RELATED APPLICATION
[0141] This patent application claims priority to Korean Patent Application No. 10-2023-0069039, filed on May 30, 2023, which is incorporated herein by reference in its entirety.
Claims
1. A method for training an instance segmentation model performed by a training device including at least one processor, A step of inputting a training image containing pixels into an instance segmentation model to obtain multiple feature maps, wherein each feature map represents predicted values of the pixels for one of the classes; A step of identifying individual instance regions corresponding to each class based on label images for the training images, and a group instance region combining at least two of the individual instance regions; and A step of updating the instance segmentation model based on the proportions of the individual instance regions on the feature maps and the proportions of the group instance regions on at least two feature maps related to the group instance regions. How to include.
2. In paragraph 1, The step of updating the above instance splitting model is: A step of selecting a first feature map and a first individual instance region that share the same class among the above feature maps and the above individual instance regions; and In order to calculate the proportion of the first individual instance region, a step of calculating the ratio of the first predicted values of the first individual instance region in the first feature map to the entire predicted values of the first feature map. How to include.
3. In paragraph 1, The step of updating the above instance splitting model is: A step of selecting at least two feature maps among the above feature maps that share classes with at least two individual instance regions within the group instance region; and As a proportion of the group instance region, a step of calculating the ratio of the predicted values of the group instance region within the at least two feature maps to the total predicted values of the at least two feature maps How to include.
4. In paragraph 1, The step of updating the above instance splitting model is: A step of updating the instance segmentation model based further on the contrast between the predicted values in each individual instance region and the predicted values outside each individual instance region. How to include.
5. In paragraph 1, The above group instance area is, A method of combining at least two adjacent individual instance areas among the above individual instance areas.
6. In paragraph 1, The step of updating the above instance splitting model is: A step of identifying a predicted instance region from each feature map; and A step of calculating the proportion of the area that each predicted instance area overlaps with the corresponding individual instance area, as a proportion of each individual instance area. How to include.
7. In paragraph 1, The step of updating the above instance splitting model is: A step of selecting at least two feature maps among the above feature maps that share classes with at least two individual instance areas included in the group instance area; a step of identifying predicted instance regions from at least two feature maps; and A step of calculating the area ratio of the predicted instance areas overlapping with at least two corresponding individual instance areas as a proportion of the group instance area. How to include.
8. In paragraph 1, Each label image is, A method for indicating the location of an instance corresponding to one of the classes within the training image.
9. In a device for training an instance splitting model, memory that stores commands; and Containing at least one processor, At least one processor executes the instructions, A training image containing pixels is input to an instance segmentation model to obtain multiple feature maps, each feature map representing the predicted values of the pixels for one of the classes. Identifying individual instance regions corresponding to each class based on label images for the above training images, and group instance regions that combine at least two of the individual instance regions, A device for updating the instance segmentation model based on the weights of the individual instance areas on the feature maps and the weights of the group instance areas on at least two feature maps related to the group instance areas.
10. In a device for instance segmentation for an input image, memory that stores commands; and Containing at least one processor, At least one processor executes the instructions, Segment instances within the input image using a pre-trained instance segmentation model, The above instance splitting model is, A step of inputting a training image containing pixels into an instance segmentation model to obtain multiple feature maps, wherein each feature map represents predicted values of the pixels for one of the classes; A step of identifying individual instance regions corresponding to each class based on label images for the training images, and a group instance region combining at least two of the individual instance regions; and A step of updating the instance segmentation model based on the proportions of the individual instance regions on the feature maps and the proportions of the group instance regions on at least two feature maps related to the group instance regions. A device that has been trained, including:
Citation Information
Patent Citations
Segmentation method
JP2023046290A
Apparatus and method for generating prediction model
KR102274069B1
Method for image recognition and processing through classification model creation, and computer program recorded on record-medium for executing method therefor
KR102410752B1
Instance partitioning method and device, electronic device, program and medium
KR102438095B1
KR20220112662A
