Training methods and systems for instance segmentation models
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-06
- Publication Date
- 2026-08-11
AI Technical Summary
[0010]例如，在共同工作场所中与人类并肩工作的协作机器人(cobot)中，人体部分的不完整的实例分割可能导致严重的伤害或死亡
[0021]在一些实现方式中，处理器可以不将权重应用到与背景相对应的区域，并且可以将权重应用到与实例的组成元素相对应的区域的点。在一些实现方式中，处理器可以基于整个掩码或样本点来计算损失。
Smart Images
Figure CN122551089A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to training an instance segmentation model, and more specifically, to a training method and system for implementing an instance segmentation model that ensures that constituent elements of different sizes that make up a single instance have equal loss contributions. Background Technology
[0002] Instance segmentation involves detecting distinct instances within an image and segmenting the pixels associated with each instance.
[0003] In images with complex environments, instance segmentation emphasizes handling occlusion between instances.
[0004] Complex environmental features may be instances that are closely spaced or overlapped.
[0005] Instance segmentation in an environment with overlapping instances is called occluded instance segmentation.
[0006] Instance segmentation has found wide applications in various fields, including security, industry, and retail, where occlusion scenarios are common, making the solution of occluded instance segmentation crucial.
[0007] Occluded instance segmentation is a very challenging computer vision problem, and extensive research has focused on developing effective models.
[0008] An ideal instance segmentation algorithm should accurately segment each instance pixel (accuracy), identify all components associated with a single instance as the same instance (completeness), and ignore fuzzy conditions such as reflections or group blobs (uncertainty).
[0009] Current instance segmentation models typically prioritize accuracy while neglecting integrity and uncertainty handling. However, integrity is particularly important for security applications.
[0010] For example, in collaborative robots (cobots) that work alongside humans in a shared workplace, incomplete segmentation of human body parts can lead to serious injury or death.
[0011] Existing models use losses that are applied uniformly across the entire instance, such as binary cross-entropy (BCE) loss. This approach is good at predicting large constituent elements, but inherently ignores semantically important small constituent elements.
[0012] Therefore, it is necessary to effectively apply the training loss to instance segmentation methods that consider both large and small constituent elements. Summary of the Invention
[0013] This disclosure relates to a method for efficiently applying training loss to both large and small constituent elements during instance segmentation.
[0014] This disclosure also relates to a constituent element balance focus (CBF) loss calculation method that applies loss by focusing on the constituent elements of an instance.
[0015] This disclosure also relates to a training method and system for implementing an instance segmentation model that ensures that constituent elements of different sizes that make up an instance have equal loss contributions.
[0016] This disclosure also relates to training methods and systems for implementing instance segmentation models, which can improve the integrity of instance segmentation.
[0017] This disclosure also relates to a training method and system for implementing an instance segmentation model that ensures that constituent elements of different sizes that make up an instance have equal loss contributions.
[0018] According to one aspect of the subject matter described in this application, an instance segmentation model training method may include: applying image data to an instance segmentation model to detect an instance mask; calculating a loss based on the detected instance mask and a ground truth (GT) instance mask corresponding to the image data; and updating the instance segmentation model based on the calculated loss, wherein the loss can be calculated by applying weights per component to a binary entropy loss calculation formula.
[0019] In some implementations, weights can be set based on the proportion of each component element in the mask. In others, weights may not be applied to the region corresponding to the background, but may be applied to the region corresponding to the component elements of the instance. In still others, the loss can be calculated based on the entire mask or sample points.
[0020] According to another aspect of the subject matter described in this application, an instance segmentation model training system may include: a first database configured to store an image dataset; a second database configured to store a set of ground truth (GT) instance masks corresponding to the image dataset; and a processor. The processor may be configured to: apply the image data to an instance segmentation model, thereby detecting instance masks; calculate a loss based on the detected instance masks and the GT instance masks corresponding to the image data; and update the instance segmentation model based on the calculated loss, wherein the loss is calculated by applying weights of the constituent elements to a binary entropy loss calculation formula.
[0021] In some implementations, the processor may not apply weights to regions corresponding to the background, but may apply weights to points in regions corresponding to the constituent elements of the instance. In other implementations, the processor may compute the loss based on the entire mask or sample points.
[0022] The method of effectively applying the training loss to both large and small constituent elements in instance segmentation, based on the feature implementation described in this application, can provide technical benefits.
[0023] For example, these features can advantageously provide a component element balance focus (CBF) loss calculation method to apply loss when focusing on the component elements of an instance.
[0024] These features can also advantageously provide training methods and systems for implementing instance segmentation models that ensure that constituent elements of different sizes that make up an instance have equal loss contributions.
[0025] These features are also advantageous in using CBF loss to train instance segmentation models to improve the integrity of instance segmentation and the instance segmentation performance of both large and small components. Attached Figure Description
[0026] Figure 1 This is a diagram illustrating an example of an instance segmentation model training system.
[0027] Figure 2 This is a flowchart illustrating an example of an instance segmentation model training method.
[0028] Figure 3 This is a graph representing an example of component-level recall relative to the ratio of component area to instance area.
[0029] Figure 4 This is an example diagram showing a comparison between the training results of an instance segmentation model and the ground truth (GT) instance mask of an existing instance segmentation model. Detailed Implementation
[0030] Figure 1 This is a diagram illustrating an example of an instance segmentation model training system 100.
[0031] See Figure 1 The instance segmentation model training system 100 (hereinafter referred to as the "system") can provide a component-balance focal loss calculation method, which applies loss when focusing on the component elements of an instance.
[0032] In some implementations, system 100 can train an instance segmentation model to ensure that the constituent elements of different sizes that make up a single instance have equal loss contributions.
[0033] In some implementations, system 100 may include a first database 110, a second database 120, and a processor 130, and the configuration of system 100 is not limited thereto.
[0034] In some implementations, the first database 110 may store multiple image data used to train the instance segmentation model.
[0035] Hereinafter, the multiple image data used for training will be referred to as the image dataset.
[0036] For example, image datasets can include large open-source datasets, and the data that includes image datasets is not limited to open-source datasets.
[0037] In some implementations, the second database 120 may store truth value (GT) data that corresponds to the image dataset stored in the first database 110.
[0038] For example, ground truth data can refer to the expected answer that the instance segmentation model aims to predict, and the performance of the instance segmentation model can be considered excellent when the output of the instance segmentation model matches the ground truth data with high precision.
[0039] For example, truth data can be labeled and stored in a second database 120, and multiple truth data stored in the second database 120 can be called a truth dataset.
[0040] For example, a truth dataset may include instances that serve as a reference for comparing segmentation results of instances contained in image data (which is stored in the first database 110).
[0041] In the following text, the output of the instance segmentation model may refer to the detected instance mask, and the instance stored in the second database 120 for comparison with the output (detected instance mask) of the instance segmentation module may refer to the GT instance mask.
[0042] An image can include one or more instances.
[0043] In some implementations, processor 130 may include an instance segmentation model and may train the instance segmentation model based on an image dataset.
[0044] The processor 130 can compare the detected instance mask (i.e. the output of the instance segmentation model) with the corresponding ground truth instance mask to calculate the loss.
[0045] The processor 130 can apply the calculated loss to train the instance segmentation model to improve the performance of the instance segmentation model.
[0046] The processor 130 may include a training module 131 and a loss calculation module 132.
[0047] Training module 131 may include an instance segmentation model and train the instance segmentation model based on the input image dataset.
[0048] Instance segmentation models can be trained to detect instances in an image and output a mask of the detected instances.
[0049] The loss calculation module 132 can calculate the loss based on the detected instance mask input from the training module 131 and the GT instance mask stored in the second database 120.
[0050] The loss calculation module 132 can calculate the loss by comparing the detected instance mask with the corresponding GT instance mask.
[0051] The loss calculation module 132 can calculate the proposed component-wise balance focus (CBF) loss by applying component-wise weights to the existing binary cross-entropy (BCE) loss.
[0052] The loss calculation module 132 can provide the calculated loss to the training module 131, so that the training module 131 can train the instance segmentation model by reflecting the loss.
[0053] In some implementations, the loss (L) CBF This can be expressed as Equation 1.
[0054] [Equation 1]
[0055]
[0056] Here, w p ′ can refer to the weighting coefficient, w av It can refer to the average coefficient applied uniformly throughout the entire mask area, l bce,p A can refer to the BCE loss of each pixel P, and A can refer to the area of the constituent elements to which pixel P belongs.
[0057] The weighting coefficient w depends on whether pixel p is contained in the background or foreground of the constituent elements. p ′ can be expressed as equation 2.
[0058] [Equation 2]
[0059]
[0060] For example, when pixel p is contained in the background, the weight coefficient w p ′ can be defined as the average coefficient w applied uniformly across the entire mask area. av And when pixel p is included in any of the constituent elements, the weight coefficient w p ′ can be defined as τ i ·w av .
[0061] Here, τ i It can be pointed to the weights of the constituent elements (i.e., the weights applied to the constituent element Ci), thereby ensuring that the cumulative contribution of the loss of each constituent element is equally distributed, regardless of the area of the constituent element.
[0062] According to the weights of the constituent elements τ i It can be defined as Equation 3.
[0063] [Equation 3]
[0064]
[0065] Here, K can refer to the number of constituent elements. It can refer to the area of the i-th constituent element. It can refer to the area of the j-th constituent element.
[0066] For example, suppose there are two constituent elements (C1, C2) in an image, and the area of constituent element C1 is... The area is considerably larger than that of constituent element C2. Right now The weights (τ1) applied to component element C1 and (τ2) applied to component element C2 can be calculated as follows:
[0067]
[0068] Equations 1 through 3 apply when processing the entire mask; however, the Component Balanced Focus (CBF) loss calculation method can also be applied to point-based loss calculation, which calculates the loss at the sampling point.
[0069] In point-based loss calculation, the loss (L) can be calculated based on the modified equations (Equations 4 to 6) derived from Equations 1 to 3. CBF ).
[0070] [Equation 4]
[0071]
[0072] [Equation 5]
[0073]
[0074] [Equation 6]
[0075]
[0076] In equations 4 to 6, N points It can refer to the number of sampling points, w p ′ can refer to the weighting coefficient, w av It can refer to the average coefficient applied uniformly throughout the entire mask area, l bce,p The BCE loss of P can be indicated, and K can refer to the number of constituent elements. It can refer to the number of sampling points for the i-th component, and It can refer to the number of sampling points for the j-th component.
[0077] Here, τ i It can be used to ensure that the cumulative contribution of the loss of each component is equally distributed according to the weight of the component, regardless of the number of sampling points.
[0078] Figure 2 This is a flowchart illustrating an example of an instance segmentation model training method.
[0079] Figure 2 The step-by-step operations described in the text can be performed by Figure 1 The processor 130 of the instance segmentation model training system 100 is implemented in the example.
[0080] See Figure 1 and Figure 2 The processor 130 can receive image data in step S210 and apply the received image data to the instance segmentation model in step S220 to detect instance masks.
[0081] Next, the processor 130 can compare the detected instance mask with the GT instance mask in step S230, and calculate the CBF loss based on the loss calculation method in step S240.
[0082] In step S240, when configured to calculate the loss based on the entire mask, processor 130 can calculate the CBF loss based on equations 1 to 3.
[0083] In step S240, when configured to calculate loss based on sampling points, processor 130 can calculate CBF loss based on equations 4 to 6.
[0084] Then, in step S250, the processor 130 can update the instance segmentation model by incorporating the CBF loss.
[0085] Then, in step S260, the processor can determine whether the predefined stopping conditions have been met.
[0086] If the predefined stopping condition is met (S260 - Yes), the processor 130 may stop training the instance segmentation model. If the predefined stopping condition is not met (S260 - No), the processor 130 may execute steps S210 to S250 again. For example, the processor 130 may repeat steps S210 to S250 until the predefined stopping condition is met.
[0087] Stop conditions can be defined in various ways depending on the application and / or model, and are not limited to the current implementation of this disclosure.
[0088] For example, stopping conditions may include when the difference (or loss) between the detected instance mask and the ground truth instance mask is less than a predefined threshold, when the rate of change of the difference (or loss) between the detected instance mask and the ground truth instance mask is less than a predefined threshold, when the accuracy (or F1 score) of the performance evaluation metric is greater than a predefined threshold, when the instance segmentation model has been trained a predefined number of times, and so on.
[0089] Table 1 shows the test results of an instance segmentation model trained using the implementation of the features described in this application and an instance segmentation model trained according to conventional techniques.
[0090] [Table 1]
[0091]
[0092] Tested on the OCHumanEX benchmark (Occluded Human Instance Segmentation Dataset).
[0093] It can be observed that the instance segmentation model applying the CBF loss according to the implementation of this disclosure outperforms other models. This performance improvement can be attributed to the enhanced instance integrity resulting from the application of CBF loss.
[0094] Therefore, it can be concluded that the instance segmentation model trained based on CBF loss according to the implementation of this disclosure can outperform other existing models, proving that CBF loss is very effective for implementing instance segmentation models.
[0095] Figure 3 This is a graph representing an example of component-level recall relative to the ratio of component area to instance area.
[0096] See Figure 3 As can be seen, smaller components have lower recall rates compared to larger components.
[0097] The recall performance of the instance segmentation model can be improved when the CBF loss is applied according to the implementation of the features described in this application.
[0098] It is worth noting that, compared with traditional instance segmentation models, applying CBF loss can significantly improve the recall performance of both large and small components. This demonstrates that CBF loss is highly effective in improving instance integrity while ensuring that it does not compromise the segmentation performance of large components.
[0099] Figure 4 This is an illustration showing an example of comparing the training results of an instance segmentation model according to an implementation of this disclosure and existing instance segmentation models with ground truth (GT) instance masks.
[0100] exist Figure 4 In the above, (a) represents the GT instance mask, (b) represents the detected instance mask output by the conventional instance segmentation model, and (c) represents the detected instance mask output by the instance segmentation model according to the implementation of this disclosure.
[0101] See Figure 4 Traditional instance segmentation models misidentify element _a of instance _A as element of instance _B. Figure 4 (b)), and the instance segmentation model according to the implementation of this disclosure can correctly identify the constituent element _a as belonging to instance _A( Figure 4 (c)).
[0102] In some implementations, the processor 130 may be implemented as a hardware data processing device with circuitry having a physical structure for performing the desired operation. For example, the desired operation may encompass code or instructions contained in a program.
[0103] For example, hardware-implemented data processing devices can include microprocessors, central processing units (CPUs), processor cores, multi-core processors, multiprocessors, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).
[0104] Processor 130 can be configured to store data related to algorithms or software instructions controlling the operation of various components associated with the features of this disclosure in memory, and to use the data stored in memory to perform the operations described herein. Here, the memory and processor can be implemented as separate chips or integrated into a single chip.
[0105] For example, the memory can store various algorithms, data, and other information necessary for the operation of the processor 130. The memory can include volatile memory and / or non-volatile memory. Volatile memory can include dynamic random access memory (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), and ferroelectric RAM (FeRAM). Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and flash memory.
Claims
1. A method for training an instance segmentation model, comprising: Image data is applied to an instance segmentation model to detect instance masks. The loss is calculated based on (i) the detected instance mask and (ii) the ground truth instance mask corresponding to the image data; as well as The instance segmentation model is updated based on the calculated loss. The loss is calculated by applying the weights of the constituent elements to the binary entropy loss calculation formula.
2. The method of claim 1, wherein, The weights are set based on the proportion of the constituent elements in the detected instance mask.
3. The method of claim 1, wherein, The weights are: (i) not applied to the region corresponding to the background, and (ii) applied to the region corresponding to the constituent elements of the instance.
4. The method of claim 1, wherein, The loss is calculated based on the entire mask or sampling points.
5. The method of claim 1, wherein, Based on the entire mask, the loss is calculated using the following Equation 1: where w′ p represents a weight coefficient, l bce,p represents the binary cross-entropy loss of each pixel point P, and A represents the area of the constituent element to which the pixel point P belongs.
6. The method according to claim 5, wherein, The weight coefficients w' are calculated using equation 2 and equation 3 p : [Equation 2] [Equation 3] Among them, C i w represents the i-th constituent element. av τ represents the average coefficient applied uniformly across the entire mask area. i This indicates the weighting of the constituent elements, where K represents the number of constituent elements. Let represent the area of the i-th component, and This represents the area of the j-th component element.
7. The method of claim 1, wherein, Based on the sampling points, the loss is calculated using the following Equation 4: wherein N points represents the number of sampling points, w′ p represents the weight coefficient, l bce,p represents the binary cross-entropy loss of each pixel point P.
8. The method of claim 7, wherein, The weight coefficients w' are calculated using equation 5 and equation 6 p : [Equation 5] [Equation 6] Among them, C i w represents the i-th constituent element. av τ represents the average coefficient applied uniformly across the entire mask area. i This indicates the weighting of the constituent elements, where K represents the number of constituent elements. This represents the number of sampling points for the i-th component element, and... This represents the number of sampling points for the j-th component element.
9. An instance segmentation model training system, comprising: The first database is configured to store image datasets; The second database is configured to store a set of truth instance masks corresponding to the image dataset; as well as The processor is configured as follows: Image data is applied to the instance segmentation model to detect instance masks. A loss is calculated based on the detected instance masks and the ground truth instance masks corresponding to the image data. The instance segmentation model is updated based on the calculated loss. The loss is calculated by applying the weights of the constituent elements to the binary entropy loss calculation formula.
10. The system of claim 9, wherein, The weights are set based on the proportion of the constituent elements in the detected instance mask.
11. The system of claim 9, wherein, The processor is configured to: (i) not apply the weights to the region corresponding to the background, and (ii) apply the weights to points in the region corresponding to the constituent elements of the instance.
12. The system of claim 9, wherein, The processor is configured to calculate the loss based on the entire mask or sampling points.
13. The system of claim 9, wherein, The processor is configured to calculate the loss based on the entire mask using the following Equation 1: Among them, w′ p Represents the weighting coefficient, l bce,p Let A represent the binary cross-entropy loss of each pixel P, and let A represent the area of the constituent elements to which pixel P belongs.
14. The system of claim 13, wherein, The processor is configured to calculate the weight coefficient w' using Equation 2 and Equation 3 p : [Equation 2] [Equation 3] Among them, C i w represents the i-th constituent element. av τ represents the average coefficient applied uniformly across the entire mask area. i This indicates the weighting of the constituent elements, where K represents the number of constituent elements. Let represent the area of the i-th component, and This represents the area of the j-th component element.
15. The system of claim 9, wherein, The processor is configured to calculate the loss based on the sampling points using the following Equation 4: wherein N points represents the number of sampling points, w p represents the weight coefficient, l bce,p represents the binary cross-entropy loss of each pixel point P.
16. The system of claim 15, wherein, The processor is configured to calculate the weighting coefficient wp using Equations 5 and 6: [Equation 5] [Equation 6] Among them, C i w represents the i-th constituent element. av τ represents the average coefficient applied uniformly across the entire mask area. i This indicates the weighting of the constituent elements, where K represents the number of constituent elements. This represents the number of sampling points for the i-th component element, and This represents the number of sampling points for the j-th component element.