A deep neural network training data augmentation method for image segmentation

Abstract

The present application relates to the field of life science, medicine and industrial detection technology, and particularly relates to a deep neural network training data enhancement method for image segmentation; comprising the following steps: S1, collecting training data to form original sample pairs; S2, selecting a structure of interest to be data enhanced; S3, generating an initial coordinate field, and then performing a normalization operation; S4, setting data enhancement parameters; S5, calculating a gradient field of the structure of interest to be data enhanced; S6, generating a deformation coordinate field; S7, restoring the deformation coordinate field; S8, mapping the deformation coordinate field to form a deformation sample pair; S9, obtaining a final sample pair; the relationship between structures in an image is used to simulate the deformation of various structures, greatly alleviating the problem of image texture detail loss, realizing real-time data enhancement, improving the calculation rate, providing sufficient and reliable training data for subsequent deep neural network training, and improving the training effect of the deep neural network.

Description

Technical Field

[0001] This invention relates to the fields of life sciences, medicine, and industrial inspection technology, and in particular to a method for augmenting training data of deep neural networks for image segmentation. Background Technology

[0002] In recent years, machine learning has achieved remarkable results in image analysis tasks across various fields, including image recognition, medical diagnosis, defect identification, and building health assessment, fully demonstrating its strong development potential. Machine learning methods are widely applied to various medical image analysis tasks, including ultrasound imaging, X-ray imaging, magnetic resonance imaging (MRI), retinal scans, histopathological images (HI), computed tomography (CT), positron emission tomography (PET), and dermoscopy images. Simultaneously, efficient machine learning algorithms have been proposed and applied to digital pathological image analysis to help pathologists obtain rapid, stable, and quantitative examination results, thereby enabling more accurate diagnoses. In industrial production, defect detection is crucial for ensuring high product quality and production efficiency, and it is necessary to improve detection speed while maintaining high accuracy in defect identification.

[0003] Currently, when training deep neural networks for image segmentation, only simple spatial data augmentation operations are used on the training data to avoid the loss of image texture details. However, this data augmentation operation limits the shape diversity of structures in the image, thus limiting the generalization ability of deep neural networks on structures with large deformations. Model-driven spatial deformation can simulate realistic structural deformation while avoiding the loss of image texture details. However, model-driven deformation computation is inefficient and cannot achieve real-time data augmentation.

[0004] Therefore, there is an urgent need to provide a deep neural network training data augmentation method for image segmentation that, compared with existing technologies, improves computation speed, satisfies data augmentation requirements for large deformation structures, and alleviates the problem of image texture detail loss. Summary of the Invention

[0005] This invention addresses the technical problems existing in the prior art and provides a method for augmenting training data of deep neural networks for image segmentation.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for augmenting training data of a deep neural network for image segmentation includes the following steps:

[0008] S1. Collect training data to form several original sample pairs of training data;

[0009] S2. For each pair of original samples formed in step S1, select the structure of interest to be augmented.

[0010] S3. For the structures of interest selected in step S2 that need data augmentation, generate an initial coordinate field, and then normalize the initial coordinate field.

[0011] S4. For the structures of interest selected in step S2 for data augmentation, set the data augmentation parameters, including deformation amplitude and deformation direction;

[0012] S5. Calculate the gradient field of the structure of interest to be augmented;

[0013] S6. Combining the normalized initial coordinate field from step S3, the deformation amplitude and direction set in step S4, and the gradient field calculated in step S5, generate the deformation coordinate field for the structure of interest to be augmented.

[0014] S7. Restore the deformation coordinate field generated in step S6.

[0015] S8. Map the deformation coordinate field obtained in step S7 to form deformation sample pairs;

[0016] S9. Randomly merge the original sample pairs obtained in step S1 with the deformed sample pairs obtained in step S8 to obtain the final sample pairs.

[0017] Furthermore, the training data collected in step S1 is represented as follows:

[0018] D = {(x1, ..., x} i , ..., x n ), (y1, ..., y i , ..., y n )}

[0019] in,

[0020]

[0021] In the above formula, D represents the training data, x1 represents the first image data, and x i Let x represent the i-th image data. n Let y1 represent the nth image data, and y1 represent the annotation data corresponding to x1. i x represents i The corresponding labeled data, y n x represents n Corresponding labeled data; This represents the labeled data for the first type of structure of interest. This represents the labeled data for the j-th structure of interest. This represents the annotation data for the m-th structure of interest;

[0022] The original sample pairs formed in step S1 are represented as: (x i y i ).

[0023] Furthermore, the specific method for step S2 is as follows: from y i From the m structures of interest, select B structures of interest that require data augmentation to form the structure of interest to be augmented, represented as:

[0024]

[0025] In the above formula, B i This represents the structure of interest to be augmented. This represents the labeled data for the structure of interest, type B.

[0026] Furthermore, the initial coordinate field generated in step S3 is represented as:

[0027]

[0028] in,

[0029]

[0030] Specifically, the initial coordinate field is normalized using the following formula:

[0031]

[0032] In the above formula, C i Represented as B i Initialize the coordinate field. Represented as Initialize the coordinate field. Represented as Initialize the coordinate field. for Initialize the coordinate field. This represents the coordinates of the first dimension of x1. Represented as x i The coordinates of the k-th dimension, x represents i The coordinates of the d-th dimension, where d is the x-th dimension. i The number of dimensions, Represented as normalized x i The coordinates of the k-th dimension, Represented as x i The size in the k-th dimension.

[0033] Furthermore, the gradient field of the structure of interest to be augmented in step S5 is expressed by the following equation.

[0034]

[0035] in,

[0036]

[0037] Specifically, it is calculated using the following formula:

[0038]

[0039]

[0040]

[0041] In the above formula, δ i Indicates the corresponding B i gradient field, Indicates correspondence gradient field, Indicates correspondence gradient field, Indicates correspondence gradient field, express In the gradient field of the first dimension, express In the gradient field of the k-th dimension, express The gradient field in the d-th dimension; This represents the z-th pixel value of the j-th structure of interest along the k-th dimension in yi. Let represent the (z+1)th pixel value of the j-th structure of interest along the k-th dimension in yi. Indicates y i The (z-1)th pixel value of the j-th structure of interest along the k-th dimension.

[0042] Furthermore, the deformation coordinate field generated in step S6 is specifically represented as follows:

[0043]

[0044] Specifically, it is calculated using the following formula:

[0045]

[0046] In the above formula, Indicating Bi Deformation coordinate field, express In the first-dimensional deformation coordinate field, express In the k-th dimension of the deformation coordinate field, express Deformation coordinate field in the d-th dimension; Represented as The deformation amplitude in the k-th dimension, Represented as The deformation direction in the k-th dimension. Time indicates Perform data augmentation operations on the k-th dimension. Time indicates No data augmentation operation is performed on the k-th dimension.

[0047] Furthermore, step S7 specifically restores the deformed coordinate field using the following formula:

[0048]

[0049] In the above formula, express The restored deformation coordinate field, x represents i The size in the k-th dimension.

[0050] Furthermore, the specific method of step S8 is as follows: the restored deformation coordinate field is applied to the x coordinates in step S1. i y i Then, deformed sample pairs are formed, represented as

[0051] Furthermore, the final sample pair in step S9 is represented as Specifically, it is calculated using the following formula:

[0052]

[0053]

[0054] In the above formula, α is a manually set fusion parameter, and f is one or more additional data augmentation operations, including but not limited to random cropping, random flipping, random rotation, gamma transformation, and Gaussian blur, among any suitable data augmentation operations.

[0055] Furthermore, the image data in S1 is medical image data or industrial inspection image data; when the image data in S1 is medical image data, it includes, but is not limited to, one or more of the following medical image modal data: positron emission tomography, magnetic resonance imaging, magnetic resonance tomography, single-photon emission computed tomography, and computed tomography.

[0056] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0057] (1) This invention utilizes the relationship between structures in an image to simulate the deformation of various structures, which greatly alleviates the problem of image texture detail loss, realizes real-time data enhancement, improves the calculation speed, provides sufficient and reliable training data for subsequent deep neural network training, facilitates subsequent training, and improves the training effect of deep neural networks. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0059] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0060] Example 1

[0061] like Figure 1 As shown, the present invention provides a method for augmenting training data of a deep neural network for image segmentation, comprising the following steps:

[0062] S1. Collect training data. Obtain labeled training data from any suitable source to form several pairs of original training data samples. The training data is represented as follows:

[0063] D = {(x1, ..., x} i , ..., x n ), (y1, ..., y i , ..., y n )}

[0064] In the above formula, D represents the training data, x1 represents the first image data, and x i Let x represent the i-th image data. n Let y1 represent the nth image data, and y1 represent the annotation data corresponding to x1. i x represents i The corresponding labeled data, y n x represents n The corresponding labeled data.

[0065] Furthermore, y i This can be expressed by the following formula:

[0066]

[0067] In the above formula, This represents the labeled data for the first type of structure of interest. This represents the labeled data for the j-th structure of interest. This represents the labeled data for the m-th structure of interest, where m represents y. i There are m structures of interest; for example, if y i If three structures of interest are marked above, then m is 3. The structures of interest are the bladder, small intestine, and lungs. Label data representing the bladder, The annotation data represents the small intestine. Labeling data representing the lungs.

[0068] Furthermore, the resulting training data original sample pairs are represented as: (x i y i ).

[0069] To go further, x i Medical image data, including but not limited to one or more of the following medical image modalities: positron emission tomography (PET), magnetic resonance imaging (MRI), magnetic resonance tomography (MPT), single-photon emission computed tomography (SPECT), and computed tomography (CT).

[0070] To go further, One of the appropriate data relating to the location, shape, and size of the target area, organs at risk (OARs), and any other structures of interest (such as skin, bone, etc.).

[0071] S2. For each pair of training data samples formed in step S1, select the structure of interest to be augmented: from y i From m structures of interest, select B structures of interest that require data augmentation, where B ≤ m. Typically, the selected B is less than m. These B structures form the structures of interest to be augmented, specifically represented as follows:

[0072]

[0073] In the above formula, B i This indicates the structure of interest for which data augmentation is desired.

[0074] To elaborate further, this step is for y i Data augmentation can be performed by selecting data from a pool of data, or in some cases, by selecting data randomly.

[0075] S3. For the structures of interest selected in step S2, generate the initial coordinate field: B i Each structure of interest in the array generates an initial coordinate field, represented as:

[0076]

[0077] in,

[0078]

[0079] In the above formula, C i Represented as B i Initialize the coordinate field. Represented as Initialize the coordinate field. Represented as Initialize the coordinate field. for Initialize the coordinate field. x represents i First-dimensional coordinates, Represented as x i The coordinates of the k-th dimension, Is and x ； Matrices of the same size x represents i The coordinates of the d-th dimension, where d is the x-th dimension. i The number of dimensions, for example, in three-dimensional coordinate data, d is 3, then

[0080] Furthermore, this step will also Coordinate field of each dimension Perform normalization as follows:

[0081]

[0082] In the above formula, Represented as normalized x i The coordinates of the k-th dimension, x represents i The coordinates of the k-th dimension, Represented as x i The size in the k-th dimension.

[0083] S4. For the structures of interest selected in step S2, set the data augmentation parameters to B. i For each structure of interest, the deformation amplitude and direction are randomly assigned, specifically as follows:

[0084]

[0085]

[0086] in,

[0087]

[0088]

[0089] In the above formula, Ai Indicates the deformation amplitude. They are respectively the corresponding The deformation amplitude, express The deformation amplitude in the first dimension, express The deformation amplitude in the k-th dimension, express The deformation amplitude in the d-th dimension; O i Indicates the direction of deformation. They are respectively the corresponding The direction of deformation, express The direction of deformation in the first dimension express The deformation direction in the k-th dimension. express The deformation direction in the d-th dimension; represent Whether to perform data augmentation in the k-th dimension, when Time indicates When performing data augmentation operations in the k-th dimension, Time indicates No data augmentation operation is performed on the k-th dimension.

[0090] In this step, you can set the deformation amplitude and direction for the data augmentation operation. Since deformation is not required in every direction, you can... To judge Whether to perform a deformation operation in the k-th dimension, and the specific deformation amplitude of the deformation operation are determined by A. i To bestow.

[0091] S5. For the structures of interest selected in step S2, calculate the gradient field of the structures of interest to be augmented: B i The gradient field corresponding to each structure of interest is represented as follows:

[0092]

[0093] in,

[0094]

[0095] In the above formula, δ i Indicates the corresponding B i gradient field, Indicates correspondence gradient field, Indicates correspondence gradient field, Indicates correspondence gradient field, express In the gradient field of the first dimension, express In the gradient field of the k-th dimension, express The gradient field in the d-th dimension.

[0096] Specifically, it is calculated using the following formula:

[0097]

[0098]

[0099]

[0100] In the above formula, Indicates y i The value of the z-th pixel along the k-th dimension of the j-th structure of interest. Indicates y i The (z+1)th pixel value along the k-th dimension of the j-th structure of interest. Indicates y i The (z-1)th pixel value of the j-th structure of interest along the k-th dimension.

[0101] Setting up the gradient field in this step can help understand the reasonable deformation trends of the structure of interest in data augmentation.

[0102] S6. For the structures of interest selected in step S2, generate the deformation coordinate field: for B i For each structure of interest, the generated deformation coordinate field is represented as:

[0103]

[0104] Specifically, it is calculated using the following formula:

[0105]

[0106] In the above formula, B i In Deformation coordinate field, express In the first-dimensional deformation coordinate field, express In the k-th dimension of the deformation coordinate field, express The deformation coordinate field in the d-th dimension.

[0107] Furthermore, combining the above equation, it can be seen that the generated deformation coordinate field is generated based on the initial coordinate field, deformation amplitude, and gradient field, where the local k-dimensional deformation direction is... The time indicates that no deformation is performed in the k-th dimension, and the coordinate field is initialized using the k-th dimension. Directly used as its k-th dimension deformation coordinate field

[0108] S7, Restore the deformation coordinate field: for B i Deformation coordinate field of each structure of interest Each dimension The deformed coordinate field can be restored using the following formula:

[0109]

[0110] In the above formula, express The restored deformation coordinate field, This represents the magnitude of xi in the k-th dimension. This step restores the deformed coordinate field because a normalization operation was performed during the previous initialization of the coordinate field.

[0111] S8. Mapping the deformed coordinate field: This involves mapping the restored deformed coordinate field... Acting on Generate the j-th structure of interest after deformation Thus forming the deformed x i Corresponding labeled data At the same time, T i Each of the ' Both act on x i Obtain deformed medical image data Forming deformed sample pairs

[0112] S9. Fusion Data: Randomly fusion of the original sample pairs (x) obtained in step S1. i y i The deformed sample pairs obtained from steps S8 and S8 The final sample pairs are obtained, denoted as Specifically, it is calculated using the following formula:

[0113]

[0114]

[0115] In the above formula, α is a manually set fusion parameter, and f is any reasonable one or more additional data augmentation operations. f is one or more additional data augmentation operations among any suitable data augmentation operations including but not limited to random cropping, random flipping, random rotation, gamma transformation and Gaussian blur.

[0116] Example 2

[0117] The difference between this embodiment and the previous embodiment is that in step S1, the collected images are industrial inspection images, such as industrial CT images, which include industrial inspection images and their corresponding structural segmentation maps. The structural segmentation maps include, but are not limited to, various industrial defects and part structures.

[0118] This invention utilizes the relationships between structures in an image to simulate the deformation of various structures, greatly alleviating the problem of image texture detail loss, achieving real-time data augmentation, improving computation speed, providing sufficient and reliable training data for subsequent deep neural network training, facilitating subsequent training, and improving the training effect of deep neural networks. It is worthy of promotion.

[0119] Finally, it should be noted that the above content is only used to illustrate the technical solution of the present invention, and is not intended to limit the scope of protection of the present invention. Simple modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention do not depart from the essence and scope of the technical solution of the present invention.

Claims

1. A method for augmenting training data of a deep neural network for image segmentation, characterized in that, Includes the following steps: S1. Collect training data to form several original sample pairs of training data; S2. For each pair of original samples formed in step S1, select the structure of interest to be augmented. S3. For the structures of interest selected in step S2 that need data augmentation, generate an initial coordinate field, and then normalize the initial coordinate field. S4. For the structures of interest selected in step S2 for data augmentation, set the data augmentation parameters, including deformation amplitude and deformation direction; S5. Calculate the gradient field of the structure of interest to be augmented; S6. Combining the normalized initial coordinate field from step S3, the deformation amplitude and direction set in step S4, and the gradient field calculated in step S5, generate the deformation coordinate field for the structure of interest to be augmented. S7. Restore the deformation coordinate field generated in step S6. S8. Map the deformation coordinate field obtained in step S7 to form deformation sample pairs; S9. Randomly merge the original sample pairs obtained in step S1 with the deformed sample pairs obtained in step S8 to obtain the final sample pairs.

2. The method for augmenting training data of a deep neural network for image segmentation according to claim 1, characterized in that, The training data collected in step S1 is represented as follows: D={(x1,…,x i ,…,x n ),(y1,…,y i ,…,y n )} in, In the above formula, D represents the training data, x1 represents the first image data, and x i Let x represent the i-th image data. n Let y1 represent the nth image data, and y1 represent the annotation data corresponding to x1. i x represents i The corresponding labeled data, y n x represents n Corresponding labeled data; This represents the labeled data for the first type of structure of interest. This represents the labeled data for the j-th structure of interest. This represents the annotation data for the m-th structure of interest; The original sample pairs formed in step S1 are represented as: (x i ,y i ).

3. The method for augmenting training data of a deep neural network for image segmentation according to claim 2, characterized in that, The specific method for step S2 is as follows: From y i From the m structures of interest, select B structures of interest that require data augmentation to form the structure of interest to be augmented, represented as: In the above formula, B i This represents the structure of interest to be augmented. This represents the labeled data for the structure of interest, type B.

4. The method for augmenting training data of a deep neural network for image segmentation according to claim 3, characterized in that, The initial coordinate field generated in step S3 is represented as follows: in, Specifically, the initial coordinate field is normalized using the following formula: In the above formula, C i Represented as B i Initialize the coordinate field. Represented as Initialize the coordinate field. Represented as Initialize the coordinate field. for Initialize the coordinate field. This represents the coordinates of the first dimension of x1. Represented as x i The coordinates of the k-th dimension, x represents i The coordinates of the d-th dimension, where d is the x-th dimension. i The number of dimensions, Represented as normalized x i The coordinates of the k-th dimension, Represented as x i The size in the k-th dimension.

5. The method for augmenting training data of a deep neural network for image segmentation according to claim 4, characterized in that, In step S5, the gradient field of the structure of interest to be augmented by the data is expressed by the following equation. in, Specifically, it is calculated using the following formula: In the above formula, δ i Indicates the corresponding B i gradient field, Indicates correspondence gradient field, Indicates correspondence gradient field, Indicates correspondence gradient field, express In the gradient field of the first dimension, express In the gradient field of the k-th dimension, express The gradient field in the d-th dimension; Indicates y i The value of the z-th pixel along the k-th dimension of the j-th structure of interest. Indicates y i The (z+1)th pixel value along the k-th dimension of the j-th structure of interest. Indicates y i The (z-1)th pixel value of the j-th structure of interest along the k-th dimension.

6. The method for augmenting training data of a deep neural network for image segmentation according to claim 5, characterized in that, The deformation coordinate field generated in step S6 is specifically represented as follows: Specifically, it is calculated using the following formula: In the above formula, B i In Deformation coordinate field, express In the first-dimensional deformation coordinate field, express In the k-th dimension of the deformation coordinate field, express Deformation coordinate field in the d-th dimension; Represented as The deformation amplitude in the k-th dimension, Represented as The deformation direction in the k-th dimension. Time indicates Perform data augmentation operations on the k-th dimension. Time indicates No data augmentation operation is performed on the k-th dimension.

7. The method for augmenting training data of a deep neural network for image segmentation according to claim 6, characterized in that, Step S7 specifically restores the deformed coordinate field using the following formula: In the above formula, express The restored deformation coordinate field, x represents i The size in the k-th dimension.

8. The method for augmenting training data of a deep neural network for image segmentation according to claim 7, characterized in that, The specific method for step S8 is as follows: Apply the restored deformation coordinate field to the x-axis in step S1. i y i Then, deformed sample pairs are formed, represented as 9. A method for augmenting training data of a deep neural network for image segmentation according to claim 8, characterized in that, The final sample pair in step S9 is represented as follows: Specifically, it is calculated using the following formula: In the above formula, α is a manually set fusion parameter, and f is one or more additional data augmentation operations, including but not limited to random cropping, random flipping, random rotation, gamma transformation, and Gaussian blur, among any suitable data augmentation operations.

10. A method for augmenting training data of a deep neural network for image segmentation according to claim 2, characterized in that, The image data in S1 is medical image data or industrial inspection image data; when the image data in S1 is medical image data, it includes, but is not limited to, one or more of the following medical image modal data: positron emission tomography, magnetic resonance imaging, magnetic resonance tomography, single photon emission computed tomography, and computed tomography.

Images