Method and device for training a chest-abdomen image recognition model

Abstract

The present application provides a kind of thoracoabdominal image recognition model training method and equipment, the method comprises: obtaining training data, including labeled thoracoabdominal image and unlabeled thoracoabdominal image;Thoracoabdominal image is identified using neural network model, neural network model includes teacher network and student network;Based on the first identification result of uncertainty data and teacher network output, the boundary voxel of the target of interest is determined, the feature data of at least part of boundary voxel is extracted in the first identification result, and then the boundary contrast loss is calculated, the consistency loss is calculated according to the first identification result output by teacher network and student network and uncertainty data, the supervised loss is calculated according to the difference between second identification result and label;According to boundary contrast loss, consistency loss and supervised loss, calculate the overall loss, and update the parameters of neural network model based on overall loss.

Description

Technical Field

[0001] This invention relates to the field of digital medical technology, specifically to a training method and device for a chest and abdomen image recognition model. Background Technology

[0002] With the rapid development of modern computer technology, digital diagnosis and treatment are playing an increasingly important role in the medical field. To non-invasively observe the actual condition inside a patient's body, various medical imaging technologies are being widely used in clinical practice. Different modalities of medical imaging have different imaging characteristics. For example, computed tomography (CT) images can be used to diagnose abnormalities related to muscle and bone tissue, while magnetic resonance imaging (MRI) images at different times can provide information about soft tissues. Directly observing corresponding images for diagnosis requires doctors to possess relevant professional knowledge and rich experience. Fully automated multimodal medical image segmentation delineates the doctor's targets of interest, such as cells, tissues, blood vessels, and organs, from medical images. Therefore, medical image segmentation can, to some extent, reduce the workload of doctors, decrease the experience required for observing medical images, assist doctors in making better diagnoses, prepare for subsequent surgical planning and navigation, and to a certain extent improve the success rate of subsequent surgeries.

[0003] However, training a high-precision, fully supervised deep neural network often requires a large amount of labeled data. Meanwhile, segmentation tasks can be viewed as assigning labels to each pixel or voxel in an image to be segmented. Compared to tasks such as classification, detection, and keypoint localization, pixel-by-pixel segmentation annotation often requires significantly more time and effort from annotators. Furthermore, unlike the annotation of two-dimensional natural images, medical images such as CT and MRI are three-dimensional images. While containing richer information, they also suffer from significant noise and interference during imaging, and the boundaries between different tissues and organs are blurred. For example, the annotation of certain specific targets in various modalities of medical images, such as chest and abdominal CT and MRI, is highly dependent on professional personnel, resulting in long and laborious annotation times and high costs. Annotated chest and abdominal medical image data is also often scarce. Therefore, training a high-precision, fully supervised deep network is difficult to achieve in the medical field. Summary of the Invention

[0004] In view of this, the present invention provides a training method for a chest and abdomen image recognition model, comprising: Acquire training data, including labeled chest and abdominal images and unlabeled chest and abdominal images, wherein the labels are annotation information for targets of interest in the chest and abdominal images, and the targets of interest are tissues, organs or lesions; The chest and abdomen images are identified using a neural network model, which includes a teacher network and a student network. The teacher network and the student network respectively identify the unlabeled chest and abdomen images and output a first identification result to represent the target of interest. The student network also identifies the labeled chest and abdomen images and outputs a second identification result to represent the target of interest. The teacher network also identifies multiple chest and abdomen images with added perturbation data and outputs a third identification result to represent the target of interest, and calculates uncertainty data based on the multiple third identification results. Based on the uncertainty data and the first recognition result output by the teacher network, the boundary voxels of the target of interest are determined. Feature data of at least some of the boundary voxels are extracted from the first recognition result, and then the boundary contrast loss is calculated. The consistency loss is calculated based on the first identification results output by the teacher network and the student network, as well as the uncertainty data. A supervised loss is calculated based on the difference between the second identification result and the label. ; Loss based on boundary comparison Consistency loss and supervised losses Calculate the overall loss and update the parameters of the neural network model based on the overall loss.

[0005] Optionally, the teacher network and the student network each include an encoder and two decoders, the two decoders outputting the first recognition result, the second recognition result, and the third recognition result in parallel; and calculating the contrast loss and the consistency loss. At that time, the two decoders of the teacher network and the student network are configured to cross-correspond.

[0006] Optionally, the parameters of the teacher network are calculated using the parameters of the student network: in, These are the parameters of the teacher network at the t-th training time. The teacher network mentioned in the first... The parameters at each training time point These are the parameters of the student network at t training times. It is a coefficient that controls the degree of moving average.

[0007] Optionally, the first recognition result, the second recognition result, and the third recognition result respectively include the segmentation result for the target of interest, the distance from each voxel to the edge of the target of interest, and high-level feature data.

[0008] Optionally, the uncertainty data includes segmentation result uncertainty data. and distance uncertainty data .

[0009] Optionally, the boundary voxels of the target of interest are determined, and feature data of at least a portion of the boundary voxels are extracted from the first identification result, specifically including: Using the uncertainty data of the segmentation results The boundary voxels of the target of interest indicated by the segmentation result in the first recognition result output by the teacher network are filtered using a preset uncertainty threshold to obtain a high-confidence set of boundary voxels. Further select a subset of boundary voxels from the set of boundary voxels with high confidence. High-level feature data of some boundary voxels are extracted from the high-level feature data output by the student network and the teacher network, respectively.

[0010] Optionally, calculate the boundary contrast loss. Specifically, it includes: High-level feature data of some background voxels are extracted from the high-level feature data output by the student network; The boundary contrast loss is calculated using the high-level feature data of the partial boundary voxels and the high-level feature data of the partial background voxels. .

[0011] Optionally, a consistency loss is calculated based on the first identification results output by the teacher network and the student network, as well as the uncertainty data. Specifically, it includes: Calculate the segmentation loss based on the segmentation results output by the teacher network and the student network. ; The distance loss is calculated based on the distance from each voxel output by the teacher network and the student network to the edge of the target of interest. ; Using the uncertainty data of segmentation results Distance uncertainty data Segmentation loss and distance loss Calculate consistency loss .

[0012] Optionally, the overall loss is calculated as follows: ,in For the overall loss, These are preset coefficients.

[0013] Accordingly, the present invention provides a training device for a chest and abdomen image recognition model, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the above-described training method for the chest and abdomen image recognition model.

[0014] According to the training method and apparatus for a chest and abdomen image recognition model provided in this embodiment of the invention, this scheme alleviates the coupling problem between the teacher network and the student network, obtains better latent feature representations and better pseudo-labels, and ensures the consistency of the model's output when the input is slightly perturbated. The boundary comparison mechanism in this scheme addresses the problem of easily misidentifying the boundaries of targets of interest in chest and abdomen images. It focuses on sampling the features of boundary points in a high-dimensional feature space, introduces feature-level contrast loss, distinguishes between the target of interest and the background, thereby imposing shape constraints at the feature level and ensuring the model's ability to distinguish different categories and clusters. This scheme also uses uncertainty estimation operations, using uncertain data to calculate consistency loss and boundary contrast loss, ensuring that the model learns more robustly. When using this scheme, users only need a small amount of labeled chest and abdomen image data, combined with unlabeled data, to train a neural network model with high recognition accuracy, achieving semi-supervised training while ensuring high recognition accuracy. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram illustrating the training process of the recognition model in an embodiment of the present invention; Figure 2 This is a schematic diagram of a preferred teacher network and student network structure in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of collecting positive samples in an embodiment of the present invention. Detailed Implementation

[0017] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] In the description of this invention, it should be noted that the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, the technical features involved in the different embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0019] This invention provides a training method for a chest and abdomen image recognition model. The recognition model described in this application refers to a model based on neural network results, which includes a teacher network and a student network. The structures of these two networks can be completely identical, for example, each including an encoder and a decoder.

[0020] The training data used in this scheme includes labeled and unlabeled chest and abdomen images. The labels are annotation information for targets of interest in the chest and abdomen images.

[0021] The target of interest can be an organ or lesion, and the specific chest and abdominal images with appropriate annotations are selected based on clinical needs. Labeled chest and abdominal images are created by first observing the images and identifying a target of interest (which could be a lung lobe, atrium, blood vessel, tumor, or other lesion, etc.), and then using annotation tools to mark its extent in the image.

[0022] Chest and abdominal images can be MRI (Magnetic Resonance Imaging) or CT (Computed Tomography) images, or ultrasound images. A primary purpose of identifying tissues, organs, or lesions from these images is for surgical planning, such as determining the puncture path to the lesion and avoiding blood vessels or other vital organs.

[0023] In one embodiment, to increase the amount of training data, the original chest and abdomen image was data augmented to generate more samples. For example, operations such as adjusting the brightness, rotating, and horizontally flipping the original chest and abdomen image by 180 degrees were performed to augment the data.

[0024] Then, the amplified chest and abdomen image set was randomly divided into training and test sets at a ratio of 4:1; and the training set was further divided into labeled training dataset and unlabeled training dataset at a ratio of 1:4.

[0025] Complete chest and abdominal images are quite large. Considering computational and hardware costs, in actual training, the complete images need to be sampled before being input into the neural network model. For example, multiple chest and abdominal CT images can be sampled to obtain four training data blocks with dimensions of 128, 128, and 96 respectively. Two of these blocks are labeled, and the remaining two are unlabeled. The labeled and unlabeled data are subjected to strong transformations (color transformations and cropping, etc.) and then used as input to the student network. The training data blocks are subjected to small Gaussian random noise and used as input to the teacher network.

[0026] like Figure 1 As shown, during training, the neural network model identifies chest and abdominal images. The teacher network and student network identify unlabeled chest and abdominal images, each outputting a first identification result representing the target of interest. The student network also identifies labeled chest and abdominal images, outputting a second identification result representing the target of interest. The teacher network further identifies multiple chest and abdominal images with added perturbation data, outputting a third identification result representing the target of interest, and calculates uncertainty data based on these third identification results. Based on the uncertainty data and the first identification results output by the teacher network, the boundary voxels of the target of interest are determined. Feature data of at least some boundary voxels are extracted from the first identification results, and then the boundary contrast loss is calculated. The consistency loss is calculated based on the initial identification results output by the teacher network and the student network, as well as the uncertainty data. Supervised loss is calculated based on the difference between the second identification result and the label. .

[0027] Loss based on boundary comparison Consistency loss and supervised losses Calculate the overall loss and update the parameters of the neural network model based on the overall loss. In a specific embodiment, the overall loss is calculated as follows: ,in For the overall loss, These are preset coefficients.

[0028] Based on the overall loss, stochastic gradient descent is used to update the student network parameters, and the updated student network parameters are used to update the teacher network parameters in a certain way. The initial learning rate is 10⁻³, and the learning rate changes to 1 / 10 every 2000 iterations.

[0029] The relationship between the parameters of the teacher network and the student network can be calculated using a moving average method: ,in These are the parameters of the teacher network at training time t. The teacher network is in the first Parameters at each training time. These are the parameters of the student network at t training times. It is a coefficient that controls the degree of moving average.

[0030] It should be noted that the above calculation method for overall loss and the relationship between the parameters of the teacher network and the student network are only the methods selected in the specific implementation process, and do not mean that this calculation method is the only option.

[0031] like Figure 2 As shown, in a preferred embodiment, the teacher network and the student network each include an encoder and two decoders, with the two decoders outputting a first recognition result, a second recognition result, and a third recognition result in parallel. The contrast loss is calculated... and consistency loss At that time, the two decoders of the teacher network and the student network are configured in a cross-correspondence manner. That is, on unlabeled data, the output of the teacher network supervises the student network in an alternating manner, i.e., the first decoder (decoder) The output of the supervised student network is the second decoder (decoder) ), the second decoder (decoder) The output of the supervised student network is the first decoder (decoder) ).

[0032] Furthermore, in this embodiment, the teacher network and student network are configured to perform three recognition tasks, namely, outputting segmentation results for the target of interest. Distance from each voxel to the edge of the target of interest and high-level feature data Therefore, each decoder has three outputs. Among them, the high-level feature data... Also known as a feature map, it is a high-dimensional feature vector (N-dimensional matrix) that is a feature extracted from an image by a higher layer in a neural network. This refers to the distance (Euclidean distance) from a voxel to the boundary of the target of interest. For example, a voxel located outside the target of interest has a distance of... For voxels that are positive and located within the target of interest For negative, target boundary voxels =0. Segmentation results for the target of interest. This can also be called a segmentation map, specifically the probability that each voxel in the image belongs to the target of interest, with a value ranging from 0 to 1. To ensure the low-entropy property of the network, i.e., to ensure a relatively deterministic output, the final segmentation output is obtained by applying a sharpening function to the original probability distribution: in This indicates the final segmentation result. The original probability. This is the set value.

[0033] Accordingly, when calculating uncertain data, it is necessary to calculate the uncertainty data of the segmentation task. and Uncertainty data of the task Specifically, this embodiment uses Monte Carlo random loss and excludes certain neural units from computation, adds subtle perturbations to the teacher network input, and performs calculations. This calculation process is repeated 10 times to obtain a series of segmentation tasks and... Predict the output of the task and use the entropy of the segmentation result as the segmentation uncertainty. Using the variance of SDF prediction results to approximate the uncertainty of SDF prediction : The calculated uncertainty data is used to weight the corresponding loss function.

[0034] Regarding the above boundary comparison, combined with Figure 3 As shown, the present invention provides a preferred embodiment, which specifically includes the following operations: Using the uncertainty data of segmentation results By using a preset uncertainty threshold to filter the boundary voxels of the target of interest indicated by the segmentation results in the first recognition result output by the teacher network, a high-confidence set of boundary voxels is obtained, thereby improving the robustness of the boundary sampling strategy.

[0035] Next, since there are often many edge points in the 3D chest and abdomen image, in order to limit the number of edge points sampled in the final sample, a random mask will be generated in this embodiment. Only a small portion of the high confidence edge points (voxels) obtained in the previous step will be selected to participate in the subsequent calculation. That is, a portion of the boundary voxels will be further selected from the high confidence boundary voxel set, thereby saving GPU memory of the computing device.

[0036] The finally selected boundary voxels are called anchor points, and the coordinates corresponding to the anchor point set can be obtained. Then, the corresponding anchor point feature vector set is sampled from the high-level feature output of the student network, and the positive sample feature set is sampled from the high-level feature output of the corresponding teacher network. That is, the high-level feature data of the aforementioned boundary voxels are extracted from the high-level feature data output by the student network and the teacher network respectively, and then the boundary contrast loss is calculated. .

[0037] This embodiment calculates the boundary contrast loss. Furthermore, it is also necessary to obtain the features of negative samples for computation. Specifically, high-level feature data of some background voxels are extracted from the high-level feature data output by the student network. The background refers to the area outside the target of interest. Background voxels can be selected by random sampling to obtain the corresponding high-level feature data.

[0038] Then, the boundary contrast loss is calculated using the high-level feature data of some boundary voxels and some background voxels. Specifically, feature-level contrastive loss InfoNCE was employed. in, Represents feature-level boundary contrast loss. It is the feature vector (high-level feature data) of the anchor point obtained by sampling the foreground boundary from the feature map output by the student network. Representative teacher network feature graph and anchor points The corresponding feature vector (high-level feature data). This represents the negative feature vector (high-level feature data) obtained by randomly sampling background voxels from the student network feature map with pseudo-labels.

[0039] Regarding the consistency loss in this scheme This is used to measure the consistency of outputs from the teacher network and the student network when performing the same recognition task on the same unlabeled chest and abdomen image. Since the two networks in this scheme perform two tasks—predicting the segmentation result and predicting the distance from each voxel to the edge of the target of interest—the consistency of these two tasks must be calculated separately, while also utilizing the aforementioned segmentation result uncertainty data. Distance uncertainty data It participates in the calculation. Therefore, the consistency loss is calculated. Specifically, it includes: Calculate the segmentation loss based on the segmentation results output by the teacher network and the student network. ; The distance loss is calculated based on the distance from each voxel output by the teacher network and student network to the edge of the target of interest. ; Using the uncertainty data of segmentation results Distance uncertainty data Segmentation loss and distance loss Calculate consistency loss In a specific embodiment, the calculation is performed as follows: ; in" " indicates a combined operation of multiplying and averaging two matrices. and The loss was calculated using the mean square error method.

[0040] After the neural network model is trained according to the above embodiments, the teacher network can be used to identify the chest and abdomen image to be identified. It should be noted that if... Figure 2 The network structure shown can be used to average the outputs of the two decoders to obtain the final recognition result.

[0041] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0042] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0043] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0044] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0045] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A training method for a chest and abdomen image recognition model, characterized in that, include: Acquire training data, including labeled chest and abdominal images and unlabeled chest and abdominal images, wherein the labels are annotation information for targets of interest in the chest and abdominal images, and the targets of interest are tissues, organs or lesions; The chest and abdomen image is identified using a neural network model, which includes a teacher network and a student network. The teacher network and the student network identify the unlabeled chest and abdomen image respectively and output a first identification result to characterize the target of interest. The student network also identifies the labeled chest and abdomen images and outputs a second identification result to characterize the target of interest; The teacher network also identifies multiple chest and abdominal images with added perturbation data, outputs a third identification result to characterize the target of interest, and calculates uncertainty data based on multiple third identification results; Based on the uncertainty data and the first recognition result output by the teacher network, the boundary voxels of the target of interest are determined. Feature data of at least some of the boundary voxels are extracted from the first recognition result, and then the boundary contrast loss is calculated. The consistency loss is calculated based on the first identification results output by the teacher network and the student network, as well as the uncertainty data. A supervised loss is calculated based on the difference between the second identification result and the label. ; Loss based on boundary comparison Consistency loss and supervised losses Calculate the overall loss and update the parameters of the neural network model based on the overall loss; The first identification result, the second identification result, and the third identification result respectively include the segmentation result for the target of interest, the distance from each voxel to the edge of the target of interest, and high-level feature data; the uncertainty data includes segmentation result uncertainty data. and distance uncertainty data ; Determine the boundary voxels of the target of interest, and extract feature data of at least a portion of the boundary voxels from the first identification result, specifically including: Using the uncertainty data of the segmentation results The boundary voxels of the target of interest indicated by the segmentation result in the first recognition result output by the teacher network are filtered using a preset uncertainty threshold to obtain a high-confidence set of boundary voxels. Select a subset of boundary voxels from the set of boundary voxels with high confidence. High-level feature data of the partial boundary voxels are extracted from the high-level feature data output by the student network and the teacher network, respectively. Calculate boundary contrast loss Specifically, it includes: High-level feature data of some background voxels are extracted from the high-level feature data output by the student network, where the background voxels refer to the region outside the target of interest; The boundary contrast loss is calculated using the high-level feature data of the partial boundary voxels and the high-level feature data of the partial background voxels. .

2. The method according to claim 1, characterized in that, The teacher network and the student network each include an encoder and two decoders. The two decoders output the first recognition result, the second recognition result, and the third recognition result in parallel. The boundary contrast loss is calculated during the calculation of the boundary contrast loss. and the aforementioned consistency loss At that time, the two decoders of the teacher network and the student network are configured to cross-correspond.

3. The method according to claim 1, characterized in that, The parameters of the teacher network are calculated using the parameters of the student network: ， in, These are the parameters of the teacher network at the t-th training time. The teacher network mentioned in the first... The parameters at each training time point These are the parameters of the student network at t training times. It is a coefficient that controls the degree of moving average.

4. The method according to claim 1, characterized in that, The consistency loss is calculated based on the first identification results output by the teacher network and the student network, as well as the uncertainty data. Specifically, it includes: Calculate the segmentation loss based on the segmentation results output by the teacher network and the student network. ; The distance loss is calculated based on the distance from each voxel output by the teacher network and the student network to the edge of the target of interest. ; Using the uncertainty data of segmentation results Distance uncertainty data Segmentation loss and distance loss Calculate consistency loss .

5. The method according to claim 1, characterized in that, The overall loss calculation method is as follows ,in For the overall loss, These are preset coefficients.

6. A training device for a chest and abdomen image recognition model, characterized in that, include: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the training method of the chest and abdomen image recognition model as described in any one of claims 1-5.

Images