Medical image segmentation method in low-resource scene

Abstract

The invention discloses a medical image segmentation method in a low-resource scene, and the method comprises the steps: training a NanoDet model based on a preset training data set until the labeling precision of a labeling box of a focus region obtained through the NanoDet model is greater than a first precision value; setting an increase item matrix for a weight matrix of a segmentation model SAM, and updating the increase item matrix based on the training data set, so that the segmentation precision of the segmentation model SAM is greater than a second precision value; generating an annotation box of a to-be-segmented image based on the NanoDet model, and obtaining a positioning point of a focus area based on the annotation box; and based on the positioning point coordinates in the marking box, performing image segmentation of a focus area by using the segmentation model SAM after weight updating, and generating a focus mask corresponding to the image to be segmented. The method has better segmentation capability and precision.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing, and in particular relates to a medical image segmentation method in a low-resource scenario. Background Art

[0002] Early detection and treatment of diseases such as cancer are of decisive significance for improving patients' survival rate and treatment effect. Among many examination methods, the acquisition of images of lesion areas has become an important tool for breast cancer screening and diagnosis. However, the interpretation of medical images such as ultrasound images and CT images and the accurate extraction of lesion areas are extremely challenging tasks. Due to the existence of image quality problems such as noise, artifacts, and low contrast, as well as the diversity of breast tumor morphology and the fuzziness of boundaries, the difficulty of image data segmentation is significantly increased. In addition, the grayscale distribution of tumor tissue overlaps with that of surrounding normal tissues such as breast glands and fat, further increasing the risk of regional misdetection and missed detection during the segmentation process.

[0003] Deep learning models such as convolutional neural network (CNN), U-Net and its variants, and SAM have made significant progress in the field of medical image segmentation. However, there are still problems such as high degree of model coupling, high model learning cost, and poor segmentation effect.

[0004] Therefore, in view of the above technical problems, it is necessary to provide an improved effective segmentation strategy for medical-related images.

[0005] The information disclosed in this background technology section is only intended to enhance the understanding of the overall background of the invention and should not be regarded as an acknowledgment or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the invention

[0006] The purpose of the present invention is to provide a medical image segmentation method in a low-resource scenario, which can accurately segment the lesion area.

[0007] In order to achieve the above object, a technical solution provided by a specific embodiment of the present invention is as follows:

[0008] In a first aspect, the present invention provides a medical image segmentation method in a low-resource scenario, comprising:

[0009] Training the NanoDet model based on a preset training data set until the annotation accuracy of the annotation box of the lesion area obtained by the NanoDet model is greater than a first accuracy value, the training data set comprising an original image set, an annotation box of the lesion area of ​​the original image, and a lesion mask after the original image is segmented;

[0010] Setting an additional item matrix for the weight matrix of the segmentation model SAM, and updating the additional item matrix based on the training data set so that the segmentation accuracy of the segmentation model SAM is greater than the second accuracy value, and the number of rows and columns of the additional item matrix and the weight matrix are the same;

[0011] Generate a labeling frame of the image to be segmented based on the NanoDet model, and obtain a positioning point of the lesion area based on the labeling frame;

[0012] Based on the coordinates of the positioning points in the annotation frame, the image segmentation of the lesion area is performed using the segmentation model SAM with updated weights to generate a lesion mask corresponding to the segmented image.

[0013] In one or more embodiments of the present invention, generating a lesion region positioning point of an image to be segmented includes:

[0014] Obtaining the pixel value of each point in the image within the marked frame;

[0015] Based on the distribution of pixel values ​​in the annotation box, the coordinates of the positioning points of the lesions in the annotation box are generated.

[0016] In one or more embodiments of the present invention, the positioning point coordinate formula is:

[0017]

[0018] Among them, x′ is the horizontal coordinate of the positioning point, y′ is the vertical coordinate of the positioning point, and array(x,y) is the pixel value at the pixel (x,y).

[0019] In one or more embodiments of the present invention, updating the added item matrix based on the training data set includes:

[0020] Decomposing the added item matrix into a first matrix and a second matrix to be multiplied;

[0021] Keeping the weight matrix unchanged, obtaining the back propagation loss function in the segmentation model SAM training;

[0022] Based on the back-propagation loss function, the first matrix and the second matrix are updated.

[0023] In one or more embodiments of the present invention, the formula for updating the first matrix and the second matrix is:

[0024]

[0025] Wherein, A is the first matrix before updating, B is the second matrix before updating, A' is the first matrix after updating, B' is the second matrix after updating, and η is the preset learning rate.

[0026] In one or more embodiments of the present invention, the method further comprises:

[0027] The ranks of the first matrix and the second matrix are both smaller than the rank of the segmentation model SAM weight matrix.

[0028] In one or more embodiments of the present invention, the labeling accuracy and / or segmentation accuracy calculation formula is:

[0029]

[0030] Among them, P is the annotation accuracy or segmentation accuracy; α and β are custom weight values; S c S is the area of ​​the overlap between the model-annotated annotation box and the manually annotated annotation box, or the area of ​​the overlap between the mask output by the segmentation model SAM and the manually annotated mask; p is the area of ​​the manually labeled annotation box or the area of ​​the manually labeled mask; S m The area of ​​the annotation box for the model, or the area of ​​the mask output by the segmentation model SAM.

[0031] Compared with the prior art, the medical image segmentation method in low-resource scenarios provided by the present invention optimizes the amount of data updated during model training by redefining the weight matrix of the segmentation model SAM, thereby reducing the load pressure of the model during training. At the same time, the generation of the annotation box is realized through the NanoDet model, and the training difficulty is relatively small, but the accuracy of the mask generated for further segmentation of the image is greatly improved, and the segmentation accuracy is improved in low-resource scenarios with poor image quality. On the other hand, the present invention realizes the decoupling of medical image processing, and can accurately locate the source of errors through the output of the model, and debug each sub-model separately. Furthermore, on the basis of the annotation box, the present invention sets the calculation logic of the positioning point, so that the segmentation accuracy of the segmentation model SAM is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 is a schematic diagram of a medical image segmentation scene in one embodiment of the present invention;

[0034] Figure 2 is a schematic flow chart of a medical image segmentation method in one embodiment of the present invention;

[0035] Figure 3 It is a flowchart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0036] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific implementation modes.

[0037] Unless explicitly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising”, etc., will be understood to include the stated elements or components but not to exclude other elements or components.

[0038] Please refer to Figure 1 , which is a schematic diagram of an application scenario of the medical image segmentation method provided by the present invention under an implementation mode, and the scenario specifically includes:

[0039] It should be noted that communication connections are provided between the NanoDet model 101, the segmentation model SAM 102 and the user terminal 103, and the communication network derived from the above communication connection may include various connection types, including but not limited to: wired connection, wireless connection or optical fiber cable connection, etc. At the same time, the communication network may be a local area network, a metropolitan area network, a wide area network or any combination of the three.

[0040] The NanoDet model 101 is used to generate an identification frame for identifying the lesion area in the corresponding image based on the input medical image, and analyze the pixels in the identification frame according to the position of the identification frame to obtain the lesion area positioning point. The segmentation model SAM102 is used to segment the lesion area based on the identification frame and the positioning point to obtain the lesion area mask. The user terminal 103 is used to provide the user with a configurable interface to configure relevant weights, model parameters, cleaning degree calculation formulas, etc.

[0041] It should be noted that the user terminal 103 is installed with a computer software program that matches the medical image segmentation method provided by the present method; the user terminal 103 may include but is not limited to a desktop computer (PC), a desktop computer, a smart phone, a handheld computer, a tablet computer, a personal digital assistant (PDA), and other portable electronic devices or wearable electronic devices, and the embodiments of the present invention do not limit the above contents.

[0042] It should also be noted that the medical image segmentation method of the embodiment of the present invention can be applied to the medical image segmentation system of the embodiment of the present invention. The hierarchical quantitative learning system can be configured in a terminal. The terminal can include but is not limited to a PC (Personal Computer), a PDA (tablet computer), a smart phone, a smart wearable device, and the like.

[0043] Please refer to Figure 2 FIG. 1 is a flow chart of a medical image segmentation method according to an embodiment of the present invention. The medical image segmentation method specifically comprises the following steps:

[0044] S201: Training a NanoDet model based on a preset training data set until the labeling accuracy of the labeling box of the lesion area obtained by the NanoDet model is greater than a first accuracy value.

[0045] In the medical image segmentation method provided by the present invention, the training process of the NanoDet model adopts a back-propagation algorithm, and the classification performance is improved by optimizing the loss function and adjusting the weight parameters of each layer.

[0046] When it is necessary to explain, the training data set includes an original image set, a labeling frame of the lesion area of ​​the original image, and a lesion mask after the original image is segmented. Among them, the original image refers to the relevant image of the lesion site directly obtained based on the medical image acquisition device. The labeling frame of the original image is used to train the NanoDet model, so that the model can use the labeling frame to label the lesion area based on the input image; the mask is used to train the segmentation model SAM, so that the segmentation model SAM can segment the lesion area in a given area and generate a mask. In an embodiment of the present invention, the training data set for training the segmentation model SAM and the training data set for training the neural network model of the labeling frame may include the same, different or not completely the same original images. In order to ensure the implementation of the solution in a low-resource scenario and reduce the cost of manual annotation, it is preferred to use the same original image to generate the corresponding training data set. Further, in an embodiment of the present invention, the original image may include but is not limited to: ultrasound images, CT images, nuclear magnetic resonance images, etc. The embodiment of the present invention is not limited to this.

[0047] Further, in the training data set, the annotation frame of the original image is artificially set on each of the original images to mark the lesion area in the original image. It is understandable that in an original image, the number of the annotation frames is equal to the number of independent lesion areas in the image, that is, the number of annotation frames in the same original image can be one or more. A annotation frame should completely surround a lesion area. And the embodiment of the present invention does not limit the shape and size of the annotation frame. In the processing of medical images of a specific category, since the properties of the lesion are relatively single, a annotation frame that is more in line with the shape of the lesion can be used for marking, so as to maximize the annotation accuracy and reduce the number of pixels in the non-lesion area in the annotation frame. For example, for lesion areas that are usually spherical, such as local cysts, a circular annotation frame can be used for marking.

[0048] In order to enable the subsequent segmentation process to achieve the best segmentation effect within the annotation box and enhance the adaptability of the medical image segmentation method provided by the present invention, a relatively universal shape with better compatibility with other systems can be used as the annotation box, while avoiding the presence of too many non-lesion area pixels in the annotation box. Therefore, preferably, the annotation box can be the minimum circumscribed rectangle of the lesion area.

[0049] It is understandable that in the training process of the neural network model, in addition to the training data set, a test data set is also required to confirm the accuracy of the neural network model and to determine whether the current model meets the requirements. The test data set also includes the original image and the lesion area identification frame and lesion area mask manually annotated on the original image.

[0050] In an exemplary embodiment of the present invention, testing the annotation accuracy of the model specifically includes: inputting the original image in the test data set into the trained NanoDet model, and outputting the corresponding medical image containing the lesion area annotation box after processing by the model. At this time, the annotation accuracy of the model is:

[0051]

[0052] Among them, P is the annotation accuracy, α and β are custom weight values, S c is the area of ​​the overlap between the model annotation box and the manually annotated annotation box, S p is the area of ​​the manually labeled box, S m The area of ​​the dimension box that dimensions the model.

[0053] It can be understood that the above-mentioned weight value α determines the influence of whether the lesion area is completely labeled on your accuracy value in the calculation of the labeling accuracy, and the weight value β determines the influence of the non-lesion area in the labeled area on the labeling accuracy in the calculation of the labeling accuracy. When the calculated labeling accuracy is greater than the preset first accuracy value, the model training is completed; if the calculated labeling accuracy is less than or equal to the preset first accuracy value, the model accuracy is insufficient and further training is required. The embodiment of the present invention does not limit the size of the weight value and the size of the first accuracy value. Based on different requirements for model processing in different scenarios, the weight value and the size of the first accuracy value can be adaptively adjusted to improve the overall adaptability of the solution.

[0054] S202: Setting an additional item matrix for the weight matrix of the segmentation model SAM, and updating the additional item matrix based on the training data set so that the segmentation accuracy of the segmentation model SAM is greater than a second accuracy value.

[0055] In an exemplary embodiment of the present invention, updating the added item matrix based on the training data set includes: decomposing the added item matrix into a first matrix and a second matrix that are multiplied; keeping the weight matrix unchanged, and obtaining a back propagation loss function in the segmentation model SAM training; and updating the first matrix and the second matrix based on the back propagation loss function.

[0056] It is understandable that when the segmentation model SAM is forward propagated, the input is multiplied by the weight matrix to obtain the corresponding output. The update of the weight matrix will cause the original performance of the model to be lost. Therefore, in an embodiment of the present invention, an increase item matrix is ​​set in the weight matrix. The number of rows and columns of the increase item matrix is ​​consistent with the number of columns of the weight matrix, so that the two can be added. During the training process of back propagation, only the increase item matrix is ​​trained, and the original weight matrix is ​​frozen so that the original weight matrix does not change due to training.

[0057] On the other hand, since the dimension of the added item matrix is ​​consistent with that of the original weight matrix, that is, the number of rows and columns of the added item matrix and the weight matrix are the same. In this case, due to the limitation of the dimension of the model matrix, updating the added item matrix will still result in low update efficiency and large workload. Therefore, the embodiment of the present invention further decomposes the added item matrix into a first matrix and a second matrix. The ranks of the first matrix and the second matrix are both less than the rank of the segmentation model SAM weight matrix.

[0058] Furthermore, the formula for updating the first matrix and the second matrix is:

[0059]

[0060]

[0061] Wherein, A is the first matrix before updating, B is the second matrix before updating, A' is the first matrix after updating, B' is the second matrix after updating, and η is the preset learning rate.

[0062] For example, the weight matrix W∈R d×k , decompose W into two smaller low-rank matrices A and B, W = W0 + ΔW. Where W0 is the original weight matrix and ΔW is the added term matrix. ΔW is expressed as: ΔW = AB, A∈R d×r ,B∈R r×k ,r<<min(d,k), where r is the rank of the low-rank matrix. During fine-tuning, only A and B are updated, while W0 remains unchanged and A and B are initialized. Random initialization or an approximation obtained from the singular value decomposition (SVD) of W can be used. The new weight matrix W is calculated during the forward propagation process. new =AB, the loss function calculated during the back propagation process is equivalent to W new The gradient is: Then use the chain rule to calculate the loss function which is equivalent to the gradient of A and B: Use the gradient descent optimization algorithm to update A and B, Where η is the learning rate, and the forward propagation and back propagation are repeated to update the parameters until the training is completed.

[0063] It should also be noted that, similar to the annotation accuracy, after model training, a corresponding test data set is also required to test the segmentation accuracy. When the segmentation accuracy is greater than the preset second accuracy value, the segmentation model SAM training is completed; conversely, when the segmentation accuracy is less than or equal to the second accuracy value, it means that the current segmentation model SAM does not meet the requirements and needs further training. The formula for segmentation accuracy refers to the previous text. For the annotation accuracy calculation method, just replace the corresponding annotation box area with the mask area.

[0064] S203: generating a labeling frame of the image to be segmented based on the NanoDet model, and acquiring a positioning point of the lesion area based on the labeling frame;

[0065] In the prior art, segmentation models such as SAM cannot accurately locate the segmented object, and the segmentation results contain a large amount of invalid image information. In the embodiment of the present invention, the corresponding annotation box is generated by the aforementioned NanoDet model, which can replace the segmentation model SAM to narrow the segmented area.

[0066] Since the annotation box does not completely fit the lesion area, there are still pixels in the non-lesion area. Therefore, in order to further improve the segmentation accuracy of the segmentation model SAM, the present invention further determines the positioning point information of the lesion area based on the acquired annotation box information, and divides the lesion area in the annotation box based on the positioning point information.

[0067] In an exemplary embodiment of the present invention, generating a lesion area positioning point of an image to be segmented includes: obtaining pixel values ​​of each point of the image within the annotation frame; and generating the positioning point coordinates of the lesion within the annotation frame based on the distribution of pixel values ​​within the annotation frame.

[0068] The pixel value is the value assigned by the computer when the image is digitized. It represents the average brightness information of a small square in the original image, or the average reflection (transmission) density information of the small square. When converting a digital image into a halftone image, the dot area rate (dot percentage) is directly related to the pixel value (grayscale value) of the digital image, that is, the dot represents the average brightness information of a small square in the original with its size.

[0069] The above-mentioned small squares are approximated as points. The lesion area is usually lower in brightness than the normal area. Therefore, the approximate area where the lesion is located can be divided based on the distribution of pixel values. For example, based on the clustering algorithm, the number of data clusters obtained by the clustering algorithm is set to 2; the two pixel points with the largest pixel value difference are selected as the first centroid and the second centroid; the pixel points in the annotation box are clustered using the difference in pixel values ​​of the pixels in the image as the distance to obtain two clusters of pixel points. The group with higher pixel values ​​is considered to be the pixel points in the lesion area, and then the positioning point is set in the lesion area.

[0070] In order to facilitate the computer to accurately divide the positioning points and simplify the operation, the present invention preferably sets the positioning point coordinate formula as follows:

[0071]

[0072] Among them, x′ is the horizontal coordinate of the positioning point, y′ is the vertical coordinate of the positioning point, and array(x,y) is the pixel value at the pixel (x,y).

[0073] S204: Based on the coordinates of the positioning points in the annotation frame, the segmentation model SAM with updated weights is used to perform image segmentation of the lesion area to generate a lesion mask corresponding to the segmented image.

[0074] In order to further clarify the technical solution of the present invention, a specific embodiment is now given as follows:

[0075] like Figure 3The flowchart of the specific embodiment is as follows. First, a small amount of breast ultrasound data images are collected, and pre-processing techniques such as image denoising and contrast enhancement are used to improve the quality of ultrasound images, and then data annotation is performed on the area to be segmented. Next, the NanoDet network is used to classify and identify breast ultrasound data, and the tumor area is calibrated by box selection. In this process, the image centroid is obtained through network training, and the centroid and the target box are passed to the SAM model as segmentation prompt information, and the SAM model is fine-tuned to further improve the segmentation accuracy. Finally, the SAM model accurately segments the breast ultrasound image based on these prompt information and generates a segmentation mask for the tumor area. This series of operations not only improves the accuracy of segmentation, but also realizes the integrated processing of image detection, classification and segmentation.

[0076] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A medical image segmentation method for low-resource scenarios, characterized in that: include: Step 1, training the NanoDet model based on a preset training data set until the annotation accuracy of the annotation box of the lesion area obtained by the NanoDet model is greater than a first accuracy value, the training data set includes an original image set, an annotation box of the lesion area of ​​the original image, and a lesion mask after the original image is segmented; As a further optimization explanation, the improved method of NanoDet is used in step 1 to improve the model's ability to cope with complex tasks and enhance the model's detection accuracy of breast ultrasound images. Step 2, setting an additional item matrix for the weight matrix of the segmentation model SAM, and updating the additional item matrix based on the training data set so that the segmentation accuracy of the segmentation model SAM is greater than the second accuracy value, and the number of rows and columns of the additional item matrix and the weight matrix are the same; As a further optimization description, the added item matrix is ​​decomposed into a first matrix and a second matrix that are multiplied; the weight matrix is ​​kept unchanged, and a back propagation loss function is obtained in the segmentation model SAM training; Based on the back propagation loss function, update the first matrix and the second matrix. Step 2 The formulas of the first matrix and the second matrix are: Wherein, A is the first matrix before updating, B is the second matrix before updating, A' is the first matrix after updating, B' is the second matrix after updating, and η is the preset learning rate. Step 3: Generate a labeling frame of the image to be segmented based on the NanoDet model, and obtain the positioning point of the lesion area based on the labeling frame; As a further optimization explanation, the lesion area positioning points of the image to be segmented are generated in step 3, including: Obtain the pixel value of each point in the image within the annotation frame; based on the distribution of the pixel values ​​within the annotation frame, generate the coordinates of the positioning point of the lesion within the annotation frame. The formula for the positioning point coordinates is: Wherein, x' is the horizontal coordinate of the positioning point, y' is the vertical coordinate of the positioning point, and array(x,y) is the pixel value at the pixel (x,y). Step 4: Based on the coordinates of the positioning points in the annotation frame, the weighted segmentation model SAM is used to perform image segmentation of the lesion area to generate a lesion mask corresponding to the segmented image. As a further optimization explanation, the calculation formula for the annotation accuracy and / or segmentation accuracy in step 4 is: Among them, P is the annotation accuracy or segmentation accuracy; α and β are custom weight values; S c S is the area of ​​the overlap between the model-annotated annotation box and the manually annotated annotation box, or the area of ​​the overlap between the mask output by the segmentation model SAM and the manually annotated mask; p is the area of ​​the manually labeled annotation box or the area of ​​the manually labeled mask; S m The area of ​​the annotation box for the model, or the area of ​​the mask output by the segmentation model SAM.

Images