Ultrasonic endoscope diagnosis method, device, equipment, medium and product for pancreatitis

Abstract

The invention discloses an ultrasonic endoscope diagnosis method, device and equipment for pancreatitis, a medium and a product. The method comprises the following steps: acquiring pancreatic ultrasonic image data; based on a pre-trained ultrasonic endoscope image segmentation model, pancreatitis lesion area identification is carried out; wherein the ultrasonic endoscope image segmentation model comprises a cue word model and a pancreatitis lesion area identification module, the pancreatitis lesion area identification module adopts an SAM network and comprises an image encoder, a cue encoder and a mask decoder, and the output end of the cue word model is connected with the input end of the cue encoder; and taking the pancreatic ultrasonic image data as inputs of a cue word model and an image encoder, and identifying the pancreatitis lesion area. According to the method, the object can be accurately detected, so that the boundary of the object is described, richer context information is provided for a prompt encoder, and the identification precision of the pancreatitis lesion area is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer-aided medical technology, and in particular relates to an ultrasonic endoscopic diagnosis method, device, equipment, medium and product for pancreatitis. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Autoimmune pancreatitis (AIP) is a special type of chronic pancreatitis, which is divided into two subtypes: Type 1 AIP, also known as lymphoplasmacytic sclerosing pancreatitis (hereinafter referred to as type 1 AIP), is more common in elderly men and is caused by IgG4-related diseases involving the pancreas. Typical cases are often manifested by diffuse swelling of the pancreas and significantly increased serum IgG4 levels. This subtype is more common in East Asians; Type 2 AIP, also known as idiopathic ductal central pancreatitis, is rare in Asians. In focal lesions of autoimmune pancreatitis, it is often difficult to differentiate from pancreatic cancer because of the presence of peripancreatic lymphadenopathy and vascular invasion.

[0004] Endoscopic ultrasonography (EUS) can provide more information about the detailed features of the pancreatic parenchyma and pancreatic duct, but it also has its limitations. Ultrasound imaging has many interference factors and is prone to artifacts, which affects the accuracy of pancreatitis identification and still depends on the operator's subjective judgment and operating skills. Although there are some image segmentation algorithms, when applied to pancreatitis image segmentation, they still need to rely on manually provided annotation boxes, which is cumbersome. Summary of the invention

[0005] In order to overcome the deficiencies of the above-mentioned prior art, the present invention provides an ultrasonic endoscopic diagnosis method, device, equipment, medium and product for pancreatitis. By combining target detection and SAM network, the construction of a refined segmentation model of pancreatitis lesions is realized under coarse labeling and few-shot labeling, without the need for a large amount of manual labeling.

[0006] To achieve the above object, the first aspect of the present invention provides an endoscopic ultrasound diagnosis method for pancreatitis, comprising the following steps:

[0007] Acquire original pancreatic ultrasound image data;

[0008] Identify pancreatitis lesion areas based on a pre-trained endoscopic ultrasound image segmentation model;

[0009] Among them, the ultrasonic endoscopic image segmentation model is a multi-level recognition model, including an image encoder, a prompt encoder, a mask decoder and a prompt word model. The prompt word model provides prompt words for the multi-level training model through the prompt encoder; the original pancreatic ultrasound image data is used as input to the prompt word model and the image encoder respectively to identify the pancreatitis lesion area.

[0010] A second aspect of the present invention provides an ultrasonic endoscopic diagnostic device for pancreatitis, comprising:

[0011] The data acquisition module is configured to: acquire original pancreatic ultrasound image data;

[0012] The pancreatitis diagnosis module is configured to: identify pancreatitis lesion areas based on a pre-trained endoscopic ultrasound image segmentation model;

[0013] Among them, the ultrasonic endoscopic image segmentation model is a multi-level recognition model, including an image encoder, a prompt encoder, a mask decoder and a prompt word model. The prompt word model provides prompt words for the multi-level training model through the prompt encoder; the original pancreatic ultrasound image data is used as input to the prompt word model and the image encoder respectively to identify the pancreatitis lesion area.

[0014] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method described.

[0015] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0016] A fifth aspect of the present invention provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the method described is implemented.

[0017] One or more of the above technical solutions have the following beneficial effects:

[0018] By introducing the cue word model in the multi-level recognition network, the object can be accurately detected and its boundary can be depicted, providing the cue encoder with richer contextual information and improving the recognition accuracy of pancreatitis lesion areas; by introducing a multi-level training method, training can be performed based on roughly annotated pancreatic ultrasound image data, solving the problem of difficult or even impossible model training when the data annotation quality and amount are small. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0020] Figure 1 An example of ultrasound image annotation in an embodiment of the present invention;

[0021] Figure 2 This is a diagram of the architecture of an ultrasound endoscopic image segmentation model in an embodiment of the present invention;

[0022] Figure 3 Schematic diagram of predicted results and actual results in an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0024] In the description of the embodiments of the present application, the term “including” and similar terms should be understood as open inclusion, that is, “including but not limited to.” The term “based on” should be understood as “based at least in part on.”

[0025] One or more embodiments of the present invention provide an endoscopic ultrasonic diagnostic method for pancreatitis, comprising the following steps: acquiring original pancreatic ultrasonic image data; identifying pancreatitis lesion areas based on a pre-trained endoscopic ultrasonic image segmentation model; wherein the endoscopic ultrasonic image segmentation model is a multi-level recognition model, comprising an image encoder, a prompt encoder, a mask decoder and a prompt word model, wherein the prompt word model provides prompt words for the multi-level training model through the prompt encoder; and using the original pancreatic ultrasonic image data as input to the prompt word model and the image encoder, respectively, to identify pancreatitis lesion areas.

[0026] By introducing the cue word model into the multi-level recognition network, the object can be accurately detected and its boundary can be depicted, providing richer contextual information for the cue encoder and improving the recognition accuracy of pancreatitis lesion areas.

[0027] The training method of the ultrasonic endoscopic image segmentation model includes:

[0028] Step 1: Obtain pancreatic ultrasound image data with a first-level mask annotation, where the first-level mask annotation is the lesion area annotation.

[0029] All images were taken at high resolution and the endoscope models included (various models + companies). Only standard EUS images were considered eligible. The following images were excluded: images with poor quality due to blur or defocus, and non-endoscopic images.

[0030] Three experienced EUS endoscopists (two marking physicians and one endoscopist who reviews the image data) evaluated whether the image data were qualified (all physicians were senior EUS endoscopists with more than 3 years of EUS operation experience and more than 1000 practical cases). The endoscopists cooperated to demarcate and annotate the lesion area. Each image had to be demarcated and annotated by one physician and reviewed and supervised by another endoscopist. The image was qualified after the two physicians reached a consensus on the annotation.

[0031] For the EUS dataset, its annotation samples are as follows: Figure 1 As shown, based on the annotation data, the original image of each EUS and the corresponding Mask annotation of the original image can be obtained, and the bounding box annotation can be automatically generated based on the Mask annotation.

[0032] The labeled imaging data were divided into four different subgroups: AIP (autoimmune pancreatitis) group, chronic pancreatitis group, PDAC group, non-AIP pancreatic disease group, and NP (normal pancreas) group.

[0033] The imaging data of each subgroup were randomly divided into three data sets in equal proportions; training data set: internal validation set: internal test set = 8:1:1. The training data set and internal validation set were used to train the EUSAM model, and the internal validation set was used to preliminarily evaluate the performance of EUS-Intelligent.

[0034] After obtaining the training data set, the training data set is also enhanced. In some embodiments, three types of data enhancement methods are randomly selected to perform data enhancement on the ultrasound image data in the training data set to expand the training data set. Specifically, the data enhancement methods include pixel-level transformation methods, spatial-level transformation methods, and multi-sample synthesis methods, wherein the pixel-level transformation methods include blur (Gaussian blur, average blur, median blur, glass blur, dynamic blur), noise (Gaussian noise, random channel pixel inactivation, regional random inactivation, color space random transformation) and color interference (contrast, protection and brightness), and the degree of random transformation can be set. Spatial-level transformations include optical distortion, elastic transformation, affine transformation, etc.; multi-sample synthesis methods include mixup, cutmix, mosaic, etc.

[0035] Step 2: Train the endoscopic ultrasound image segmentation model. The endoscopic ultrasound image segmentation model (EUS-Intelligent) is constructed based on the segmentation model of the visual large model (EUSAM). Figure 2 As shown, the model architecture of the ultrasonic endoscopic image segmentation model includes a cue word model and a pancreatitis lesion area recognition module. The pancreatitis lesion area recognition module adopts a multi-level recognition model, including an image encoder, a cue encoder, a mask decoder and a cue word model. The output end of the cue word model is connected to the input end of the cue encoder.

[0036] The EUSAM model was trained and its diagnostic accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were tested using a test set. The ROC curve and MIoU were used to show the subgroup recognition ability of the EUSAM model, and the EUS-Intelligent algorithm was developed.

[0037] The training of the ultrasonic endoscopic image segmentation model adopts a three-stage pre-training + fine-tuning training strategy. The first stage trains the prompt word model, which is used to provide accurate bounding box prompt information for the large visual model; the second stage trains a pancreatitis lesion area recognition module based on the prompt word model, which is used to perform boundary segmentation of various types of pancreatitis based on the original image and the bounding box prompt information; the third stage trains the prompt word model and the pancreatitis lesion area recognition module jointly, and this training stage unifies the feature representation of the prompt word model and the pancreatitis lesion area recognition module.

[0038] The first stage: training the prompt word model to output pancreatic ultrasound image data with bounding box annotations.

[0039] Specifically, the prompt word model is trained based on the original image and the bounding box annotation information generated based on the mask. The prompt word model includes a backbone network (Backbone), a feature pyramid (Feature Pyramid) and a detection head (Head). The EUS image is subjected to data enhancement and data preprocessing, and the backbone network extracts image features. The feature pyramid enables the model to better adapt to the detection of lesion areas of different resolutions and sizes. The detection head is used to output the final pancreatitis lesion area detection results.

[0040] The pre-training loss function at this stage consists of two parts: category loss and position loss, where the category loss is defined as follows:

[0041]

[0042] The position loss is defined as follows:

[0043]

[0044] The evaluation indicators of the training process are mAP and mAR, where mAP is defined as follows:

[0045]

[0046] AP = 0.753

[0047] That is, first draw a PR curve based on the recognition results. The area under the PR curve is mAP. Similarly, mAR can be calculated.

[0048] The second stage: using the original pancreatic ultrasound image data as the input of the image encoder, using the pancreatic ultrasound image data carrying the bounding box annotation as the input of the prompt encoder, training the multi-level training model and outputting the pancreatic ultrasound image data carrying the secondary mask annotation.

[0049] Specifically, the pre-trained network of the first stage is fixed, and the original EUS image is passed through the pre-trained prompt word model to obtain the bounding box prediction of the pancreatitis lesion area in the original image. The bounding box prediction result is used as the input of the prompt encoder. At the same time, the original EUS image is subjected to data enhancement and image preprocessing as the input of the image encoder. This module extracts EUS image features and uses them together with the bounding box prompt information as the input of the mask decoder. Finally, the mask decoder outputs the pancreatitis Mask predicted based on the bounding box prompt information.

[0050] The loss function used in this stage of training is Dice Loss, which is defined as follows. The Dice coefficient is a function that measures the similarity of a set. It is usually used to calculate the similarity of two samples and its value range is [0,1]:

[0051]

[0052] Among them, |X∩Y| is the number of elements in the intersection of sets X and Y, and |X| and |Y| represent the number of elements in X and Y respectively. The numerator coefficient is 2 because the common elements between X and Y are repeatedly counted in the denominator. The larger the Dice, the more similar the elements of the two sets are, which means the better the model is trained. Therefore, Dice Loss is defined as:

[0053]

[0054] The evaluation index of the training process is MIoU, which is a standard evaluation index for segmentation problems. It calculates the overlap ratio of the intersection of two sets and their union. In the embodiment of the present invention, it calculates the intersection-over-union ratio between the real segmentation and the segmentation predicted by the system.

[0055] This ratio can be redefined as the number of true positive examples (intersection) divided by the total number (including true positive examples, false negative examples, and false positive examples (union)). IoU is also calculated by class and the average is taken.

[0056]

[0057] The third stage: freezing the image encoder, the prompt encoder and the prompt word model, training the mask decoder and outputting the pancreatic ultrasound image data annotated with three levels of mask.

[0058] In the third stage, which is also the last stage, the learning rate is adjusted to a very small value of about 1e-6, and the prompt word model and the pancreatitis lesion area recognition module are trained together, and the loss function and evaluation index of the training are the same as those of the second stage of pre-training. So far, the embodiment of the present invention has completed the pre-training stage of the EUSAM model.

[0059] By introducing a multi-level training method, training can be performed based on roughly labeled pancreatic ultrasound image data, which solves the problem of difficult or even impossible model training when the data annotation quality and amount are small.

[0060] In order to improve the accuracy of regional positioning of pancreatitis lesions in EUS data, the embodiment of the present invention has performed a second fine-tuning on the above-mentioned pre-trained model. Based on this, the embodiment of the present invention has constructed a training data set for fine-tuning that is only 1 / 10 of the original EUS pancreatitis diagnosis and annotation pre-training data set. The data set is first recognized by the pre-trained model EUSAM and then modified and annotated in more detail by the annotating doctor. Its characteristic is that the annotation accuracy is higher than that of the pre-training data set, but the number is less than that of the pre-training data set.

[0061] Specifically, the secondary fine-tuning includes: freezing the prompt word model, the image encoder and the prompt encoder structure in the pancreatitis lesion area recognition module, and the fine-tuning process only fine-tunes the mask decoder structure in the pancreatitis lesion area recognition module. This method can quickly fine-tune the EUSAM model and avoid the overfitting problem in the training process on a limited amount of data. The loss function and evaluation index used in the fine-tuning process are similar to those in the second stage of pre-training.

[0062] The EUSAM model after secondary fine-tuning is used as the online model for EUS pancreatitis lesion area identification and positioning finally deployed in the embodiment of the present invention, and is used as the core component of the EUS-Intelligent algorithm.

[0063] The embodiment of the present invention uses different model test indicators according to different tasks during the pre-training and fine-tuning process. In the training process of the prompt word model, the model test indicators of the embodiment of the present invention are mAP and mAR. In the training process of the pancreatitis lesion area recognition module, MIoU is used as the model test indicator. In the final fine-tuning stage, MIoU is used as the test indicator of the EUSAM model. This indicator accurately reflects the gap between the EUSAM prediction results and the actual results. Figure 3 shown.

[0064] That is, the closer the predicted Mask is to the Ground Truth Mask, the larger the MIoU will be, otherwise the MIoU will be small.

[0065] Negative sample data is created to reduce the misidentification rate of the lesion area. Negative sample data refers to images that are misidentified, that is, images that are not lesion areas are mistakenly identified as lesion areas. The purpose of training negative samples is to reduce the misidentification rate and improve the generalization ability of the neural network model. The collection of negative samples is to use the model trained with positive samples for testing, and get images that are misidentified. These images are collected as negative samples, and a special category is added to the training set for training, forming a new training set, and retraining a new model.

[0066] Based on the above method, one or more embodiments of the present invention further provide an ultrasonic endoscopic diagnostic device for pancreatitis, comprising:

[0067] The data acquisition module is configured to: acquire pancreatic ultrasound image data;

[0068] The pancreatitis diagnosis module is configured to: identify pancreatitis lesion areas based on a pre-trained endoscopic ultrasound image segmentation model;

[0069] Among them, the ultrasonic endoscopic image segmentation model is a multi-level recognition model, including an image encoder, a prompt encoder, a mask decoder and a prompt word model. The prompt word model provides prompt words for the multi-level training model through the prompt encoder; the original pancreatic ultrasound image data is used as input to the prompt word model and the image encoder respectively to identify the pancreatitis lesion area.

[0070] One or more embodiments of the present invention further provide an electronic device that can be used to implement the ultrasonic endoscopic diagnosis method for pancreatitis in the above embodiments. The electronic device includes one or more processors, one or more memories coupled to the processors, and a communication module coupled to the processors.

[0071] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic storage and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not last during the power outage duration. The computer program may be stored in the ROM. When the processor executes the computer program, the above-mentioned endoscopic ultrasound diagnosis method for pancreatitis is implemented.

[0072] In some embodiments, the program may be tangibly contained in a computer-readable medium, which may be included in a device (such as in a memory) or other storage device accessible by the device. The program may be loaded from the computer-readable medium to the RAM for execution. The computer-readable medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, and the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for endoscopic ultrasound diagnosis of pancreatitis.

[0073] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a server or terminal, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a server or terminal or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (such as a floppy disk, a hard disk and a tape, etc.), an optical medium (such as a digital video disk (digital video disk, DVD), etc.), or a semiconductor medium (such as a solid-state hard disk, etc.).

[0074] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the application. Some features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0075] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An endoscopic ultrasound diagnosis method for pancreatitis, characterized in that: The following steps are involved: Acquire original pancreatic ultrasound image data; Identify pancreatitis lesion areas based on a pre-trained endoscopic ultrasound image segmentation model; Among them, the ultrasonic endoscopic image segmentation model is a multi-level recognition model, including an image encoder, a prompt encoder, a mask decoder and a prompt word model. The prompt word model provides prompt words for the multi-level training model through the prompt encoder; the original pancreatic ultrasound image data is used as input to the prompt word model and the image encoder respectively to identify the pancreatitis lesion area.

2. The method for endoscopic ultrasound diagnosis of pancreatitis according to claim 1, characterized in that: The training method of the ultrasonic endoscopic image segmentation model includes: Obtain pancreatic ultrasound image data with primary mask annotation; Training the prompt word model to output pancreatic ultrasound image data with bounding box annotations; Using the original pancreatic ultrasound image data as the input of the image encoder, using the pancreatic ultrasound image data with bounding box annotations as the input of the prompt encoder, training the multi-level training model and outputting the pancreatic ultrasound image data with secondary mask annotations; The image encoder, the prompt encoder and the prompt word model are frozen, and the mask decoder is trained to output pancreatic ultrasound image data annotated with three levels of mask.

3. The method for endoscopic ultrasound diagnosis of pancreatitis according to claim 2, characterized in that: After obtaining the pancreatic ultrasound image data with annotated lesion areas as training data, a variety of data enhancement methods are randomly selected to enhance the ultrasound image data.

4. The method for endoscopic ultrasound diagnosis of pancreatitis according to claim 2, characterized in that: Acquire a recognition result obtained by the ultrasonic endoscopic image segmentation model, and perform fine-tuning based on the recognition result as a fine-tuning training data set; The frozen cue word model, as well as the image encoder and cue encoder in the pancreatitis lesion area recognition module, optimize the mask decoder based on the fine-tuning training data set.

5. The method for endoscopic ultrasound diagnosis of pancreatitis according to claim 2, characterized in that: The loss function used to train the prompt word model includes a category loss function and a position loss function.

6. The method for endoscopic ultrasound diagnosis of pancreatitis according to claim 2, characterized in that: The loss function used to train the pancreatitis lesion area recognition module and the ultrasound endoscopic image segmentation model is the Dice Loss function.

7. An ultrasonic endoscopic diagnostic device for pancreatitis, characterized in that: include: The data acquisition module is configured to: acquire original pancreatic ultrasound image data; The pancreatitis diagnosis module is configured to: identify pancreatitis lesion areas based on a pre-trained endoscopic ultrasound image segmentation model; Among them, the ultrasonic endoscopic image segmentation model is a multi-level recognition model, including an image encoder, a prompt encoder, a mask decoder and a prompt word model. The prompt word model provides prompt words for the multi-level training model through the prompt encoder; the original pancreatic ultrasound image data is used as input to the prompt word model and the image encoder respectively to identify the pancreatitis lesion area.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images