A Multimodal and Differential Enhancement-Based Method for Detecting Lymph Node Calcifications in Cholangiocarcinoma

Abstract

This invention discloses a method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement. Targeting the characteristics of calcifications in metastatic lymph nodes of cholangiocarcinoma under ultrasound imaging, this invention improves the image-text fusion module in the YOLO-World network by adding a differential enhancement algorithm, thereby increasing the accuracy of calcification detection without affecting detection speed. This invention is used for the target detection of calcifications in metastatic lymph nodes of cholangiocarcinoma under ultrasound images, offering advantages such as earlier detection, higher detection accuracy, and more precise localization.

Description

Technical Field

[0001] This invention belongs to the field of image processing and pattern recognition in computer vision, specifically relating to a method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement. Background Technology

[0002] Cholangiocarcinoma is characterized by its high malignancy and tendency to metastasize to lymph nodes. Recently, the development of novel drugs that can induce calcification in tumor cells and visualize it on imaging has made early and accurate prediction of cholangiocarcinoma possible. Therefore, the detection of calcification points in ultrasound images of cholangiocarcinoma lymph node metastases has become an important indicator after drug administration. However, ultrasound images of cholangiocarcinoma lymph node metastases have low resolution, high noise levels, and the calcification points often blend into the background, resulting in complex features. Currently used algorithms are mostly traditional image processing methods or machine learning classification tasks; there is still no technology for target detection of calcification points in lymph node metastases of various cancers, including cholangiocarcinoma.

[0003] In recent years, with the widespread application of deep learning in computer vision and the rapid development of GPUs, people have increasingly shifted their attention to deep learning, which has been widely applied in various computer vision fields and has become the mainstream algorithm. This invention addresses the problems of low resolution, high noise, complex calcification features, and frequent integration with the background in ultrasound images of cholangiocarcinoma lymph node metastases. It employs a multimodal and differential enhancement-based target detection method for detecting calcification points in ultrasound images of cholangiocarcinoma lymph node metastases, offering advantages such as earlier detection, higher detection accuracy, and more precise localization. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement. This method optimizes the YOLO-World network structure, enhances the sensitivity to calcifications, and effectively improves the accuracy of detecting calcifications in metastatic lymph nodes of cholangiocarcinoma.

[0005] A first aspect of the present invention provides a method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement, the method comprising the following steps:

[0006] Step 1: Obtain a dataset of calcification points from ultrasound images of lymph node metastases in cholangiocarcinoma;

[0007] Step 2: Train the YOLO-World network to obtain the YOLO-World pre-trained network model;

[0008] Step 3: Modify the structure of the YOLO-World pre-trained network model;

[0009] Step 4: On the dataset of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis, input the ultrasound image data of cholangiocarcinoma lymph node metastasis into the modified YOLO-World pre-trained network model, and retrain the modified YOLO-World pre-trained network model to obtain the target network model.

[0010] Step 5: Load the target network model into the real-time calcification detection system, and load the collected real-time ultrasound image data of cholangiocarcinoma lymph node metastasis into the real-time calcification detection system to realize calcification detection.

[0011] Furthermore, step 1 specifically involves using an ultrasound machine to acquire ultrasound images of lymph node metastasis in cholangiocarcinoma, manually marking calcification points, and obtaining a dataset of calcification points from ultrasound images of lymph node metastasis in cholangiocarcinoma.

[0012] Furthermore, step 2 involves training the YOLO-World network on the Objects365v1 large object detection dataset.

[0013] Furthermore, the modification in step 3 specifically involves changing the multi-scale feature concat connection in the YOLO-World pre-trained network model to a differential enhancement connection, thereby enhancing the sensitivity of the YOLO-World pre-trained network model to calcification points.

[0014] Furthermore, the multi-scale features specifically refer to three multi-scale features on the I-Pooling Attention module in the RepVL-PAN fusion module of the YOLO-World network.

[0015] Furthermore, step 3 modifies the structure of the YOLO-World pre-trained network model as follows:

[0016] The feature map is used to calculate the difference map and then global average pooling is performed. A gating mechanism is used to suppress weights below a set threshold to reduce noise interference. The processed weights are multiplied by a fusion base layer and background suppression is performed to further suppress background noise. The weighted and background-suppressed features are added to the original image to generate the final enhanced image output.

[0017] Furthermore, the working method of the real-time calcification point detection system is as follows:

[0018] S1. Real-time detection system for calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis: real-time reading of ultrasound images of cholangiocarcinoma lymph node metastasis.

[0019] S2. Ultrasound images and text information of cholangiocarcinoma lymph node metastasis are input into the target network model for forward inference.

[0020] S3. The real-time calcification point detection system marks the calcification points and indicates the presence of calcification points in the image.

[0021] Furthermore, S3 specifically refers to:

[0022] S3.1 Determine whether there are calcifications in the ultrasound image of lymph node metastasis in cholangiocarcinoma. If there are calcifications, proceed to S32; otherwise, proceed to S33.

[0023] S3.2 The real-time calcification point detection system marks the calcification points and indicates that calcification points exist in the image.

[0024] S3.3 Determine if there are any unread images. If so, return to S1; otherwise, end the detection.

[0025] A second aspect of the present invention provides an electronic device for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement methods, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the above-described method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement methods.

[0026] A third aspect of the present invention provides a computer-readable storage medium storing a computer program for performing the above-described method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement.

[0027] The technical solution provided by this invention has the following beneficial effects:

[0028] This invention modifies the text-image fusion module of the YOLO-World network by adding difference calculation to the I-PoolingAttention module. This calculates the pairwise differences between the input feature maps to capture variations and unique features between images. Global average pooling is used to process these difference maps, reducing the spatial information of each channel to a single average value to extract global features. A gating mechanism is then used to suppress weights below a set threshold to reduce noise and focus on important difference features. The processed weights are multiplied by a fusion layer to enhance important difference features. Background suppression is then performed to further highlight meaningful features and reduce background interference. Finally, the weighted and background-suppressed features are added to the original image to generate the final enhanced image output. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the real-time calcification point detection system according to an embodiment of this application.

[0030] Figure 2This is a modified I-Pooling Attention module structure diagram according to an embodiment of this application;

[0031] Figure 3 This is a structural diagram of the modified real-time calcification point detection system according to an embodiment of this application;

[0032] Figure 4 This paper compares the detection performance of the embodiments of this application with that of the benchmark network on a dataset of calcification points in ultrasound images of cholangiocarcinoma lymph node metastases. Detailed Implementation

[0033] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] This application addresses the challenges of limited data volume, small size, low resolution, and easy integration with the background of calcification points in ultrasound images of cholangiocarcinoma lymph node metastases, which significantly increase the difficulty of identification and detection using deep learning methods. It provides a method for detecting calcification points in cholangiocarcinoma lymph node metastases based on multimodal and differential enhancement techniques.

[0035] See Figure 1 and Figure 2 This application provides a method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement methods, the method comprising:

[0036] Step 1: Use an ultrasound machine to acquire ultrasound images of lymph node metastasis in cholangiocarcinoma, manually mark calcification points, and obtain a dataset of calcification points from ultrasound images of lymph node metastasis in cholangiocarcinoma.

[0037] Step 2: Train the YOLO-World network on the Objects365v1 large object detection dataset to obtain the pre-trained YOLO-World network model.

[0038] Step 3, as follows Figure 2 As shown, the structure of the pre-trained network model of YOLO-World is modified.

[0039] The low resolution of calcifications in lymph node metastases under ultrasound imaging makes them difficult to identify clearly. Furthermore, calcifications often blend into the background, making their features indistinct and susceptible to noise interference. In addition, the significant variations in the morphology and size of calcifications further complicate the detection process.

[0040] To address these characteristics, this application's embodiment modifies the concat connection of three multi-scale features into a difference-enhancing connection in the I-PoolingAttention module of the RepVL-PAN fusion module in YOLO-World. The feature maps are used to calculate difference maps and undergo pooling to capture variations between images, helping the model better identify these subtle changes and enhance their attention. A gating mechanism is used to suppress weights below a set threshold, thereby reducing noise interference to the model and focusing on the true signals of calcification points. The processed weights are multiplied by a fusion layer and background suppression is applied to further suppress background noise. Finally, the weighted and background-suppressed features are added to the original image to generate the final enhanced image output.

[0041] Specifically, the features in the improved YOLO-World pre-trained network model structure are processed according to the following steps:

[0042] (1) When the three input feature maps X1, X2, and X3 pass through the I-Pooling Attention module in the image-text fusion module of YOLO-World, the fusion base layer Fuse of the three feature maps is calculated, and then the difference maps are calculated pairwise and global average pooling is performed on the difference maps to obtain the difference matrix, where W diff (X a ,X b The difference matrix is ​​calculated from each pair of feature maps.

[0043] Fuse = X1 + X2 + X3

[0044] W diff (X a ,X b =GlobalAvgPool(X) a -X b (a,b∈{1,2,3},a≠b)

[0045] (2) For each difference matrix W obtained diff (X a ,X b The input is mapped to the range [0,1] using the Sigmoid function, then the set hyperparameter threshold is subtracted, and finally the difference weight W(X) between each pair of feature maps is obtained by the ReLU function. a ,X b ):

[0046] W(X a ,X b =ReLU(Sigmoid(W) diff (X a ,X b))-threshold)(a,b∈1,2,3,a≠b)

[0047] (3) The difference weights W(X) between each feature map and the other two feature maps are calculated. n ,X b ), W(X) n ,X b Add and merge with the underlying Fuse and the learnable background weights W background Multiplication yields the weighted fusion feature D Fn Finally, it is compared with the original feature map X. n The sum is the final output X′. n :

[0048] D Fn =(W(X) n ,X b )+W(X n ,X b ))·Fuse·W background (a,b,n∈{1,2,3},n≠a≠b)

[0049] X′ n =X n +D Fn (n∈{1,2,3})

[0050] Step 4: On the dataset of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis, input the ultrasound image data of cholangiocarcinoma lymph node metastasis into the modified YOLO-World pre-trained network model, and retrain the modified model to obtain the target network model.

[0051] Step 5: Load the target network model into the real-time calcification detection system, and load the real-time ultrasound image data of cholangiocarcinoma lymph node metastasis acquired by the ultrasound machine into the system to realize calcification detection.

[0052] The workflow of the real-time calcification point detection system is as follows: Figure 1 As shown, the architecture of the real-time calcification point detection system is as follows: Figure 3 As shown. The specific working steps of the real-time calcification point detection system include:

[0053] (1) The real-time calcification detection system reads ultrasound images of lymph node metastasis in bile duct cancer in real time.

[0054] (2) Ultrasound images and text information of cholangiocarcinoma lymph node metastasis are input into the target network model for forward inference.

[0055] (3) Determine whether there are calcifications in the ultrasound image of lymph node metastasis of bile duct cancer. If there are calcifications, proceed to step (4); otherwise, proceed to step (5).

[0056] (4) The real-time calcification point detection system marks the calcification points and indicates that there are calcification points in the image.

[0057] (5) Determine if there are any unread images. If there are, return to step (1); otherwise, end the detection.

[0058] Experimental results show that, compared with the benchmark network YOLO-World, the present invention significantly improves mAP50, mAP50-95, and recall, as detailed in Table 1.

[0059] Table 1 Comparison of detection performance between the present invention and the benchmark network

[0060] Model mAP50 (%) mAP 50-95 (%) Recall (%) YOLO-World 45.5 17.9 46.1 This invention 46.3(+0.8) 18.1(+0.3) 49.2(+3.1)

[0061] Meanwhile, this invention significantly outperforms the benchmark network YOLO-World on the validation set, with a significantly improved detection rate, as detailed in [link to details]. Figure 3 .

[0062] This application also discloses an electronic device for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement methods. The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of a method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement methods. The memory may include main memory, such as high-speed random access memory, or it may also include non-volatile memory, such as at least one disk storage device. The processor, network interface, and memory are interconnected via an internal bus, which may be an industry-standard architecture bus, a peripheral component interconnection standard bus, an extended industry-standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory stores the program; specifically, the program may include program code, which includes computer operation instructions. The memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0063] This application also discloses a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of a method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement. Specifically, the computer-readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. The volatile memory may include random access memory (RAM) and / or cache memory, etc. The non-volatile memory may include read-only memory (ROM), hard disk, flash memory, optical disk, magnetic disk, etc.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement, characterized in that, The method includes the following steps: Step 1: Obtain a dataset of calcification points from ultrasound images of lymph node metastases in cholangiocarcinoma; Step 2: Train the YOLO-World network to obtain the YOLO-World pre-trained network model; Step 3: Modify the structure of the YOLO-World pre-trained network model; Step 4: On the dataset of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis, input the ultrasound image data of cholangiocarcinoma lymph node metastasis into the modified YOLO-World pre-trained network model, and retrain the modified YOLO-World pre-trained network model to obtain the target network model. Step 5: Load the target network model into the real-time calcification detection system, and load the collected real-time ultrasound image data of cholangiocarcinoma lymph node metastasis into the real-time calcification detection system to realize calcification detection; Step 3 specifically involves: The feature map is used to calculate the difference map and then subjected to global average pooling. A gating mechanism is used to suppress weights below a set threshold in order to reduce noise interference; The processed weights are multiplied by a fused base layer and background suppression is applied to further suppress background noise; The weighted and background-suppressed features are added to the original image to generate the final enhanced image output.

2. The method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement according to claim 1, characterized in that, Step 1 specifically involves: Ultrasound images of lymph node metastases in cholangiocarcinoma were acquired using an ultrasound machine, and calcification points were manually marked to obtain a dataset of calcification points in ultrasound images of lymph node metastases in cholangiocarcinoma.

3. The method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement according to claim 1 or 2, characterized in that, Step 2 involves training the YOLO-World network on the Objects365v1 large object detection dataset.

4. The method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement according to claim 1, characterized in that, The specific modification in step 3 is to change the multi-scale feature concat connection in the YOLO-World pre-trained network model to a differential enhancement connection, so as to enhance the sensitivity of the YOLO-World pre-trained network model to calcification points.

5. The method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement according to claim 4, characterized in that, The multi-scale features are specifically the three multi-scale features on the I-PoolingAttention module in the RepVL-PAN fusion module of the YOLO-World network.

6. The method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement according to claim 1, characterized in that, The working method of the real-time calcification point detection system is as follows: S1. Real-time detection system for calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis: real-time reading of ultrasound images of cholangiocarcinoma lymph node metastasis. S2. Ultrasound images and text information of cholangiocarcinoma lymph node metastasis are input into the target network model for forward inference; S3. The real-time calcification point detection system marks the calcification points and indicates the presence of calcification points in the image.

7. The method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement according to claim 6, characterized in that, Specifically, S3 is: S3.1 Determine whether there are calcifications in the ultrasound image of lymph node metastasis in cholangiocarcinoma. If there are calcifications, proceed to S32; otherwise, proceed to S33. S3.2 The real-time calcification point detection system marks the calcification points and indicates that calcification points exist in the image; S3.3 Determine if there are any unread images. If so, return to S1; otherwise, end the detection.

8. An electronic device for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement methods, characterized in that, include: A memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the program, implements the method for detecting calcifications in lymph nodes of cholangiocarcinoma based on multimodal and differential enhancement as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the method for detecting calcifications in cholangiocarcinoma lymph nodes based on multimodal and differential enhancement as described in any one of claims 1-7.

Images