Bile duct cancer lymph node calcification point detection method based on multimodality and difference enhancement

Abstract

The invention discloses a biliary duct cancer lymph node calcification point detection method based on multiple modes and difference enhancement. According to the characteristics of bile duct cancer lymph node metastasis calcification points under ultrasonic images, an image text fusion module in a YOLO-World network is improved, a difference enhancement algorithm is added in the module, and the accuracy of calcification point detection is improved on the premise that the detection speed is not affected. The method is used for detecting the calcification point target of the bile duct cancer lymph node metastasis ultrasound image, and has the advantages of earlier detection, higher detection accuracy, more accurate positioning and the like.

Description

Technical Field

[0001] The present invention belongs to the field of image processing and pattern recognition in computer vision, and in particular relates to a method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and difference enhancement. Background Art

[0002] Cholangiocarcinoma is characterized by high malignancy and easy lymph node metastasis. Recently, the invention of some new drugs can induce calcification of tumor cells and image them in imaging, making it possible to accurately predict cholangiocarcinoma in the early stage. Therefore, the detection of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis has become an important indicator after medication. However, ultrasound images of cholangiocarcinoma lymph node metastasis have low resolution and many noise points, and the calcification points in them often blend with the background and have complex features. The algorithms currently used are mostly traditional image processing methods or machine learning classification tasks. There is no technology for target detection of calcification points in lymph node metastasis of various cancers, including cholangiocarcinoma.

[0003] In recent years, with the widespread application of deep learning in the field of computer vision and the rapid development of GPU, people have increasingly shifted their attention to deep learning, which has been widely used in a variety of computer vision fields and has become the current mainstream algorithm. Taking into account the problems of low resolution, a large number of noise points, complex features of calcification points, and the fact that they are often integrated with the background in ultrasound images of cholangiocarcinoma lymph node metastasis, the present invention adopts a target detection method based on multimodality and difference enhancement for target detection of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis, which has the advantages of earlier detection, higher detection accuracy, and more accurate positioning. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and difference enhancement. The method optimizes the network structure of YOLO-World, enhances the sensitivity to calcification points, and effectively improves the accuracy of detecting calcification points in cholangiocarcinoma lymph node metastasis.

[0005] The first aspect of the present invention provides a method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement, the method comprising the following steps:

[0006] Step 1, obtaining a dataset of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis;

[0007] Step 2: Train the YOLO-World network to obtain the YOLO-World pre-trained network model;

[0008] Step 3: Modify the structure of the YOLO-World pre-trained network model;

[0009] Step 4: On the calcification point dataset of cholangiocarcinoma lymph node metastasis ultrasound images, input the cholangiocarcinoma lymph node metastasis ultrasound image data into the modified YOLO-World pre-trained network model, retrain the modified YOLO-World pre-trained network model, and obtain the target network model;

[0010] Step 5: Load the target network model into the real-time calcification point detection system, and load the collected real-time cholangiocarcinoma lymph node metastasis ultrasound image data into the real-time calcification point detection system to realize calcification point detection.

[0011] Furthermore, the step 1 specifically involves using an ultrasound machine to collect ultrasound images of cholangiocarcinoma lymph node metastasis, manually marking calcification points, and obtaining a data set of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis.

[0012] Furthermore, step 2 trains the YOLO-World network on the Objects365v1 large-scale target detection dataset.

[0013] Furthermore, the modification in step 3 is specifically to modify the multi-scale feature concat connection in the YOLO-World pre-trained network model to a difference enhancement connection, so as to enhance the sensitivity of the YOLO-World pre-trained network model to calcification points.

[0014] Furthermore, the multi-scale features are specifically: 3 multi-scale features on the I-Pooling Attention module in the fusion module RepVL-PAN of the YOLO-World network.

[0015] Furthermore, step 3 modifies the structure of the YOLO-World pre-trained network model as follows:

[0016] The feature map is calculated as a difference map and subjected to global average pooling. The gating mechanism is used to suppress weights below the set threshold to reduce noise interference. The processed weights are multiplied with a fusion base layer and background suppression is performed to further suppress background noise. The weighted and background-suppressed features are added to the original image to generate the final enhanced image output.

[0017] Furthermore, the working method of the real-time calcification point detection system is as follows:

[0018] S1. The real-time detection system for calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis reads ultrasound images of cholangiocarcinoma lymph node metastasis in real time.

[0019] S2. Ultrasound images and text information of cholangiocarcinoma lymph node metastasis are input into the target network model for forward reasoning.

[0020] S3. The real-time calcification point detection system marks the calcification points and indicates that the image has calcification points.

[0021] Further, the S3 is specifically:

[0022] S3.1. Determine whether there are calcification points in the ultrasound image of cholangiocarcinoma lymph node metastasis. If there are calcification points, proceed to S32; otherwise, proceed to S33.

[0023] S3.2. The real-time calcification point detection system marks the calcification points and indicates that the image has calcification points.

[0024] S3.3. Determine whether there are any unread images. If so, return to S1. Otherwise, end the detection.

[0025] The second aspect of the present invention provides an electronic device for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement is implemented.

[0026] A third aspect of the present invention provides a computer-readable storage medium storing a computer program for executing the above-mentioned method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement.

[0027] The technical solution provided by the present invention has the following beneficial effects:

[0028] The present invention modifies the text image fusion module of the YOLO-World network, adds difference calculation in the I-PoolingAttention module, and calculates the difference between the input feature maps to capture the changes and unique features between the images. The above difference map is processed by global average pooling, the spatial information of each channel is reduced to a single average value, the global features are extracted, and the weights below the set threshold are suppressed using a gating mechanism to reduce noise and focus on important difference features. The processed weights are multiplied with a fusion base layer to strengthen important difference features, and then background suppression is performed to further highlight meaningful features and reduce background interference. Finally, the weighted and background-suppressed features are added to the original image to generate the final enhanced image output. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flowchart of the real-time calcification point detection system of the embodiment of the present application;

[0030] Figure 2This is a structural diagram of the I-Pooling Attention module after modification in the embodiment of the present application;

[0031] Figure 3 This is a structural diagram of the real-time calcification point detection system modified in the embodiment of the present application;

[0032] Figure 4 The detection effect of the embodiment of the present application and the benchmark network on the calcification point dataset of cholangiocarcinoma lymph node metastasis ultrasound images is compared. DETAILED DESCRIPTION

[0033] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0034] This application mainly addresses the problems of small amount of calcification point data in ultrasound images of cholangiocarcinoma lymph node metastasis, small targets and low resolution, easy blending with the background, and large morphological changes, which greatly increase the difficulty of using deep learning methods for identification and detection. It provides a cholangiocarcinoma lymph node calcification point detection method based on multimodality and difference enhancement.

[0035] See also Figure 1 and Figure 2 The present application embodiment provides a method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement, the method comprising:

[0036] Step 1: Use an ultrasound machine to collect ultrasound images of cholangiocarcinoma lymph node metastasis, manually mark calcification points, and obtain a cholangiocarcinoma lymph node metastasis ultrasound image calcification point dataset.

[0037] Step 2: Train the YOLO-World network on the Objects365v1 large-scale target detection dataset to obtain the YOLO-World pre-trained network model.

[0038] Step 3: Figure 2 As shown, the structure of the pre-trained network model of YOLO-World is modified.

[0039] The resolution of lymph node metastasis calcification points in ultrasound images is low, making it difficult to clearly identify the calcification points in the image. At the same time, the calcification points are often integrated with the background, making their features unclear and easily affected by noise. In addition, the shape and size of the calcification points vary greatly, making the detection task more complicated.

[0040] In view of these characteristics, the embodiment of the present application modifies the concat connection of three multi-scale features into a difference enhancement connection on the I-PoolingAttention module in the fusion module RepVL-PAN of YOLO-World. The feature map is calculated as a difference map and pooled to capture the changes between images, helping the model to better identify these subtle changes and enhance its attention. The gating mechanism is used to suppress weights below the set threshold, thereby reducing the interference of noise on the model and focusing on the real signals of calcification points. The processed weights are multiplied with a fusion base layer and background suppression is performed to further suppress background noise. Finally, the weighted and background-suppressed features are added to the original image to generate the final enhanced image output.

[0041] Specifically, the following steps are used to process and improve the features in the pre-trained network model structure of YOLO-World:

[0042] (1) When the three input feature maps X1, X2, and X3 pass through the I-Pooling Attention module in the image-text fusion module in YOLO-World, the fusion base layer Fuse of the three feature maps is calculated, and then the difference maps are calculated two by two and the difference maps are globally averaged and pooled to obtain the difference matrix, where W diff (X a ,X b ) is the difference matrix calculated from the pairwise feature maps:

[0043] Fuse=X1+X2+X3

[0044] W diff (X a ,X b )=GlobalAvgPool(X a -X b )(a,b∈{1,2,3},a≠b)

[0045] (2) For each difference matrix W obtained diff (X a ,X b ) is mapped to the range of [0,1] through the Sigmoid function input, and then the set hyperparameter threshold is subtracted. Finally, the difference weight W(X) between each pair of feature maps is obtained through the ReLU function. a ,X b ):

[0046] W(X a ,X b )=ReLU(Sigmoid(W diff (X a ,X b))-threshold)(a,b∈1,2,3,a≠b)

[0047] (3) The difference weight W(X) between each feature map and the other two feature maps n ,X b )、W(X n ,X b ) and combined with the fusion base layer Fuse and the learnable background weight W background Multiply to get the weighted fusion feature D Fn , and finally with the original feature map X n Add together to get the final output X′ n :

[0048] D Fn =(W(X n ,X b )+W(X n ,X b ))·Fuse·W background (a,b,n∈{1,2,3},n≠a≠b)

[0049] X′ n =X n +D Fn (n∈{1,2,3})

[0050] Step 4: On the calcification point dataset of cholangiocarcinoma lymph node metastasis ultrasound images, the cholangiocarcinoma lymph node metastasis ultrasound image data is input into the modified YOLO-World pre-trained network model, and the modified model is retrained to obtain the target network model.

[0051] Step 5: Load the target network model into the real-time calcification point detection system, and load the real-time cholangiocarcinoma lymph node metastasis ultrasound image data collected by the ultrasound machine into the system to realize calcification point detection.

[0052] The workflow of the real-time calcification point detection system is as follows: Figure 1 As shown, the architecture of the real-time calcification point detection system is as follows Figure 3 The specific working steps of the real-time calcification point detection system include:

[0053] (1) The real-time calcification point detection system reads the ultrasound image of cholangiocarcinoma lymph node metastasis in real time.

[0054] (2) The ultrasound images and text information of cholangiocarcinoma lymph node metastasis are input into the target network model for forward reasoning.

[0055] (3) Determine whether there are calcification points in the ultrasound image of cholangiocarcinoma lymph node metastasis. If there are calcification points, proceed to step (4); otherwise, proceed to step (5).

[0056] (4) The real-time calcification point detection system marks the calcification points and indicates the presence of calcification points in the image.

[0057] (5) Determine whether there are any unread images. If so, return to step (1); otherwise, terminate the detection.

[0058] After experimental verification, compared with the benchmark network YOLO-World, the mAP50, mAP50-95 and recall rate Recall of the present invention have been significantly improved. The specific results are shown in Table 1.

[0059] Table 1 Comparison of detection effects of the present invention and the benchmark network

[0060] Model mAP50(%) mAP50-95(%) Recall(%) YOLO-World 45.5 17.9 46.1 The present invention 46.3(+0.8) 18.1(+0.3) 49.2(+3.1)

[0061] At the same time, the performance of the present invention on the validation set is significantly better than that of the benchmark network YOLO-World, and the detection rate is significantly improved. Figure 3 .

[0062] The embodiment of the present application also discloses an electronic device for detecting calcification points of cholangiocarcinoma lymph nodes based on multimodality and differential enhancement, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for detecting calcification points of cholangiocarcinoma lymph nodes based on multimodality and differential enhancement when executing the computer program, wherein the memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc.; the processor, the network interface, and the memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc., and the bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store programs, and specifically, the program may include a program code, and the program code includes computer operation instructions. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0063] The embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps of the method for detecting calcification points of cholangiocarcinoma lymph nodes based on multimodality and differential enhancement are implemented. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement, characterized in that: The method comprises the following steps: Step 1, obtaining a dataset of calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis; Step 2: Train the YOLO-World network to obtain the YOLO-World pre-trained network model; Step 3: Modify the structure of the YOLO-World pre-trained network model; Step 4: On the calcification point dataset of cholangiocarcinoma lymph node metastasis ultrasound images, input the cholangiocarcinoma lymph node metastasis ultrasound image data into the modified YOLO-World pre-trained network model, retrain the modified YOLO-World pre-trained network model, and obtain the target network model; Step 5: Load the target network model into the real-time calcification point detection system, and load the collected real-time cholangiocarcinoma lymph node metastasis ultrasound image data into the real-time calcification point detection system to realize calcification point detection.

2. The method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement according to claim 1, characterized in that: The step 1 is specifically as follows: Ultrasound images of cholangiocarcinoma lymph node metastasis were collected using an ultrasound machine, and calcification points were manually annotated to obtain a dataset of calcification points in cholangiocarcinoma lymph node metastasis ultrasound images.

3. The method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement according to claim 1 or 2, characterized in that: The step 2 trains the YOLO-World network on the Objects365v1 large-scale target detection dataset.

4. The method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement according to claim 1, characterized in that: The modification in step 3 is specifically to modify the multi-scale feature concat connection in the YOLO-World pre-trained network model to a difference enhancement connection, so as to enhance the sensitivity of the YOLO-World pre-trained network model to calcification points.

5. The method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement according to claim 4, characterized in that: The multi-scale features are specifically: 3 multi-scale features on the I-PoolingAttention module in the fusion module RepVL-PAN of the YOLO-World network.

6. The method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement according to claim 1 or 4, characterized in that: Step 3 modifies the structure of the YOLO-World pre-trained network model as follows: Calculate the difference map of the feature map and perform global average pooling; The gating mechanism is used to suppress weights below the set threshold to reduce noise interference; The processed weights are multiplied with a fused base layer and background suppression is performed to further suppress background noise; The weighted and background suppressed features are added to the original image to generate the final enhanced image output.

7. The method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement according to claim 1, characterized in that: The working method of the real-time calcification point detection system is as follows: S1. The real-time detection system for calcification points in ultrasound images of cholangiocarcinoma lymph node metastasis reads ultrasound images of cholangiocarcinoma lymph node metastasis in real time; S2, the ultrasound images and text information of cholangiocarcinoma lymph node metastasis are input into the target network model for forward reasoning; S3. The real-time calcification point detection system marks the calcification points and indicates that the image has calcification points.

8. The method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement according to claim 7, characterized in that: The S3 is specifically: S3.

1. Determine whether there are calcification points in the ultrasound image of cholangiocarcinoma lymph node metastasis. If there are calcification points, proceed to S32; otherwise, proceed to S33. S3.2, the calcification point real-time detection system marks the calcification point and indicates that the image has the calcification point; S3.

3. Determine whether there are any unread images. If so, return to S1. Otherwise, end the detection.

9. An electronic device for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method for detecting calcification points in cholangiocarcinoma lymph nodes based on multimodality and differential enhancement as described in any one of claims 1 to 8.

Images