Bile duct cancer image data enhancement method based on Fourier transform and cross attention

Abstract

The invention discloses a bile duct cancer image data enhancement method based on Fourier transform and cross attention. According to the method, on the basis of the mechanism of extracting the phase and the amplitude of the image through Fourier transform, a cross attention mechanism is added, and a self-adaptive hybrid exchange method is provided, so that the calcification points on the finally obtained lymph node ultrasound image are more obvious, the contrast ratio of the calcification points and the background is effectively enhanced, and the accuracy of the image is improved. And meanwhile, tiny noisy points in the background are inhibited, and more importantly, the calcification points are accurately distinguished from white plaques with similar forms in the background. The method not only strengthens the detection capability of the model, but also shortens the time period from image acquisition to result analysis. The recognition accuracy of the target detection model on the calcification points is greatly improved, and the difficulty of detecting the calcification points by the model is greatly reduced.

Description

Technical Field

[0001] The invention belongs to the field of image processing and pattern recognition in computer vision, and specifically relates to a method for enhancing bile duct cancer image data based on Fourier transform and cross attention. Background Art

[0002] Cholangiocarcinoma is a highly heterogeneous malignant tumor originating from bile duct mucosal epithelial cells and is one of the most common malignant tumors. Inspired by the principle of biomineralization in nature, the German Journal of Angewandte Chemie published an article reporting a macromolecular drug that can selectively bind to the surface of tumor cells, enrich calcium ions in the fluid, induce spontaneous calcification and death of tumor cells, and propose a new strategy with great clinical application prospects.

[0003] When calcification reaches a certain level, a calcification point different from the surrounding tissue will appear in imaging, which can change and amplify the imaging characteristics of tumor cells, enhance the imaging characteristics of tumor cells in imaging detection, and even calcification at a microscale can be observed due to its imaging characteristics in imaging, achieving the goal of breaking through the current imaging limits and making accurate imaging detection of cholangiocarcinoma lymph node metastasis possible. Under this new strategy, ultrasound images of cholangiocarcinoma lymph node metastasis can be scanned and observed to diagnose whether the patient has lymph node metastasis. However, since calcification points are similar to lymph node margins, there is currently no research on target detection of calcification points of lymph node metastasis of various cancers, including cholangiocarcinoma, and the existing target detection methods are less effective in detecting cholangiocarcinoma lymph node calcification points.

[0004] The huge potential of AI in the field of oncology has also attracted more and more attention. It has been successfully applied in various stages. Its application methods include medical imaging recognition, big data analysis, and multimodal big data model construction, among which medical imaging analysis is a research hotspot. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention. The method is based on Fourier transform, and adds the input image and the generated image after noise addition and brightness change processing to the cross attention mechanism to obtain a reference image. At the same time, the method proposes an adaptive hybrid exchange method for hybrid exchange of the amplitude of the generated image and the input image, which greatly improves the accuracy of the baseline model in identifying calcification points.

[0006] The first aspect of the present invention provides a method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention, comprising the following steps:

[0007] Step 1, obtaining a data set of calcification points of cholangiocarcinoma lymph nodes;

[0008] Step 2: Design an image data enhancement module;

[0009] Step 3: inserting the image data enhancement module into the benchmark model, and using the cholangiocarcinoma lymph node calcification point dataset for training to obtain a modified benchmark model;

[0010] Step 4: Load the modified benchmark model into the cholangiocarcinoma lymph node calcification point detection system, and load the collected cholangiocarcinoma lymph node ultrasound data into the system for calcification point detection.

[0011] The second aspect of the present invention provides an electronic device for enhancing bile duct cancer image data based on Fourier transform and cross-attention, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for enhancing bile duct cancer image data based on Fourier transform and cross-attention when executing the program.

[0012] A third aspect of the present invention provides a computer-readable storage medium storing a computer program for executing the above-mentioned method for enhancing bile duct cancer image data based on Fourier transform and cross-attention.

[0013] The beneficial effects of the technical solution provided by the present invention are:

[0014] When processing image data, the primary challenge is the presence of a large amount of noise and non-specific features in the image, including the edges of lymph nodes and other white patches in the background. These features are highly similar to calcification points. In addition, the overall tone of the image is dim, making it difficult to accurately identify calcification points.

[0015] To address this problem, the present invention enhances the original ultrasound image. The present invention effectively enhances the contrast between the calcification point and the background, while suppressing the tiny noise points in the background. More importantly, it achieves accurate distinction between the calcification point and the white plaques with similar morphology in the background. The present invention not only enhances the detection capability of the model, but also shortens the time period from image acquisition to result analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of the cholangiocarcinoma lymph node calcification point detection system of the present application embodiment;

[0017] Figure 2 This is a structural diagram of the image data enhancement module in the embodiment of the present application;

[0018] Figure 3 This is a diagram showing the effects of the embodiments of the present application. DETAILED DESCRIPTION

[0019] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0020] like Figure 1 and Figure 2 As shown, the present invention provides a method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention, and the steps are as follows:

[0021] Step 1: Collect ultrasound images of cholangiocarcinoma lymph node calcification points and manually annotate the calcification points on the images to obtain a cholangiocarcinoma lymph node calcification point dataset.

[0022] Step 2: Design an image data enhancement module.

[0023] Further, if Figure 2 As shown, the working method of the image data enhancement module is:

[0024] S1. Process the input image to obtain a generated image, then segment the input image and the generated image, perform feature fusion on the generated image and the input image, and obtain a reference image.

[0025] S2. Perform Fourier transform on the input image and the reference image respectively, exchange the amplitude of the reference image with the amplitude of the input image, and perform inverse Fourier transform on the mixed and exchanged amplitude and phase of the input image to obtain a result image.

[0026] Further, the S1 is specifically:

[0027] The input image is processed, Gaussian noise is added, and the brightness of the image is adjusted to enhance the underlying detail information of the image to obtain the generated image.

[0028] The input image and the generated image are divided into multiple small patches, and the patches are flattened.

[0029] Both the generated image and the original input image are fed into the cross-attention module for feature fusion to produce a context-aware reference image. The reference image captures the information of the input image and the generated image, enhances the visibility of the object, and also maintains the original spatial relationship and semantic information of the image.

[0030] Further, the S2 is specifically:

[0031] Perform Fourier transform on the input image and the reference image respectively to obtain the corresponding amplitude spectrum and phase spectrum.

[0032] High-frequency amplitude is used in the input image, and low-frequency amplitude is used in the reference image. An adaptive threshold is used for hybrid exchange training, so that the amplitude exchange step can adaptively utilize the original input image. Finally, the low-frequency part of the amplitude of the input image is adaptively replaced by the amplitude of the reference image.

[0033] Among them, in the embodiment of the present application, the cross attention mechanism and the hybrid exchange are specifically processed according to the following steps to improve the features in the network structure:

[0034] (1) Set the input image to X = [X 1 ,X 2 ...X C ]∈R H×W×C , where H and W are the height and width of the image respectively, and C is the number of channels of the image.

[0035] Add noise and adjust the brightness of the input image to distinguish the calcification points from the background in the image, and make the subsequent feature extraction focus more on the calcification points rather than the background. The formula for this operation is:

[0036] I g =((I+η)-0.5)×c+0.5+b

[0037] Among them, I is the input image, I g is the generated image after enhancement, η represents Gaussian noise, c is the contrast coefficient, and b is the brightness coefficient.

[0038] (2) Set reference image I g Information must be captured from the input image I so that the amplitude of the reference image can focus on specific regions in the input data. Cross-attention between the input image and the generated image helps model the spatial correspondence between different source images. This helps extract appropriate features and achieve adaptive and balanced fusion.

[0039] The two images are divided into patches of size P×P×C (P is the set threshold), resulting in (H / P)×(W / P) patches for each image. Then, each patch is flattened and the learnable projection matrix W is used. e Linear projection into a token of size D. The formula can be expressed as:

[0040]

[0041] Among them, I t and is the label representation of the input and generated images; I p and are patch representations of the input image and the generated image, respectively, W eis the learnable projection moment.

[0042] (3) Cross-attention is applied between the two sets of markers to allow the reference image to focus on specific regions of the input image, which can be formulated using the scaled dot product attention mechanism:

[0043]

[0044] Among them, W q , W k and W v is the learnable projection matrix for Q, K, V, and A is the attention matrix representing the attention of the generated image to each patch of the input image.

[0045] (4) The output is normalized to zero mean and unit variance, then scaled by 255 times, and finally clipped to the range [0, 255]. The final result of the cross-attention module is the reference image I R .

[0046] (5) In order to make the amplitude component of the generated reference image more suitable than the amplitude component of the original image so as to distinguish the calcification point and its surrounding area, the embodiment of the present application mixes the input image with the context-aware reference image by exchanging the amplitude component in the Fourier transform of the reference image with the original input image. By retaining the phase component in the Fourier transform of the input image, the spatial structure and semantic information of the original input image are retained.

[0047]

[0048] Among them, F 1 and F 2 Fourier transform is performed on the input image and the reference image respectively, A 1 , A 2 are the amplitude spectra of the input image and the reference image, respectively. are the phase spectra of the input image and the reference image, respectively.

[0049] (6) The embodiment of the present application proposes an adaptive hybrid exchange method, rather than simply exchanging the amplitude of the reference picture to the original picture. Because the high-frequency component is responsible for the texture and details of the picture, the high-frequency amplitude is used in the Fourier transform of the original picture, and the low-frequency amplitude is used in the Fourier transform of the reference picture to transfer texture and color information from the reference picture.

[0050] In addition, unlike the existing method of setting a fixed threshold frequency, the embodiment of the present application adopts an adaptive threshold for hybrid exchange training, so that the amplitude exchange step can adaptively use the original input image. The low-frequency part of the amplitude of the input image is adaptively replaced by the amplitude of the reference image.β Expressed as a binary mask matrix with values ​​of 1 in the center region and 0 in the rest of the region, where β∈(0,1), the inverse Fourier transform of the adaptive hybrid amplitude exchange is given by:

[0051]

[0052] Finally, the enhanced image I * .

[0053] Step 3: The embodiment of the present application uses the YOLOv8 model as the benchmark model, inserts the image data enhancement module between the input image and the first Conv convolution module, and uses the collected cholangiocarcinoma lymph node calcification point dataset for training to obtain a modified YOLOv8 model.

[0054] Step 4: Load the modified YOLOv8 model into the cholangiocarcinoma lymph node calcification point detection system, and load the collected cholangiocarcinoma lymph node ultrasound data into the system to detect calcification points.

[0055] Further, if Figure 1 As shown, the working method of the cholangiocarcinoma lymph node calcification point detection system is:

[0056] (1) The cholangiocarcinoma lymph node calcification point detection system reads the ultrasound image of the lymph node calcification points.

[0057] (2) The ultrasound image input is added to the modified YOLOv8 model for forward reasoning.

[0058] (3) Performing image data enhancement processing on the ultrasound image.

[0059] (4) Determine whether there is a calcification point on the ultrasound image. If there is a calcification point, proceed to step (5); otherwise, proceed to step (6).

[0060] (5) Mark the calcification points and indicate that the image contains calcification points.

[0061] (6) Are there any unread images? If so, return to step (1); otherwise, terminate the detection.

[0062] like Figure 3 As shown, the effect of the embodiment of the present application can be seen:

[0063] The original image of calcification points in cholangiocarcinoma lymph nodes is lighter in color, with low contrast, and the calcification points are very similar to the tissues surrounding the lymph nodes.

[0064] The processed images of calcification points in cholangiocarcinoma lymph nodes have high contrast and can effectively distinguish the calcification points from the surrounding tissues of the lymph nodes.

[0065] The position of the calcification point is marked to determine whether the highlighted position of the calcification point in the processed image is correct. After comparison, it can be found that the position is correct and the effect is obvious.

[0066] At the same time, in order to verify the effectiveness of this method, corresponding experiments were carried out using multiple target detection networks, as shown in Table 1. Through experiments, it can be found that this method improves the detection ability of the target detection network for cholangiocarcinoma lymph node calcification points, proving that this method is effective.

[0067] Table 1 Comparison of the effects of this method and multiple target detection networks

[0068]

[0069]

[0070] The embodiment of the present application also discloses an electronic device for enhancing the data of bile duct cancer images based on Fourier transform and cross attention, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for enhancing the data of bile duct cancer images based on Fourier transform and cross attention when executing the computer program, wherein the memory may include a memory, such as a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk memory, etc.; the processor, the network interface, and the memory are interconnected through an internal bus, and the internal bus may be an industrial standard architecture bus, a peripheral component interconnection standard bus, an extended industrial standard architecture bus, etc., and the bus may be divided into an address bus, a data bus, a control bus, etc. The memory is used to store a program, and specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0071] The embodiment of the present application also discloses a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method for enhancing the cholangiocarcinoma image data based on Fourier transform and cross attention are implemented. Specifically, the computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory may include a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include a read-only memory (ROM), a hard disk, a flash memory, an optical disk, a magnetic disk, etc.

[0072] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention, characterized in that: The following steps are involved: Step 1, obtaining a data set of calcification points of cholangiocarcinoma lymph nodes; Step 2: Design an image data enhancement module; Step 3: inserting the image data enhancement module into the benchmark model, and using the cholangiocarcinoma lymph node calcification point dataset for training to obtain a modified benchmark model; Step 4: Load the modified benchmark model into the cholangiocarcinoma lymph node calcification point detection system, and load the collected cholangiocarcinoma lymph node ultrasound data into the system for calcification point detection.

2. The method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention according to claim 1, characterized in that: The step 1 is specifically as follows: Collect ultrasound images of calcification points in lymph nodes of cholangiocarcinoma; Obtain ultrasound images of cholangiocarcinoma lymph nodes and mark the calcification points on the images; Obtain a dataset of calcification points in lymph nodes of cholangiocarcinoma.

3. The method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention according to claim 1, characterized in that: The image data enhancement module performs the following operations: S1. Process the input image to obtain a generated image, segment the input image and the generated image, and perform feature fusion on the generated image and the input image to obtain a reference image. S2. Perform Fourier transform on the input image and the reference image respectively, exchange the amplitude of the reference image with the amplitude of the input image, and perform inverse Fourier transform on the mixed and exchanged amplitude and phase of the input image to obtain a result image.

4. The method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention according to claim 3, characterized in that: The S1 is specifically: S1.1, adding noise and adjusting the brightness of the input image to enhance the underlying detail information of the image and obtain a generated image; S1.2, split the input image and the generated image, and flatten each obtained patch; S1.

3. Perform feature fusion on the generated image and the original input image to obtain a reference image.

5. The method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention according to claim 4, characterized in that: The S1.2 is specifically: Set the width and height of the input image and the reference image to W and H respectively, and divide the input image and the reference image into patches of size P×P×C, so as to obtain H / P × W / P patches for each image; Flatten each patch and linearly project it into a token using a learnable projection matrix; Among them, P is the set patch size threshold, and C is the number of channels of the image.

6. The method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention according to claim 3, characterized in that: The S2 is specifically: Perform Fourier transform on the input image and the reference image respectively to obtain the corresponding amplitude spectrum and phase spectrum; Use high frequency amplitude in the input image and low frequency amplitude in the reference image to transfer texture and color information from the reference image; An adaptive threshold is used for hybrid exchange training so that the low-frequency part of the amplitude of the input image is adaptively replaced by the amplitude of the reference image; The mixed and swapped amplitude and phase of the input image are inverse Fourier transformed to obtain the resulting image.

7. The method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention according to claim 1, characterized in that: The step 3 is specifically as follows: The YOLOv8 model is preferably used as the benchmark model, and the image data enhancement module is inserted between the input image and the first Conv convolution module; The modified YOLOv8 model was trained using the cholangiocarcinoma lymph node calcification dataset.

8. The method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention according to claim 1, characterized in that: The cholangiocarcinoma lymph node calcification point detection system performs the following operations: Read ultrasound images of calcified lymph nodes in cholangiocarcinoma; Adding the ultrasound image to the modified baseline model for forward reasoning; performing image data enhancement processing on the ultrasound image; Determine whether there are calcification points on the ultrasound image. If yes, mark the calcification points and indicate that the image has calcification points. Otherwise, determine whether there are any unread images. If yes, return to the beginning. Otherwise, end the detection.

9. An electronic device for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for enhancing cholangiocarcinoma image data based on Fourier transform and cross-attention as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method for enhancing cholangiocarcinoma image data based on Fourier transform and cross attention as described in any one of claims 1 to 8.

Images