Unsupervised deep learning image splicing method for portable fundus color photo equipment

Abstract

The invention discloses an unsupervised deep learning image splicing method for portable fundus color photo equipment. The method comprises the following steps: constructing a multi-scale feature extraction module of an image registration network; the feature matching module is used for constructing an image registration network; the multi-scale feature extraction module and the feature matching module are sequentially connected to form the constructed image registration network; constructing a shallow feature extractor; constructing a feature reconstruction module; constructing a joint generator; sequentially connecting the shallow feature extractor, the feature reconstruction module and the seam generator to form a fusion network; constructing a fusion network; connecting the image registration network and the image fusion network to form an image splicing model; collecting data; preprocessing the data; obtaining a trained image registration network; obtaining a trained image fusion network; forming a fundus image splicing model by using the trained image registration network and the trained image fusion network; and splicing the images by using the eye fundus image splicing model. According to the method, a high-quality fundus spliced image can be efficiently generated.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and medical image processing, and specifically relates to an unsupervised deep learning image stitching method for a portable fundus color photography device. Background Art

[0002] As an important tool for ophthalmic diagnosis, fundus color photography can directly reflect the health status of the retina, macula, optic disc and other important tissues, and is widely used in the screening and diagnosis of diseases such as diabetic retinopathy, macular degeneration, and glaucoma. However, traditional fundus image acquisition equipment is usually bulky and expensive, and has high technical requirements for operators, which limits its large-scale popularization and application in primary medical institutions, remote areas and home scenarios. In recent years, the rapid development of portable fundus color photography equipment has provided a new direction for solving this problem. These devices have become an important tool for promoting early screening of fundus diseases with their miniaturization, ease of use and economy.

[0003] Due to the limitations of optical design, portable fundus color photography equipment can usually only obtain local images with a small field of view, while comprehensive and clear fundus images are essential for a comprehensive assessment of retinal health. Therefore, stitching multiple local fundus images to generate a large-scale panoramic retinal image has become an important requirement in diagnosis and treatment. However, images from portable devices usually have problems such as low resolution, high noise, and distortion at the edge of the field of view, which increases the difficulty of image stitching. In addition, due to the instability of handheld devices during shooting, there are significant translations, rotations, and scale changes between images. These factors make image stitching a technical challenge.

[0004] Existing fundus image stitching methods mainly rely on manually annotated feature points or supervised deep learning models. Although these methods have improved the stitching accuracy to a certain extent, they still have significant shortcomings in practical applications, which seriously restricts their promotion and application in complex medical scenarios.

[0005] First, traditional methods rely heavily on manual annotation. In order to achieve high-quality fundus image stitching, it is usually necessary to accurately annotate the feature points in the image. This process not only requires the annotator to have professional medical knowledge, but also takes a lot of time and effort, resulting in a significant increase in the annotation cost. In addition, since fundus images contain complex vascular structures, lesion areas, and lighting differences, it is difficult to annotate feature points. The annotation results are easily affected by human factors. For example, the subjective judgment of different annotators may be biased, resulting in unstable quality of the annotation data. This problem is particularly prominent in the training of deep learning models that require a large amount of high-quality annotated data support, which not only restricts the training effect of the model, but also limits its application in diversified data.

[0006] Secondly, the supervised deep learning model's reliance on labeled data also brings obvious limitations. Although this method performs well under specific conditions, it usually requires a large number of labeled samples to learn the feature matching relationship of image stitching. However, fundus images vary significantly between different devices, patients, and shooting conditions, resulting in the model's high dependence on training data, making it difficult to maintain a stable stitching effect when the data distribution changes. This problem is further exacerbated by the diversity and complexity of fundus images taken with portable devices. Especially when labeled data is scarce or completely unavailable, the performance of supervised models will drop significantly and cannot meet the needs of complex application scenarios.

[0007] In summary, the traditional fundus image stitching method cannot effectively cope with the challenges of high cost of portable fundus color photo annotation, unstable data quality, and poor model generalization ability due to its over-reliance on manual annotation and supervised learning. Unsupervised learning can automatically mine feature relationships from unlabeled images, which not only avoids the dependence on labeled data, but also shows stronger adaptability and robustness in diverse scenarios, providing a new direction for the development of fundus image stitching technology. Summary of the invention

[0008] In view of the problems existing in the above-mentioned prior art, the present invention provides an unsupervised deep learning image stitching method for portable fundus color photography equipment, which can effectively solve the registration error problem caused by scale change and illumination difference in traditional stitching methods, and can efficiently generate high-quality fundus stitching images.

[0009] In order to achieve the above object, the present invention provides an unsupervised deep learning image stitching method for a portable fundus color photography device, comprising the following steps:

[0010] Step 1: Build an image registration network and use it to achieve accurate feature point detection and matching of images;

[0011] A1: Construct a multi-scale feature extraction module for the image registration network;

[0012] The FOCUS module based on the improved YOLO backbone network structure, the deformable convolution module DCM, the C3 module based on the improved YOLO backbone network structure, the deformable convolution module DCM, and the C3 module based on the improved YOLO backbone network structure are connected in sequence to form a multi-scale feature extraction module;

[0013] A2: Constructing the feature matching module of the image registration network:

[0014] The image registration network is constructed using the Concat operation, the improved regression network, the Tensor DLT module, and the spatial transformation layer. The improved regression network consists of three convolutional layers, a depth-separable convolutional layer, and a fully connected layer. The improved regression network uses SiLU as the activation function, and the depth-separable convolutional layer is used to downsample the feature map. The Tensor DLT module is used to process dense feature maps, which extracts key geometric information through input features to reduce the difficulty of regression.

[0015] A3: Connect the multi-scale feature extraction module and the feature matching module in sequence to form an image registration network, and at the same time, construct the objective function of the image registration network; wherein, the construction process of the objective function of the image registration network is as follows:

[0016] A3-1: Improve the loss function of traditional unsupervised learning and obtain the improved overlap region alignment loss L′ in formula (1) p ;

[0017]

[0018] In the formula, I a and I b Represent the input reference image and target image respectively; L 1 {·} indicates the calculation of mean absolute error loss; Φ i represents the distortion operation corresponding to the homography matrix obtained for feature maps of different scales, i∈{1,2,3}; Φ(·) represents the distortion of one image to align it with another image using the estimated homography matrix; · represents the pixel-to-pixel multiplication; λ 1 , 2 , 3 are the weight factors of the loss items of different layers respectively; M overlap is the calculated overlap area, M overlap =M a &Φ(M b ), where & represents an AND operation; M a and M b Respectively represent the reference image I a and the target image I b All-1 masks of the same size;

[0019] A3-2: Introduce the content loss function in formula (2) to fully consider the content of the image during the image stitching process;

[0020] L c =L 1 {Canny(I a ′),Canny(Φ i (I b′))}(2);

[0021] Where, i∈{1,2,3}; Canny(·) represents the Canny operator;

[0022] A3-3: The final loss L of the image registration network estimated by homography is obtained according to formula (3);

[0023] L=λ p L′ p +λ c L c (3);

[0024] In the formula, λ p and λ c They are the loss functions L′ p and L c The weight of

[0025] Step 2: Build a fusion network and use it to complete high-quality fundus image stitching;

[0026] B1: Construct a shallow feature extractor with weight sharing; construct a shallow feature extractor using extraction branch 1, extraction branch 2 and extraction branch 3 connected in sequence, wherein extraction branch 1 includes a normalization layer, a 1×1 convolution layer, a 3×3 convolution layer and a laser function, extraction branch 2 includes a normalization layer, a 1×1 separable convolution layer, a 3×3 separable convolution layer and a laser function, and extraction branch 3 includes a normalization layer, a 1×1 convolution layer and a laser function;

[0027] B2: Construct a feature reconstruction module with a UNet structure; construct a feature reconstruction module with a UNet structure using four heavy parameter feature perception modules with convolution downsampling, four heavy parameter feature perception modules with convolution upsampling, and one self-attention module with self-attention feature perception;

[0028] B3: Constructing the seam generator; using two upsampling operations, two re-parameterized convolution operations and a Sigmoid activation function to construct the seam generator;

[0029] B4: The shallow feature extractor with weight sharing, the feature reconstruction module with the UNet structure and the seam generator are sequentially connected to form a fusion network. At the same time, the objective function of the fusion network is constructed. The construction process of the objective function of the fusion network is as follows:

[0030] B4-1: According to formula (4), the joint quality L is obtained f ;

[0031] L f =L b +L d (4);

[0032] Where, L b Indicates the start and end constraints of the seam, that is, the splicing boundary constraints, L d represents the minimum energy seam constraint, i.e., the seam smoothness constraint;

[0033] B4-2: Using stitched images I f With the registered reference image I wr and the target image I wt The alignment degree is used to form the splicing boundary constraint, and the splicing constraint loss L is obtained by formula (5) b , splicing constraint loss L b By minimizing the stitching result and the reference image I wr and the target image I wt The boundary differences between them are also given higher weights in the edge areas;

[0034] L b =||(I f -I wr )·M br || 1 +||(I f -I wt )·M bt || 1 (5);

[0035] In the formula, ||·|| 1 represents the L1 norm; ||I f -I wr || 1 and||I f -I wt || 1 Respectively represent the splicing results I f With reference image I wr and the target image I wt The sum of the absolute values ​​of the pixel-by-pixel differences between bt and M br Respectively represent the boundary mask processing results of the target image and the reference image, combined with the constraint weight of the edge feature information; M bt =M wt ·τ(M wr );M br =M wr ·τ(M wt );M wr and M wt I wr and I wt The corresponding original mask image; τ(·) represents the edge feature extraction operation based on the Sobel operator;

[0036] B4-3: Calculate the reference image I using the L2 norm using formula (6) wr and the target image I wt The global pixel difference L D , used to constrain the boundary position of the stitching area so that the stitching boundary falls in the similar area of ​​the two images as much as possible;

[0037] L D =||I wr -I wt || 2 (6);

[0038] B4-4: Compare the reference image I using the L2 norm using formula (7) wr and the target image I wt The edge feature difference L E ;

[0039] L E =||τ(I wr )-τ(I wt )|| 2 (7);

[0040] B4-5: Use formula (8) to convert the global pixel difference L D and edge feature difference L E Combined together to form a comprehensive loss function L S ;

[0041] L S =L D +α·L E (8); L

[0042] Where α is a hyperparameter used to adjust the weight between pixel difference and edge difference;

[0043] B4-6: Combined pixel point stitching matching degree L S , according to formula (9), the loss L used to constrain the smoothness and accuracy of the splicing boundary is calculated o ;

[0044]

[0045] In the formula, S r is the seam mask, indicating the location of the stitching boundary; and These two items calculate S r The magnitude of the change in the horizontal and vertical directions is used to detect the uneven parts on the seam mask boundary;

[0046] B4-7: Combine the final stitched image I FThe pixel value is calculated according to formula (10) to constrain the continuity of the splicing boundary. f ;

[0047]

[0048] In the formula, is the final generated stitched image I F The sum of these pixel values ​​is used as a weight to balance the overall impact of discontinuities at the boundaries on the stitched image.

[0049] B4-8: According to formula (11), L o and L f Combined, we get the comprehensive loss L d ;

[0050] L d =L o +L f (11);

[0051] Step 3: Build an image stitching model;

[0052] C1: connect the image registration network and the image fusion network to form an image stitching model;

[0053] C2: Data collection;

[0054] Two types of retinal fundus images are collected, and the two types of retinal fundus images are reference images and target images;

[0055] C3: Data preprocessing;

[0056] Performing data preprocessing on the reference image and the target image respectively to obtain a reference image Input1 and a target image Input2, and generating a reference image mask Mask1 and a target image mask Mask2 corresponding to the reference image Input1 and the target image Input2;

[0057] Among them, the reference image mask Mask1 and the target image mask Mask2 have the same size;

[0058] C4: training image stitching model;

[0059] C4-1: The reference image Input1, the target image Input2, the reference image mask Mask1, and the target image mask Mask2 are stacked together as the input data of the image registration network;

[0060] C4-2: Use the multi-scale feature extraction module to extract features from input data;

[0061] C4-2-1: Extract features from the input data through the FOCUS module based on the improved YOLO backbone network structure to generate a feature map with a size of 1 / 2 of the original image. and And the feature map and Input into the deformable convolution module DCM;

[0062] C4-2-2: Use the deformable convolution module DCM to process the feature map, obtain the overall output y(q) of the deformable convolution module DCM, and then input the overall output y(q) into the C3 module based on the improved YOLO backbone network structure;

[0063] C4-2-3: The C3 module based on the improved YOLO backbone network structure is used to efficiently fuse the features in different channels through residual connections to effectively integrate the local and global information of the input feature map and enhance the feature expression ability of the network; after processing by the C3 module, a smaller-scale feature map is generated and And input to the next deformable convolution module DCM;

[0064] C4-2-4: Use the next deformable convolution module DCM to perform feature mapping and After processing, the processing result is input into the next C3 module based on the improved YOLO backbone network structure, and a smaller-scale feature map is obtained after processing by the next C3 module based on the improved YOLO backbone network structure. and

[0065] C4-2-5: Using feature maps of two input images obtained at different scales, i.e. Construct a feature pyramid, and then input the constructed feature pyramid into the feature matching module;

[0066] C4-3: Use the feature matching module to obtain the matching relationship between the two feature maps;

[0067] C4-3-1: Use a simple Concat operation to merge the feature map pairs obtained from the feature pyramid in the channel dimension, and obtain the output after the Concat operation according to formula (12)

[0068]

[0069] In the formula, is the input feature map of the reference image, is the input feature map of the target image,

[0070] C4-3-2: Calculation using improved regression network According to formula (13), the regression network output coordinate offset Δn is obtained;

[0071]

[0072] In the formula, (u 1 ,v 1 )、(u 2 ,v 2 )、(u 3 ,v 3 )、(u 4 ,v 4 ) are the coordinates of the four corner points of the image, (Δu 1 ,Δv 1 )、(Δu 2 ,Δv 2 )、(Δu 3 ,Δv 3 )、(Δu 4 ,Δv 4 ) are the offsets of the four corner points of the image respectively; n∈{1,2,3}, where n is 1;

[0073] C4-3-3: Use the Tensor DLT module to perform a direct linear transformation on the input coordinate offset Δ1 to convert it into geometric parameters for homography estimation, namely the homography matrix H 1 ;

[0074] C4-3-4: The feature map of the next layer is transformed through the spatial transformation layer To distort; first input Generate a uniform grid, then use the homography matrix H 1 The grid is transformed and finally, pixel values ​​are extracted using bilinear interpolation based on these new positions and a transformed feature map of the same size as the input feature map is generated Then input it to the lower layer;

[0075] C4-3-5: Put the layer again and Perform a Concat operation on the channel dimension, calculate the coordinate offset Δ2 through the regression network, combine it with the coordinate offset Δ1 obtained in the previous layer, and obtain the homography matrix H of this layer through the Tensor DLT module. 2 ;

[0076] C4-3-6: Feature map of the next layer through the spatial transformation layer Distort to get the next layer Repeat the process in C4-3-5 again, and after calculating the coordinate offset Δ3, combine it with the coordinate offsets Δ1 and Δ2 of the previous two layers, and then use the Tensor DLT module to obtain the final homography matrix H 3 ;

[0077] C4-3-7: Use the final homography matrix H 3 The original target image Input2 and the target image mask Mask2 are distorted to obtain the reference image I wr and the target image I wt The resulting pre-stitched image pair is the output of the registration network;

[0078] C4-4: Repeat C4-1 to C4-3 for multiple times to train the image registration network. During the training process, calculate the final loss L of the image registration network and back-propagate to update the parameters of each layer. Record the current loss and compare it with the minimum loss. When the minimum loss is updated, save the image registration network in the training process first, and then conduct the next round of training. If the minimum loss is not updated, do not save the image registration network in the training process, and directly conduct the next round of training. At the same time, calculate the difference between the loss function of the current round and the previous round, and compare the difference with the convergence threshold to determine whether it converges. When converged, save the image registration network in the training process first, and then conduct the next round of training. When not converged, do not save the image registration network in the training process, and directly conduct the next round of training. After the training is completed, a trained image registration network is obtained;

[0079] C4-5: Use the output of the registration network as the input data of the image fusion network;

[0080] C4-6: Use two shallow feature extractors to downsample and extract features from the pre-stitched image pair, capture multi-scale information, extract local details and global features, and retain the edges, textures, and regional differences of the input image to maintain the difference information at the feature level of the input image. Finally, two feature difference maps are obtained respectively;

[0081] C4-7: Use the feature reconstruction module to process the two feature difference maps; specifically, multiple convolution branches in the re-parameterized feature perception module jointly learn features of different scales to increase the richness of feature expression; the self-attention module captures the dependency and interaction information between long-distance pixels in the feature map, and strengthens the ability to extract global difference information in the feature map, so as to more accurately locate the image seam area; finally, two feature mask maps are obtained respectively;

[0082] C4-8: Use the seam generator to extract and optimize the seam information of the two feature mask images; specifically, the resolution of the feature map is gradually enlarged through upsampling operations and restored to a size close to the original input size, so as to enlarge the low-resolution feature mask map to a high-resolution map, thereby generating a more detailed seam mask; after each upsampling operation, the upsampled features are further extracted and processed through a re-parameterized convolution operation to make the seam information more accurate; the eigenvalues ​​are compressed to the range of (0,1) using the Sigmoid function to generate a soft-coded seam mask; after obtaining the seam mask, the generated seam mask is multiplied with the aligned input mask to filter out the invalid area information and retain the mask value of the valid seam area; finally, two seam masks are obtained respectively;

[0083] C4-9: Perform weighted fusion on the two pre-stitched images according to the final two seam masks to generate the final stitched image;

[0084] C4-10: Repeat C4-5 to C4-9 to train the image fusion network. During the training process, calculate the comprehensive loss L of the image fusion network. d And back propagate to update the parameters of each layer, record the current loss, and compare it with the minimum loss. When the minimum loss is updated, the image fusion network in the training process is first saved, and then the next round of training is carried out. If the minimum loss is not updated, the image fusion network in the training process is not saved, and the next round of training is carried out directly. At the same time, the difference between the loss function of the current round and the previous round is calculated, and the difference is compared with the convergence threshold to determine whether it converges. When it converges, the image fusion network in the training process is first saved, and then the next round of training is carried out. When it does not converge, the image fusion network in the training process is not saved, and the next round of training is carried out directly. After the training is completed, the trained image fusion network is obtained;

[0085] C4-11: Use the trained image registration network and the trained image fusion network to form a fundus image stitching model;

[0086] Step 4: Use the fundus image stitching model to stitch the images;

[0087] D1: Collect two types of retinal fundus images, which are reference images and target images respectively;

[0088] D2: data preprocessing;

[0089] Perform data preprocessing on the reference image and the target image respectively;

[0090] D3: The preprocessed reference image and target image are input into the fundus image stitching model. The fundus image stitching model performs a stitching process on the reference image and the target image and outputs a stitched image.

[0091] As a preferred embodiment, in step A2 of step 1, the SiLU activation function is as shown in formula (14);

[0092]

[0093] As a preferred embodiment, in step 3 A3-1, the loss function of traditional unsupervised learning is as shown in formula (15);

[0094] L p =L 1 {I a ,Φ(I b )} (15).

[0095] As a preferred embodiment, in step A3-3 of step 1, λ p and λ c The average value is 1.

[0096] As a preference, in step 3 C3, the data preprocessing process is as follows:

[0097] C3-1: Crop redundant background information. For retinal fundus images, an edge detection-based algorithm is used to accurately locate the retinal area and crop the redundant background to retain only the effective area of ​​the retina.

[0098] C3-2: Normalization operation: normalize the image pixel values ​​and map the pixel values ​​to the range of [0,1];

[0099] C3-3: Histogram equalization; Adaptive histogram equalization technology is used to adaptively adjust the histogram according to the local illumination conditions of the image, significantly enhancing the contrast of the retinal area

[0100] C3-4: Noise suppression: denoise the image using one or both of the denoising methods: Gaussian filtering and bilateral filtering;

[0101] C3-5: Scale adjustment: Use bilinear interpolation to scale the image to ensure consistent resolution while retaining image details;

[0102] C3-6: Feature enhancement: Feature enhancement method based on high-frequency filtering is used to perform feature enhancement processing on images;

[0103] C3-7: Generate image mask; process the image and generate an image mask.

[0104] As a preferred embodiment, in step C-3 and C-4 of step 3, the homography matrix H of each layer is 1 , H 2 , H 3 Each layer is calculated independently, and at the same time, the next layer uses the geometric transformation results of the previous layer as input and gradually corrects the errors to reduce the cumulative effect of misalignment.

[0105] As a preferred embodiment, in step 3 C4-2-2, the overall output y(q) of the deformable convolution module DCM is obtained as follows:

[0106] C4-2-2-1: Obtain the location y(q)′ of the traditional convolution operation at position q according to formula (16);

[0107]

[0108] In the formula, x is the original input feature map; w is the convolution kernel; q is the output position of the convolution operation; q n is the relative position in the convolution kernel w;

[0109] C4-2-2-2: For the deformable convolution operation with a resolution of k×k, the feature map is first subjected to a traditional convolution operation to obtain a k×k×3 mask map, and then the mask map is divided into an offset vector and a weight mask, as shown in formula (17);

[0110] Conv-op(x)=ΔQ+ΔM (17);

[0111] Where Conv-op(x) is the traditional convolution operation; ΔQ is the bias vector; ΔM is the weight mask;

[0112] C4-2-2-3: For each position in the input feature map, use the bias vector ΔQ to bias it to obtain a new position. At the same time, the weight mask ΔM is constrained to a value between [0,1] through the Sigmoid function. Specifically, the overall output y(q) of the deformable convolution module DCM is obtained according to formula (18);

[0113]

[0114] The present invention proposes a fundus image stitching method based on unsupervised learning. The method is divided into two key stages: an image registration stage and an image fusion stage. In the image registration stage, the present invention designs a multi-scale feature extraction and matching module and proposes an image registration network based on unsupervised learning. For the image registration network, it uses unsupervised learning to achieve accurate feature point detection and matching for any fundus image pair, extracts multi-scale features by using a feature pyramid network, and combines the self-attention mechanism to achieve alignment and matching of key features, thereby solving the registration error problem caused by scale changes and illumination differences in traditional methods. The feature pyramid module can capture global and local information from high-level to low-level feature maps by introducing a multi-scale feature extraction mechanism. This hierarchical feature representation can effectively cope with scale changes and non-rigid deformations in images, and improves the generalization ability of registration. In feature extraction, a deformable convolution module (DCM) is further added to enable the network to adaptively perceive geometric deformations in the input image, thereby enhancing the flexibility and accuracy of feature representation. The introduction of DCM makes up for the insufficiency of traditional convolution in modeling complex deformations, and provides a higher degree of freedom for the construction of feature pyramids. The design of the homography estimation module further optimizes the geometric transformation process of image registration. The module uses an improved regression network to estimate the transformation relationship between two input images from coarse to fine by predicting the homography matrix layer by layer. This hierarchical estimation strategy not only reduces the sensitivity of high-level homography estimation to detail changes, but also can refine the registration results through low-level features, thereby taking into account the global consistency and local accuracy of the registration. Therefore, the image registration network proposed in the present invention combines the feature pyramid and the homography estimation module, and realizes efficient and robust image registration through multi-scale feature extraction and precise geometric transformation modeling. In the image fusion stage, the present invention designs an image fusion network. For the image fusion network, the shallow features of the input image are first extracted, and their feature differences are processed and retained respectively to perceive the deep feature information. Then, the deep features of the distorted images of the same size generated in the registration stage are mined by the feature reconstruction strategy, the feature manifold of the pre-stitched image is shaped and improved, and the redundant information is reduced, thereby generating the best seam mask. Then, the final spliced ​​image can be formed by weighted fusion, which effectively eliminates the overlapping area artifacts, thereby efficiently generating the best seam and obtaining a high-quality fundus spliced ​​image. The shallow feature extractor introduces a re-parameterized convolution operation in feature acquisition, and captures the local details and global information of the image through a multi-branch convolution structure. This re-parameterized multi-branch convolution design not only enhances the model's perception of features of different scales, but also significantly improves the expressiveness of feature differences. By obtaining the feature difference map, the extractor can effectively focus on the regional feature differences in the two registered images, thereby laying a solid foundation for subsequent feature reconstruction. The feature reconstruction module further improves the fusion accuracy by converting the feature difference map into a feature mask map.The combination of the heavy parameter feature perception and self-attention mechanism in the module can not only adaptively adjust the feature representation and highlight the key areas, but also capture long-distance dependencies in the global context, thereby achieving fine reconstruction of image details. This design effectively addresses the problem of feature fuzziness caused by redundant data and local noise, ensuring the clarity and consistency of the fused image. The seam generator generates high-quality seam masks by predicting the mask output of the soft code using the sigmoid function. Compared with the traditional hard-coded mask, the soft mask has higher smoothness and flexibility, and can achieve a natural transition in the fusion boundary area, thereby eliminating visual faults and artificial traces. In addition, this mask generation method based on probability prediction has strong robustness and can adapt to the difference of the registered images in a variety of complex scenes. The image fusion network proposed in the present invention effectively solves the artifact problem of the spliced ​​image through the organic combination of the shallow feature extractor, the feature reconstruction module and the seam generator, the introduction of multi-branch feature extraction and reconstruction strategy, and the efficiency and robustness of the soft mask generation mechanism, achieving high-quality and seamless image fusion.

[0115] Compared with the prior art, the present invention has the following advantages:

[0116] 1) Traditional image registration methods usually rely on single-scale feature extraction or fixed geometric transformation modeling, which makes it difficult to deal with multi-scale features and non-rigid deformation problems in complex scenes. The present invention introduces a deformable convolution module in the feature pyramid, which significantly enhances the robustness and accuracy of feature extraction by flexibly sensing and modeling local deformations in the image. At the same time, the transformation relationship is predicted from coarse to fine through hierarchical homography estimation, fully capturing the changes in global consistency and local detail information. This design shows excellent adaptability and registration accuracy in complex environments such as high noise and small field of view, making up for the limitations of existing technologies in multi-scale and deformable feature processing.

[0117] 2) Unlike the traditional method of directly using the original image after registration for fusion, the present invention efficiently extracts feature differences through re-parameterized multi-branch convolution, and converts the difference map into a feature mask map in combination with the feature reconstruction module. The fusion of re-parameterized feature perception and self-attention mechanism in the reconstruction module not only highlights the key area features, but also captures long-distance dependencies in the global context, thereby achieving accurate removal of redundant information and detail optimization. In addition, this feature reconstruction-based strategy avoids the rigid processing of seams by traditional hard-coding methods, ensuring the naturalness and consistency of the fused image, especially in detail-sensitive application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0119] Figure 2 It is a flow chart of data preprocessing in the present invention;

[0120] Figure 3 It is the overall architecture diagram of the image registration network in the present invention;

[0121] Figure 4 It is the architecture diagram of the variable convolution module in the present invention;

[0122] Figure 5 It is the overall architecture diagram of the image fusion network in the present invention;

[0123] Figure 6 It is an architecture diagram of the shallow feature extraction module of the present invention. DETAILED DESCRIPTION

[0124] The purpose of the present invention is to overcome the problems of traditional fundus image stitching methods being overly dependent on manual annotation and poor generalization ability of supervised learning models, and to provide a fundus image stitching model based on unsupervised learning to achieve efficient and robust image registration and high-quality seamless image fusion.

[0125] like Figure 1 As shown, the process of the present invention is as follows: first, the retinal fundus images taken by the portable device are collected and preprocessed, so that the images are adapted to the network input on the one hand, and the image noise should be reduced as much as possible on the other hand, so that the overall image brightness is balanced, which is convenient for subsequent stitching operations; then, the image data is passed through an unsupervised fundus image registration network to perform accurate feature point detection and matching on the image pairs, so as to solve the registration error problem caused by scale changes and illumination differences in traditional methods; finally, the pre-stitched images after registration are passed through a fundus image fusion network, and the deep features of the pre-stitched images are mined using a feature reconstruction strategy to form an optimal seam and generate a high-quality fundus stitching image.

[0126] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings.

[0127] The present invention provides an unsupervised deep learning image stitching method for a portable fundus color photography device, comprising the following steps:

[0128] Step 1: Build an image registration network and use it to achieve accurate feature point detection and matching of images;

[0129] like Figure 3 As shown in Figure 1, the image registration network consists of a multi-scale feature extraction module and a feature matching module. Multi-scale features are extracted using a feature pyramid network, and the alignment and matching of key features are achieved by combining the self-attention mechanism.

[0130] A1: Construct a multi-scale feature extraction module for the image registration network;

[0131] The FOCUS module based on the improved YOLO backbone network structure, the deformable convolution module DCM, the C3 module based on the improved YOLO backbone network structure, the deformable convolution module DCM, and the C3 module based on the improved YOLO backbone network structure are connected in sequence to form a multi-scale feature extraction module;

[0132] A2: Constructing the feature matching module of the image registration network:

[0133] Generally speaking, in the registration stage, most methods calculate the correlation matrix between two feature maps to obtain the matching relationship between them. However, these methods require convolution and a large number of multiplication operations to calculate the correlation, which is very time-consuming and requires a large amount of computer memory. Therefore, the present invention designs a feature matching module based on an improved regression network.

[0134] Specifically, the image registration network is constructed using the Concat operation, the improved regression network, the Tensor DLT module and the spatial transformation layer; the improved regression network consists of three convolutional layers, a depthwise separable convolutional layer and a fully connected layer. In order to make the model have better gradient propagation characteristics, the improved regression network uses SiLU as the activation function; the depthwise separable convolutional layer is used to downsample the feature map; the Tensor DLT module is used to process the dense feature map, which extracts key geometric information through input features to reduce the difficulty of regression;

[0135] A3: Connect the multi-scale feature extraction module and the feature matching module in sequence to form an image registration network, and at the same time, construct the objective function of the image registration network; wherein, the construction process of the objective function of the image registration network is as follows:

[0136] A3-1: Based on the loss function of traditional unsupervised learning, in order to make the distorted target image close to the reference image, it is necessary to compare the reference image and the distorted target image as a whole. However, for a pair of images to be spliced, they will not completely contain each other. Therefore, performing alignment operations in the entire image does not make much sense for the final result and will prolong the running time. In order to solve this problem, this application improves the loss function of traditional unsupervised learning and obtains the improved overlap area alignment loss L′ in formula (1): p ;

[0137]

[0138] In the formula, I a and I b Represent the input reference image and target image respectively; L 1 {·} indicates the calculation of mean absolute error loss; Φi represents the distortion operation corresponding to the homography matrix obtained for feature maps of different scales, i∈{1,2,3}; Φ(·) represents the distortion of one image to align it with another image using the estimated homography matrix; · represents the pixel-to-pixel multiplication; λ 1 , 2 , 3 are the weight factors of the loss items of different layers respectively; M overlap is the calculated overlap area, M overlap =M a &Φ(M b ), where & represents an AND operation; M a and M b Respectively represent the reference image I a and the target image I b All-1 masks of the same size;

[0139] A3-2: In order to consider the image content in image stitching, the content loss function in formula (2) is introduced to fully consider the content in the image during the image stitching process;

[0140] L c =L 1 {Canny(I a ′),Canny(Φ i (I b ′))}(2);

[0141] Where, i∈{1,2,3}; Canny(·) represents the Canny operator. Since this loss function assigns higher weights to image edges, it tends to prioritize edge features during the optimization process, thereby effectively improving the quality and accuracy of the stitching results.

[0142] A3-3: The weighted sum of the above loss function is calculated, and the final loss L of the image registration network estimated by homography is obtained according to formula (3); Based on the proposed loss function, the present invention only uses the difference between pixels in the overlapping area for alignment calculation to improve work efficiency. In addition, in order to better capture subtle structural differences, it is necessary to combine edge content for alignment to ensure that higher quality alignment results are obtained in key edge and structure areas.

[0143] L=λ p L′ p +λ c L c (3);

[0144] In the formula, λ p and λ c They are the loss functions L′ p and L c The weight of

[0145] Step 2: Build a fusion network and use it to complete high-quality fundus image stitching operations, such as Figure 5 As shown;

[0146] B1: Constructing a shallow feature extractor with weight sharing; Considering that fundus images have a lot of redundancy after alignment, it is challenging to accurately perceive seams. To solve this problem, the present invention develops a feature extractor that uses re-parameterized convolution for downsampling and feature extraction.

[0147] Specifically, if Figure 6 As shown, a shallow feature extractor is constructed by sequentially connecting extraction branch one, extraction branch two and extraction branch three, wherein extraction branch one includes a normalization layer, a 1×1 convolution layer, a 3×3 convolution layer and a laser function, extraction branch two includes a normalization layer, a 1×1 separable convolution layer, a 3×3 separable convolution layer and a laser function, and extraction branch three includes a normalization layer, a 1×1 convolution layer and a laser function;

[0148] Furthermore, during the training phase, the shallow feature extractor contains three branches (including 1×1 convolution, 3×3 convolution and identity mapping) to independently process input features, which can extract rich semantic information with different receptive fields and feature scales. During the inference phase, these branches are reparameterized and integrated into an equivalent convolution operation, which can effectively simplify the calculation and thus improve the inference efficiency.

[0149] These branches process the input features through different convolution kernel sizes (1×1, 3×3) and operations (standard convolution, depthwise convolution) to capture multi-scale information, extract local details and global features, while retaining the edges, textures, and regional differences of the input image, and then distinguish the feature maps to maintain the difference information at the feature level of the input image.

[0150] B2: Construct a feature reconstruction module with a UNet structure; after obtaining the feature difference map, the seam information of the image must be extracted from it. To this end, the present invention designs a feature reconstruction block using the UNet structure to convert the feature difference map into a feature mask map.

[0151] Specifically, the present invention is based on the FastViT network optimization design, deletes the normalization operations of the first and third layers and the residual fusion operation of the second layer, reduces the computational complexity by removing unnecessary operations, and optimizes the network's sensitivity to input difference features, thereby improving the efficiency of extracting seam information.

[0152] Specifically, a feature reconstruction module with a UNet structure is constructed using four heavy parameter feature perception modules with convolution downsampling, four heavy parameter feature perception modules with convolution upsampling, and one self-attention module with self-attention feature perception;

[0153] Furthermore, the heavy parameter feature perception module adopts multiple convolution branches (convolution kernels of different sizes, standard convolution and depth separable convolution, etc.) in the training stage to jointly learn features of different scales, increase the richness of feature expression, and make it easier for the model to capture global and local information. In the reasoning stage, the convolution kernel and bias parameters of the multi-branch structure are merged into a single convolution kernel and bias through mathematical transformation. This conversion does not change the output features, but greatly simplifies the computational complexity of the reasoning stage. Furthermore, the self-attention module is used to capture the dependencies and interaction information between distant pixels in the feature map, and strengthen the ability to extract global difference information in the feature map, so as to more accurately locate the image seam area. In addition, the present invention applies feature summation instead of the typical UNET Concat operation to combine features in the same layer, thereby simplifying the computational complexity, avoiding redundant information that may be caused by the splicing operation, making feature fusion more efficient, and retaining key features.

[0154] B3: Construct a seam generator; the seam generator aims to generate the final seam mask from the feature mask map for seam information extraction and optimization in image stitching. Its structure consists of two upsampling operations, two reparameterized convolution operations and a Sigmoid activation function, and finally generates a soft-coded seam mask.

[0155] The seam generator is constructed using two upsampling operations, two reparameterized convolution operations, and a Sigmoid activation function. The upsampling operation gradually enlarges the resolution of the feature map through deconvolution (transposed convolution) and restores it to a size close to the original input size, so that the low-resolution feature mask map can be enlarged to a high-resolution map to generate a more detailed seam mask, which helps to capture more detailed seam information in the image. After each upsampling, the upsampled features are further extracted and processed through a reparameterized convolution operation to make the seam information more accurate. The reparameterized convolution has a multi-branch structure in the training phase to enhance the expressiveness of the features, but it is simplified to a single-branch structure in the inference phase, greatly reducing the computational overhead.

[0156] B4: The shallow feature extractor with weight sharing, the feature reconstruction module with UNet structure and the seam generator are sequentially connected to form a fusion network. At the same time, in order to study the optimal seam, the objective function of the fusion network is constructed. The construction process of the objective function of the fusion network is as follows:

[0157] B4-1: According to formula (4), the joint quality L is obtainedf ;

[0158] L f =L b +L d (4);

[0159] Where, L b Indicates the start and end constraints of the seam, that is, the splicing boundary constraints, L d represents the minimum energy seam constraint, i.e., the seam smoothness constraint;

[0160] B4-2: In terms of stitching boundary constraints, using stitching image I f With the registered reference image I wr and the target image I wt The goal is to make the overlapping area boundary with I r Similar pixels are close to I r , instead of I t This requires that the start and end points of the seam be located exactly at the intersection of the overlapping areas.

[0161] The splicing constraint loss L is obtained by formula (5): b , splicing constraint loss L b By minimizing the stitching result and the reference image I wr and the target image I wt The boundary differences between them are also given higher weights in the edge areas;

[0162] L b =||(I f -I wr )·M br || 1 +||(I f -I wt )·M bt || 1 (5);

[0163] In the formula, ||·|| 1 Represents L1 norm (also called Manhattan distance, absolute value norm); ||I f -I wr || 1 and||I f -I wt || 1 Respectively represent the splicing results I f With reference image I wr and the target image I wt The sum of the absolute values ​​of the pixel-by-pixel differences between bt and M brRespectively represent the boundary mask processing results of the target image and the reference image, combined with the constraint weight of the edge feature information; M bt =M wt ·τ(M wr );M br =M wr ·τ(M wt );M wr and M wt I wr and I wt The corresponding original mask image; τ(·) represents the edge feature extraction operation based on the Sobel operator;

[0164] B4-3: In the constraint term of the smooth seam, the smoothness of the product of the generated mask boundary and the pixel difference map and the smoothness of the product of the generated mask boundary and the pixel difference map are used. wr and I wt The corresponding edge detail difference map is used as a constraint. The optimal seam boundary should ideally be located at I wr and I wt This means that the reference image I corresponding to the perimeter of the seam boundary wr and the target image I wt Minimize the difference between them.

[0165] Specifically, the reference image I is calculated by using the L2 norm using formula (6): wr and the target image I wt The global pixel difference L D , used to constrain the boundary position of the stitching area so that the stitching boundary falls in the similar area of ​​the two images as much as possible;

[0166] L D =||I wr -I wt || 2 (6);

[0167] B4-4: Compare the reference image I using the L2 norm using formula (7) wr and the target image I wt The edge feature difference L E , constraining the smoothness of the stitching boundary so that the edge features of the stitching area remain consistent.

[0168] L E =||τ(I wr )-τ(I wt )|| 2 (7);

[0169] B4-5: Use formula (8) to convert the global pixel difference L D and edge feature difference L ECombined together to form a comprehensive loss function L S ; This comprehensive loss function is used to optimize the splicing area so that the boundary position is as accurate and smooth as possible.

[0170] L S =L D +α·L E (8);

[0171] Where α is a hyperparameter used to adjust the weight between pixel difference and edge difference;

[0172] B4-6: Combined pixel point stitching matching degree L S , according to formula (9), the loss L used to constrain the smoothness and accuracy of the splicing boundary is calculated o , to constrain the smoothness and accuracy of the stitching boundary;

[0173]

[0174] In the formula, S r is the seam mask, indicating the location of the stitching boundary; and These two items calculate S r The change amplitude in the horizontal direction (column index j) and vertical direction (row index i) is used to detect the non-smooth part on the seam mask boundary. The larger the difference, the discontinuous the boundary. S Introducing it into the formula ensures that the changes in the stitching boundary are not only smooth but also consistent with the pixel and edge features of the input image.

[0175] B4-7: Further combine the final stitching image I in the stitching boundary smoothness constraint F The pixel value is calculated according to formula (10) to constrain the continuity of the splicing boundary. f ;

[0176]

[0177] In the formula, is the final generated stitched image I F The sum of these pixel values ​​is used as a weight to balance the overall impact of discontinuities at the boundaries on the stitched image.

[0178] B4-8: According to formula (11), L o and L f Combined, we get the comprehensive loss L d ; L o Based on the splicing matching degree L S , focusing on the alignment of the stitching boundary with the input image features. fDirectly for the final stitched image I F , optimizes the smoothness of the stitched image boundary and is used to quantify the overall consistency of the stitched image.

[0179] L d =L o +L f (11);

[0180] Step 3: Build an image stitching model;

[0181] C1: connect the image registration network and the image fusion network to form an image stitching model;

[0182] C2: Data collection;

[0183] Two types of retinal fundus images are collected, and the two types of retinal fundus images are reference images and target images;

[0184] Specifically, data collection can be performed using clinical medical equipment or database query, and retinal fundus images can be collected from data sources such as patients and healthy subjects. The present invention preferably uses a fundus image database taken by a public portable fundus camera, which covers retinal images collected by a variety of different portable devices, and has the characteristics of high noise and small field of view. The original data is standardized and organized to meet the needs of fundus image stitching tasks. As a preferred embodiment, the data is provided in formats such as JPEG, PNG, TIF, etc., and is classified according to the eyes to which it belongs. The file structure is uniformly organized to facilitate the automated execution of the preprocessing process.

[0185] C3: Data preprocessing;

[0186] After collecting the data, the image needs to be standardized by resizing, brightness and contrast adjustment, color correction, etc.; the reference image and the target image are preprocessed to obtain the reference image Input1 and the target image Input2, and the reference image mask Mask1 and the target image mask Mask2 corresponding to the reference image Input1 and the target image Input2 are generated;

[0187] Among them, the reference image mask Mask1 and the target image mask Mask2 have the same size;

[0188] C4: training image stitching model;

[0189] C4-1: The reference image Input1, the target image Input2, the reference image mask Mask1, and the target image mask Mask2 are stacked together as the input data of the image registration network;

[0190] C4-2: Use the multi-scale feature extraction module to extract features from input data;

[0191] C4-2-1: Extract features from the input data through the FOCUS module based on the improved YOLO backbone network structure to generate a feature map with a size of 1 / 2 of the original image. and And the feature map and Input into the deformable convolution module DCM;

[0192] C4-2-2: In practical applications, the input image may produce some distortion due to various reasons, such as changes in camera posture, lighting conditions, etc. To solve this problem, the present invention innovatively adds DCM to the multi-scale feature extraction module of the registration network to expand the perception domain of the model and improve the network's adaptability to deformation.

[0193] The feature map is processed by the deformable convolution module DCM to obtain the overall output y(q) of the deformable convolution module DCM, and then the overall output y(q) is input into the C3 module based on the improved YOLO backbone network structure;

[0194] C4-2-3: The C3 module based on the improved YOLO backbone network structure is used to efficiently fuse features in different channels through residual connections, thereby enhancing the network's feature expression capabilities. This module uses pruning operations to optimize the efficiency of the feature extraction module by reducing redundant calculations, while reducing the model's computing resource requirements. This design allows the network to maintain accuracy while reducing operating costs. At the same time, this module can effectively integrate the local and global information of the input feature map, giving the network a stronger expression capability when processing multi-scale features. After processing by the C3 module, a smaller-scale feature map is generated and And input to the next deformable convolution module DCM;

[0195] C4-2-4: Use the next deformable convolution module DCM to perform feature mapping and After processing, the processing result is input into the next C3 module based on the improved YOLO backbone network structure, and a smaller-scale feature map is obtained after processing by the next C3 module based on the improved YOLO backbone network structure. and A repeated network layer assignment strategy is used here to provide a more detailed input decomposition and stabilize the propagation of gradients while maintaining feature continuity.

[0196] C4-2-5: Using feature maps of two input images obtained at different scales, i.e. Construct a feature pyramid, and then input the constructed feature pyramid into the feature matching module;

[0197] C4-3: Use the feature matching module to obtain the matching relationship between two feature maps, such as Figure 3 As shown;

[0198] C4-3-1: Use a simple Concat operation to merge the feature map pairs obtained from the feature pyramid in the channel dimension, and obtain the output after the Concat operation according to formula (12)

[0199]

[0200] In the formula, is the input feature map of the reference image, is the input feature map of the target image,

[0201] C4-3-2: Calculation using improved regression network Correlation. The regression network proposed in this application consists of three convolutional layers (Conv), a depth-wise separable convolutional layer (DW-Conv) and a fully connected layer (FC). Among them, DW-Conv is used to downsample the feature map instead of the average pooling layer. The main reason for using DW-Conv is that it can retain more position information by translating the convolution kernel during downsampling and learn the relationship between different feature map channels. The regression network output coordinate offset Δn is obtained according to formula (13); these coordinate offsets Δn uniquely determine the corresponding homography matrix.

[0202]

[0203] In the formula, (u 1 ,v 1 )、(u 2 ,v 2 )、(u 3 ,v 3 )、(u 4 ,v 4 ) are the coordinates of the four corner points of the image, (Δu 1 ,Δv 1 )、(Δu 2 ,Δv 2 )、(Δu 3 ,Δv 3 )、(Δu 4 ,Δv 4 ) are the offsets of the four corner points of the image respectively; n∈{1,2,3}, where n is 1;

[0204] C4-3-3: Use the Tensor DLT module (direct linear transformation) to perform a direct linear transformation on the input coordinate offset Δ1 to convert it into geometric parameters for homography estimation, namely the homography matrix H 1 ; Traditional DLT methods usually directly perform geometric fitting on point pairs, while Tensor DLT is an improved version of tensorization that can process dense feature maps and is suitable for end-to-end training of deep learning networks. Compared with direct regression of homography matrices, Tensor DLT uses input features to extract key geometric information, which reduces the difficulty of regression and improves the convergence speed and estimation accuracy of the network.

[0205] C4-3-4: The feature map of the next layer is transformed through the spatial transformation layer (STL) To distort; first input Generate a uniform grid, then use the homography matrix H 1 The grid is transformed and finally, pixel values ​​are extracted using bilinear interpolation based on these new positions and a transformed feature map of the same size as the input feature map is generated Then input it to the lower layer;

[0206] C4-3-5: Put the layer again and Perform a Concat operation on the channel dimension, calculate the coordinate offset Δ2 through the regression network, combine it with the coordinate offset Δ1 obtained in the previous layer, and obtain the homography matrix H of this layer through the Tensor DLT module. 2 ;

[0207] C4-3-6: Feature map of the next layer through the spatial transformation layer Distort to get the next layer Repeat the process in C4-3-5 again, and after calculating the coordinate offset Δ3, combine it with the coordinate offsets Δ1 and Δ2 of the previous two layers, and then use the Tensor DLT module to obtain the final homography matrix H 3 ;

[0208] C4-3-7: Use the final homography matrix H 3 The original target image Input2 and the target image mask Mask2 are distorted to obtain the reference image I wr and the target image I wt The resulting pre-stitched image pair is the output of the registration network;

[0209] C4-4: Repeat C4-1 to C4-3 for multiple times to train the image registration network. During the training process, calculate the final loss L of the image registration network and back-propagate to update the parameters of each layer. Record the current loss and compare it with the minimum loss. When the minimum loss is updated, save the image registration network in the training process first, and then conduct the next round of training. If the minimum loss is not updated, do not save the image registration network in the training process, and directly conduct the next round of training. At the same time, calculate the difference between the loss function of the current round and the previous round, and compare the difference with the convergence threshold to determine whether it converges. When converged, save the image registration network in the training process first, and then conduct the next round of training. When not converged, do not save the image registration network in the training process, and directly conduct the next round of training. After the training is completed, a trained image registration network is obtained;

[0210] C4-5: Use the output of the registration network as the input data of the image fusion network;

[0211] C4-6: Use two shallow feature extractors to downsample and extract features from the pre-stitched image pair (reference image Input1 and registered target image Input2), capture multi-scale information, extract local details and global features, and retain the edges, textures, and regional differences of the input images to maintain the difference information at the feature level of the input images. Finally, two feature difference maps are obtained respectively.

[0212] C4-7: Use the feature reconstruction module to process the two feature difference maps; specifically, multiple convolution branches in the re-parameterized feature perception module jointly learn features of different scales to increase the richness of feature expression; the self-attention module captures the dependency and interaction information between long-distance pixels in the feature map, and strengthens the ability to extract global difference information in the feature map, so as to more accurately locate the image seam area; finally, two feature mask maps are obtained respectively;

[0213] C4-8: Use the seam generator to extract and optimize the seam information of the two feature mask images; specifically, the resolution of the feature map is gradually enlarged through upsampling operations and restored to a size close to the original input size, so as to enlarge the low-resolution feature mask map to a high-resolution map, thereby generating a more detailed seam mask; after each upsampling operation, the upsampled features are further extracted and processed through re-parameterized convolution operations to make the seam information more accurate; the eigenvalues ​​are compressed to the range of (0,1) using the Sigmoid function to generate a soft-coded seam mask; the soft-coded mask can capture regional uncertainty and help to blend the seam area more naturally.

[0214] After obtaining the seam mask, the generated seam mask is multiplied with the aligned input mask to filter out the invalid area information and retain the mask value of the valid seam area; in this way, the seam mask can be more accurate and focused, and the stitching effect is optimized. Finally, two seam masks are obtained respectively;

[0215] C4-9: Perform weighted fusion on the two pre-stitched images according to the final two seam masks to generate the final stitched image;

[0216] C4-10: Repeat C4-5 to C4-9 to train the image fusion network. During the training process, calculate the comprehensive loss L of the image fusion network. d And back propagate to update the parameters of each layer, record the current loss, and compare it with the minimum loss. When the minimum loss is updated, the image fusion network in the training process is first saved, and then the next round of training is carried out. If the minimum loss is not updated, the image fusion network in the training process is not saved, and the next round of training is carried out directly. At the same time, the difference between the loss function of the current round and the previous round is calculated, and the difference is compared with the convergence threshold to determine whether it converges. When it converges, the image fusion network in the training process is first saved, and then the next round of training is carried out. When it does not converge, the image fusion network in the training process is not saved, and the next round of training is carried out directly. After the training is completed, the trained image fusion network is obtained;

[0217] C4-11: Use the trained image registration network and the trained image fusion network to form a fundus image stitching model;

[0218] Step 4: Use the fundus image stitching model to stitch the images;

[0219] D1: Collect two types of retinal fundus images, which are reference images and target images respectively;

[0220] D2: Data preprocessing: perform data preprocessing on the reference image and the target image respectively;

[0221] D3: The preprocessed reference image and target image are input into the fundus image stitching model. The fundus image stitching model performs a stitching process on the reference image and the target image and outputs a stitched image.

[0222] As a preferred embodiment, in step A2 of step 1, the SiLU activation function is as shown in formula (14);

[0223]

[0224] As a preferred embodiment, in step 3 A3-1, the loss function of traditional unsupervised learning is as shown in formula (15);

[0225] L p =L 1 {I a ,Φ(I b )} (15).

[0226] As a preferred embodiment, in step A3-3, in order to balance pixel alignment and structure alignment during training, λ p and λ c The average value is 1.

[0227] In the portable fundus image stitching task, data preprocessing is a key step to ensure model performance and experimental reliability. Figure 2 As shown, according to the characteristics of the task, the present invention specially designs a set of special image preprocessing processes to adapt to the characteristics of small field of view, high noise and rich details of portable fundus images, so as to ensure the quality of the stitching results. Specifically, in step 3 C3, the data preprocessing process is as follows:

[0228] C3-1: Crop redundant background information; In fundus image processing, in order to improve data quality and model training effects, it is first necessary to crop redundant background information in the image. There are usually a large number of irrelevant black backgrounds in fundus images. These areas not only do not contribute to the training of the model, but may also introduce noise, affecting the learning effect of the effective area. Therefore, for retinal fundus images, an edge detection-based algorithm is used to accurately locate the retinal area and crop the redundant background parts, retaining only the effective area of ​​the retina; this operation ensures that the model focuses on the key areas of the retina, thereby improving the efficiency and performance of training.

[0229] C3-2: Normalization operation; normalize the image pixel values ​​to ensure the consistency and stability of the input data; since the pixel value distribution of different images may be different, mapping the pixel values ​​to the range of [0,1] can effectively avoid the impact of inconsistent numerical scales on the training process, making the model smoother during feature learning and the training process easier to converge.

[0230] C3-3: Histogram equalization: To address the common problem of uneven illumination in fundus images, in order to improve the contrast of the image and enhance the visibility of details, adaptive histogram equalization (CLAHE) technology is used. CLAHE can adaptively adjust the histogram according to the local illumination conditions of the image, significantly enhance the contrast of the retinal area, and make the blood vessels, macula and other fine structures clearer. This step is crucial for the observation of detail areas and subsequent feature extraction.

[0231] C3-4: Noise suppression; To further improve image quality, noise suppression is also a necessary step. During the image acquisition process, equipment performance and external interference often introduce high-frequency noise, affecting the clarity of the image and the extraction of effective information. Through denoising methods such as Gaussian filtering and bilateral filtering, the image noise can be effectively suppressed while retaining the key texture information of the retina. For example, bilateral filtering can well retain high-frequency structural information such as image edges and blood vessels while removing noise. As a preferred embodiment, the present application uses one or both of the denoising methods of Gaussian filtering and bilateral filtering to denoise the image;

[0232] C3-5: Scale adjustment: All images need to be adjusted to a uniform resolution to meet the size requirements of the model input and reduce computational complexity. Images are scaled using bilinear interpolation to ensure consistent resolution while retaining image details and to minimize possible information loss during the interpolation process.

[0233] C3-6: Feature enhancement: After completing the basic processing, further feature enhancement can be performed to strengthen the key structures in the retinal image, such as the vascular network. Feature enhancement based on high-frequency filtering can effectively highlight high-frequency information, make the key features of the retina more obvious, and provide more effective information for the subsequent feature extraction module, thereby improving the performance of the model in detection and recognition tasks.

[0234] C3-7: Generate image mask; process the image and generate an image mask. In order to further improve the model's attention to the feature area, an image mask can be generated. In complex visual tasks, irrelevant areas in the image may interfere with the model's learning process, resulting in training convergence difficulties or unstable results. By inputting a mask, the model's attention range can be effectively limited, and computing resources can be concentrated on the parts of interest such as the retinal area. This strategy helps reduce interference from irrelevant areas and improve the robustness and accuracy of the model.

[0235] As a preferred embodiment, in step C-3 and C-4 of step 3, the homography matrix H of each layer is 1 , H 2 , H 3 Each layer is calculated independently, and the next layer uses the geometric transformation result of the previous layer as input and gradually corrects the error to reduce the cumulative effect of misalignment. At the same time, since the feature resolution is lower in the higher-level feature pyramid, it can capture global geometric changes; while the lower-level feature pyramid has a higher resolution and can capture finer local features. This strategy of estimating the homography matrix layer by layer can gradually optimize local deformation on the basis of global alignment and improve the accuracy of registration.

[0236] As a preferred embodiment, Figure 4 As shown, in step 3 C4-2-2, the acquisition process (workflow) of the overall output y(q) of the deformable convolution module DCM is as follows:

[0237] C4-2-2-1: Obtain the location y(q)′ of the traditional convolution operation at position q according to formula (16);

[0238]

[0239] In the formula, x is the original input feature map; w is the convolution kernel; q is the output position of the convolution operation; q n is the relative position in the convolution kernel w;

[0240] C4-2-2-2: For the deformable convolution operation with a resolution of k×k, the feature map is first subjected to a traditional convolution operation to obtain a k×k×3 mask map, and then the mask map is divided into an offset vector and a weight mask, as shown in formula (17);

[0241] Conv-op(x)=ΔQ+ΔM (17);

[0242] Where Conv-op(x) is the traditional convolution operation; ΔQ is the bias vector; ΔM is the weight mask;

[0243] C4-2-2-3: For each position in the input feature map, use the bias vector ΔQ to bias it to obtain a new position. At the same time, the weight mask ΔM is constrained to a value between [0,1] through the Sigmoid function. The Sigmoid function assigns different weight values ​​to the convolution points at each position, thereby providing greater freedom for sampling. Specifically, the overall output y(q) of the deformable convolution module DCM is obtained according to formula (18);

[0244]

[0245] As a further preference, the specific implementation process of the method can be carried out using hardware platforms such as computers, servers, and mobile devices.

[0246] The image fusion network proposed in the present invention effectively solves the artifact problem of spliced ​​images through the organic combination of shallow feature extractor, feature reconstruction module and seam generator. The introduction of multi-branch feature extraction and reconstruction strategy, as well as the efficiency and robustness of the soft mask generation mechanism, realizes high-quality and seamless image fusion.

Claims

1. An unsupervised deep learning image stitching method for portable fundus color photography equipment, characterized in that: The following steps are involved: Step 1: Build an image registration network and use it to achieve accurate feature point detection and matching of images; A1: Construct a multi-scale feature extraction module for the image registration network; The FOCUS module based on the improved YOLO backbone network structure, the deformable convolution module DCM, the C3 module based on the improved YOLO backbone network structure, the deformable convolution module DCM, and the C3 module based on the improved YOLO backbone network structure are connected in sequence to form a multi-scale feature extraction module; A2: Constructing the feature matching module of the image registration network: The image registration network is constructed using the Concat operation, the improved regression network, the Tensor DLT module and the spatial transformation layer. The improved regression network consists of three convolutional layers, a depth-separable convolutional layer and a fully connected layer. The improved regression network uses SiLU as the activation function, and the depth-separable convolutional layer is used to downsample the feature map. The TensorDLT module is used to process dense feature maps, which extracts key geometric information through input features to reduce the difficulty of regression. A3: Connect the multi-scale feature extraction module and the feature matching module in sequence to form an image registration network, and at the same time, construct the objective function of the image registration network; wherein, the construction process of the objective function of the image registration network is as follows: A3-1: Improve the loss function of traditional unsupervised learning to obtain the improved overlap region alignment loss L′ in formula (1) p ; In the formula, I a and I b denote the input reference image and target image respectively; L1{·} denotes the mean absolute error loss; Φ i represents the distortion operation corresponding to the homography matrix obtained for feature maps of different scales, i∈{1,2,3}; Φ(·) represents the distortion of an image using the estimated homography matrix to align it with another image; · represents the pixel multiplication; λ1, λ2, λ3 are the weight factors of the loss terms at different layers respectively; M overlap is the calculated overlap area, M overlap =M a &Φ(M b ), where & represents an AND operation; M a and M b Respectively represent the reference image I a and the target image I b All-1 masks of the same size; A3-2: Introduce the content loss function in formula (2) to fully consider the content of the image during the image stitching process; L c =L1{Canny(I a ′),Canny(Φ i (I b ′))}(2); Where, i∈{1,2,3}; Canny(·) represents the Canny operator; A3-3: The final loss L of the image registration network estimated by homography is obtained according to formula (3); L=λ p L′ p +λ c L c (3); In the formula, λ p and λ c They are the loss functions L′ p and L c The weight of Step 2: Build a fusion network and use it to complete high-quality fundus image stitching; B1: Construct a shallow feature extractor with weight sharing; construct a shallow feature extractor using extraction branch 1, extraction branch 2 and extraction branch 3 connected in sequence, wherein extraction branch 1 includes a normalization layer, a 1×1 convolution layer, a 3×3 convolution layer and a laser function, extraction branch 2 includes a normalization layer, a 1×1 separable convolution layer, a 3×3 separable convolution layer and a laser function, and extraction branch 3 includes a normalization layer, a 1×1 convolution layer and a laser function; B2: Construct a feature reconstruction module with a UNet structure; construct a feature reconstruction module with a UNet structure using four heavy parameter feature perception modules with convolution downsampling, four heavy parameter feature perception modules with convolution upsampling, and one self-attention module with self-attention feature perception; B3: Constructing the seam generator; using two upsampling operations, two re-parameterized convolution operations and a Sigmoid activation function to construct the seam generator; B4: The shallow feature extractor with weight sharing, the feature reconstruction module with the UNet structure and the seam generator are sequentially connected to form a fusion network. At the same time, the objective function of the fusion network is constructed. The construction process of the objective function of the fusion network is as follows: B4-1: According to formula (4), the joint quality L is obtained f ; L f =L b +L d (4); Where, L b Indicates the start and end constraints of the seam, that is, the splicing boundary constraints, L d represents the minimum energy seam constraint, i.e., the seam smoothness constraint; B4-2: Using stitched images I f With the registered reference image I wr and the target image I wt The alignment degree is used to form the splicing boundary constraint, and the splicing constraint loss L is obtained by formula (5) b , splicing constraint loss L b By minimizing the stitching result and the reference image I wr and the target image I wt The boundary differences between them are analyzed, while giving higher weights to the edge areas; L b =||(I f -I wr )·M br ||1+||(I f -I wt )·M bt ‖1 (5); In the formula, ||·||1 represents the L1 norm; ||I f -I wr ||1 and ||I f -I wt ||1 represents the splicing result I f With reference image I wr and the target image I wt The sum of the absolute values ​​of the pixel-by-pixel differences between bt and M br Respectively represent the boundary mask processing results of the target image and the reference image, combined with the constraint weight of the edge feature information; M bt =M wt ·τ(M wr );M br =M wr ·τ(M wt );M wr and M wt I wr and I wt The corresponding original mask image; τ(·) represents the edge feature extraction operation based on the Sobel operator; B4-3: Calculate the reference image I using the L2 norm using formula (6) wr and the target image I wt The global pixel difference L D , used to constrain the boundary position of the stitching area so that the stitching boundary falls in the similar area of ​​the two images as much as possible; L D =||I wr -I wt ||2 (6); B4-4: Compare the reference image I using the L2 norm using formula (7) wr and the target image I wt The edge feature difference L E ; L E =||τ(I wr )-τ(I wt )||2 (7); B4-5: Use formula (8) to convert the global pixel difference L D and edge feature difference L E Combined together to form a comprehensive loss function L S ; L S =L D +α·L E (8); Where α is a hyperparameter used to adjust the weight between pixel difference and edge difference; B4-6: Combined pixel point stitching matching degree L S , according to formula (9), the loss L used to constrain the smoothness and accuracy of the splicing boundary is calculated o ; In the formula, S r is the seam mask, indicating the location of the stitching boundary; and These two items calculate S r The magnitude of the change in the horizontal and vertical directions is used to detect the uneven parts on the seam mask boundary; B4-7: Combine the final stitched image I F The pixel value is calculated according to formula (10) to constrain the continuity of the splicing boundary. f ; In the formula, is the final generated stitched image I F The sum of these pixel values ​​is used as a weight to balance the overall impact of discontinuities at the boundaries on the stitched image. B4-8: According to formula (11), L o and L f Combined, we get the comprehensive loss L d ; L d =L o +L f (11); Step 3: Build an image stitching model; C1: connect the image registration network and the image fusion network to form an image stitching model; C2: Data collection; Two types of retinal fundus images are collected, and the two types of retinal fundus images are reference images and target images; C3: Data preprocessing; Performing data preprocessing on the reference image and the target image respectively to obtain a reference image Input1 and a target image Input2, and generating a reference image mask Mask1 and a target image mask Mask2 corresponding to the reference image Input1 and the target image Input2; Among them, the reference image mask Mask1 and the target image mask Mask2 have the same size; C4: training image stitching model; C4-1: The reference image Input1, the target image Input2, the reference image mask Mask1, and the target image mask Mask2 are stacked together as the input data of the image registration network; C4-2: Use the multi-scale feature extraction module to extract features from input data; C4-2-1: Extract features from the input data through the FOCUS module based on the improved YOLO backbone network structure to generate a feature map with a size of 1 / 2 of the original image. and And the feature map and Input into the deformable convolution module DCM; C4-2-2: Use the deformable convolution module DCM to process the feature map, obtain the overall output y(q) of the deformable convolution module DCM, and then input the overall output y(q) into the C3 module based on the improved YOLO backbone network structure; C4-2-3: The C3 module based on the improved YOLO backbone network structure is used to efficiently fuse the features in different channels through residual connections to effectively integrate the local and global information of the input feature map and enhance the feature expression ability of the network; after processing by the C3 module, a smaller-scale feature map is generated and And input to the next deformable convolution module DCM; C4-2-4: Use the next deformable convolution module DCM to perform feature mapping and After processing, the processing result is input into the next C3 module based on the improved YOLO backbone network structure, and a smaller-scale feature map is obtained after processing by the next C3 module based on the improved YOLO backbone network structure. and C4-2-5: Using feature maps of two input images obtained at different scales, i.e. Construct a feature pyramid, and then input the constructed feature pyramid into the feature matching module; C4-3: Use the feature matching module to obtain the matching relationship between the two feature maps; C4-3-1: Use a simple Concat operation to merge the feature map pairs obtained from the feature pyramid in the channel dimension, and obtain the output after the Concat operation according to formula (12) In the formula, is the input feature map of the reference image, is the input feature map of the target image, C4-3-2: Calculation using improved regression network According to formula (13), the output coordinate offset Δn of the regression network is obtained; Where, (u1, v1), (u2, v2), (u3, v3), (u4, v4) are the coordinates of the four corner points of the image, (Δu1, Δv1), (Δu2, Δv2), (Δu3, Δv3), (Δu4, Δv4) are the offsets of the four corner points of the image; n∈{1,2,3}, where n is 1; C4-3-3: Use the Tensor DLT module to perform a direct linear transformation on the input coordinate offset Δ1 to convert it into geometric parameters for homography estimation, namely, the homography matrix H1; C4-3-4: The feature map of the next layer is transformed through the spatial transformation layer To distort; first input Generate a uniform grid, then transform the grid using the homography matrix H1, and finally extract pixel values ​​using bilinear interpolation based on these new positions and generate a transformed feature map of the same size as the input feature map Then input it to the lower layer; C4-3-5: Put the layer again and Perform a Concat operation on the channel dimension, calculate the coordinate offset Δ2 through the regression network, combine it with the coordinate offset Δ1 obtained in the previous layer, and obtain the homography matrix H2 of this layer through the Tensor DLT module; C4-3-6: Feature map of the next layer through the spatial transformation layer Distort to get the next layer Repeat the process in C4-3-5 again, and after calculating the coordinate offset Δ3, combine it with the coordinate offsets Δ1 and Δ2 of the previous two layers, and then use the Tensor DLT module to obtain the final homography matrix H3; C4-3-7: Use the final homography matrix H3 to distort the original target image Input2 and the target image mask Mask2 to obtain the reference image I wr and the target image I wt The resulting pre-stitched image pair is the output of the registration network; C4-4: Repeat C4-1 to C4-3 for multiple times to train the image registration network. During the training process, calculate the final loss L of the image registration network and back-propagate to update the parameters of each layer. Record the current loss and compare it with the minimum loss. When the minimum loss is updated, save the image registration network in the training process first, and then conduct the next round of training. If the minimum loss is not updated, do not save the image registration network in the training process, and directly conduct the next round of training. At the same time, calculate the difference between the loss function of the current round and the previous round, and compare the difference with the convergence threshold to determine whether it converges. When converged, save the image registration network in the training process first, and then conduct the next round of training. When not converged, do not save the image registration network in the training process, and directly conduct the next round of training. After the training is completed, a trained image registration network is obtained; C4-5: Use the output of the registration network as the input data of the image fusion network; C4-6: Use two shallow feature extractors to downsample and extract features from the pre-stitched image pair, capture multi-scale information, extract local details and global features, and retain the edges, textures, and regional differences of the input image to maintain the difference information at the feature level of the input image. Finally, two feature difference maps are obtained respectively; C4-7: Use the feature reconstruction module to process the two feature difference maps; specifically, multiple convolution branches in the re-parameterized feature perception module jointly learn features of different scales to increase the richness of feature expression; the self-attention module captures the dependency and interaction information between long-distance pixels in the feature map, and strengthens the ability to extract global difference information in the feature map, so as to more accurately locate the image seam area; finally, two feature mask maps are obtained respectively; C4-8: Use the seam generator to extract and optimize the seam information of the two feature mask images; specifically, the resolution of the feature map is gradually enlarged through upsampling operations and restored to a size close to the original input size, so as to enlarge the low-resolution feature mask map to a high-resolution map, thereby generating a more detailed seam mask; after each upsampling operation, the upsampled features are further extracted and processed through a re-parameterized convolution operation to make the seam information more accurate; the eigenvalues ​​are compressed to the range of (0,1) using the Sigmoid function to generate a soft-coded seam mask; after obtaining the seam mask, the generated seam mask is multiplied with the aligned input mask to filter out the invalid area information and retain the mask value of the valid seam area; finally, two seam masks are obtained respectively; C4-9: Perform weighted fusion on the two pre-stitched images according to the final two seam masks to generate the final stitched image; C4-10: Repeat C4-5 to C4-9 to train the image fusion network. During the training process, calculate the comprehensive loss L of the image fusion network. d And back propagate to update the parameters of each layer, record the current loss, and compare it with the minimum loss. When the minimum loss is updated, the image fusion network in the training process is first saved, and then the next round of training is carried out. If the minimum loss is not updated, the image fusion network in the training process is not saved, and the next round of training is carried out directly. At the same time, the difference between the loss function of the current round and the previous round is calculated, and the difference is compared with the convergence threshold to determine whether it converges. When it converges, the image fusion network in the training process is first saved, and then the next round of training is carried out. When it does not converge, the image fusion network in the training process is not saved, and the next round of training is carried out directly. After the training is completed, the trained image fusion network is obtained; C4-11: Use the trained image registration network and the trained image fusion network to form a fundus image stitching model; Step 4: Use the fundus image stitching model to stitch the images; D1: Collect two types of retinal fundus images, which are reference images and target images respectively; D2: data preprocessing; Perform data preprocessing on the reference image and the target image respectively; D3: The preprocessed reference image and target image are input into the fundus image stitching model. The fundus image stitching model performs a stitching process on the reference image and the target image and outputs a stitched image.

2. The unsupervised deep learning image stitching method for portable fundus color photography equipment according to claim 1, characterized in that: In step 1 A2, the SiLU activation function is shown in formula (14); 3. The unsupervised deep learning image stitching method for portable fundus color photography equipment according to claim 1, characterized in that: In step 1 A3-1, the loss function of traditional unsupervised learning is shown in formula (15); L p =L1{I a ,Φ(I b )} (15).

4. The unsupervised deep learning image stitching method for portable fundus color photography equipment according to claim 1, characterized in that: In step 1, A3-3, λ p and λ c The average value is 1.

5. The unsupervised deep learning image stitching method for portable fundus color photography equipment according to claim 1 or 2, characterized in that: In step 3 C3, the data preprocessing process is as follows: C3-1: Crop redundant background information. For retinal fundus images, an edge detection-based algorithm is used to accurately locate the retinal area and crop the redundant background to retain only the effective area of ​​the retina. C3-2: Normalization operation: normalize the image pixel values ​​and map the pixel values ​​to the range of [0,1]; C3-3: Histogram equalization; Adaptive histogram equalization technology is used to adaptively adjust the histogram according to the local illumination conditions of the image, significantly enhancing the contrast of the retinal area C3-4: Noise suppression: denoise the image using one or both of the denoising methods: Gaussian filtering and bilateral filtering; C3-5: Scale adjustment: Use bilinear interpolation to scale the image to ensure consistent resolution while retaining image details; C3-6: Feature enhancement: Feature enhancement method based on high-frequency filtering is used to perform feature enhancement processing on images; C3-7: Generate image mask; process the image and generate an image mask.

6. The unsupervised deep learning image stitching method for portable fundus color photography equipment according to claim 1, characterized in that: In C-3 and C-4 of step three, the homography matrices H1, H2, and H3 of each layer are calculated independently. At the same time, the next layer uses the geometric transformation results of the previous layer as input and gradually corrects the errors to reduce the cumulative effect of misregistration.

7. The unsupervised deep learning image stitching method for portable fundus color photography equipment according to claim 1, characterized in that: In step 3 C4-2-2, the overall output y(q) of the deformable convolution module DCM is obtained as follows: C4-2-2-1: Obtain the location y(q)′ of the traditional convolution operation at position q according to formula (16); In the formula, x is the original input feature map; w is the convolution kernel; q is the output position of the convolution operation; q n is the relative position in the convolution kernel w; C4-2-2-2: For the deformable convolution operation with a resolution of k×k, the feature map is first subjected to a traditional convolution operation to obtain a k×k×3 mask map, and then the mask map is divided into an offset vector and a weight mask, as shown in formula (17); Conv-op(x)=ΔQ+ΔM (17); Where Conv-op(x) is the traditional convolution operation; ΔQ is the bias vector; ΔM is the weight mask; C4-2-2-3: For each position in the input feature map, use the bias vector ΔQ to bias it to obtain a new position. At the same time, the weight mask ΔM is constrained to a value between [0,1] through the Sigmoid function. Specifically, the overall output y(q) of the deformable convolution module DCM is obtained according to formula (18);

Images