Rock ore microscopic image splicing method and system based on deep learning

Abstract

The invention discloses a rock and ore microscopic image splicing method and system based on deep learning, and relates to the field of image processing and the technical field of microscopes, and the method comprises the steps: obtaining a local rock and ore microscopic image of a rock and ore slice, carrying out the preprocessing of the local rock and ore microscopic image, and carrying out the overlapping region coarse registration of the preprocessed local rock and ore microscopic image through a phase correlation method; based on an improved image feature detection model, basic features and description features in the local rock and ore microscopic image after coarse registration are extracted, and local image features of the local rock and ore microscopic image are obtained; performing feature matching on the local image features of the two groups of local rock and mineral microscopic images by using an image feature matching model to obtain a matching corresponding relation of the local image features; and based on an image fusion algorithm of a homography matrix and a partial differential equation, splicing and optimizing the local rock and ore microscopic images in combination with a matching corresponding relation of local image features, and generating a large-view-field rock and ore microscopic image. According to the method, the complex transformation between the images can be better processed.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and the field of microscopy, in particular to a rock and mineral microscopic image stitching method and system based on deep learning. BACKGROUND

[0002] Rock and mineral microscopic image stitching is an important technology in geological science research. In the fields of geological exploration and mineral classification, it is necessary to observe and analyze rock thin sections under a high-power microscope. However, since rock samples are usually composed of different minerals with large differences in hardness, the surface of the rock thin section is extremely uneven, making it difficult to achieve full-field high-resolution clear observation under a high-power microscope. Therefore, it is necessary to stitch multiple local microscopic images into a full-field image in order to conduct more comprehensive analysis and research.

[0003] Traditional image stitching techniques mainly rely on feature point extraction and matching algorithms such as SIFT, SURF, etc. However, these algorithms have certain limitations in rock and mineral microscopic image stitching. First, due to the rich content of rock and mineral microscopic images and the presence of a large number of similar and easily confused areas, there are many false matching points, resulting in inaccurate stitching. Second, the edge image content of some thin sections is sparse, and traditional algorithms cannot complete the registration of full-field images. In addition, traditional algorithms also have certain difficulties in handling complex scenes such as large parallax and low texture.

[0004] In recent years, with the rapid development of deep learning technology, deep learning-based image stitching technology has received widespread attention and research, and has achieved good performance in various complex scenarios. Deep learning technology can automatically extract and match feature points in images by learning the feature relationships in a large amount of data, thereby achieving more accurate image stitching. In addition, deep learning technology can also handle problems that traditional algorithms cannot solve, such as large parallax stitching and low texture scene alignment.

[0005] Therefore, a deep learning-based microscopic image stitching technology is needed to further improve the speed and accuracy of stitching.

[0006] However, there is currently no effective solution to the problems in the related art. SUMMARY

[0007] To overcome the above technical problems existing in the prior art, the present application proposes a rock and mineral microscopic image stitching method and system based on deep learning.

[0008] To this end, the specific technical solutions adopted by the present application are as follows:

[0009] In a first aspect, the present application provides a rock and mineral microscopic image stitching method based on deep learning, which comprises the following steps:

[0010] Obtain a local rock micro-image of a rock thin section and pre-process the local rock micro-image, and use a phase correlation method to perform coarse registration on the local rock micro-image after pre-processing;

[0011] Based on the improved image feature detection model, extract the basic features and description features in the local rock micro-image after coarse registration, and obtain the local image features of the local rock micro-image;

[0012] Use an image feature matching model to perform feature matching on the local image features of the two groups of local rock micro-images, and obtain the matching corresponding relationship of the local image features;

[0013] Based on the homography matrix and the image fusion algorithm of partial differential equation, combine the matching corresponding relationship of the local image features to splice and optimize the local rock micro-image, and generate a large field rock micro-image.

[0014] Further, obtaining a local rock micro-image of a rock thin section and pre-processing the local rock micro-image, and using a phase correlation method to perform coarse registration on the local rock micro-image after pre-processing includes:

[0015] Use an optical microscope to perform microscopic imaging on the rock thin section, and scan the collected continuous rock micro-image sequence row by row;

[0016] Perform grayscale, denoising and contrast enhancement processing on the collected continuous rock micro-image sequence, and use a phase correlation method to perform coarse registration on the overlapping area to obtain the coarse registration rock micro-image.

[0017] Further, using a phase correlation method to perform coarse registration on the overlapping area to obtain the coarse registration rock micro-image includes:

[0018] Based on the Fourier transform formula, perform translation phase difference processing on the same two rock micro-images after coarse registration.

[0019] Further, based on the improved image feature detection model, extract the basic features and description features in the local rock micro-image after coarse registration to obtain the local image features of the local rock micro-image includes:

[0020] Based on the lightweight convolutional neural network architecture and decoupled fully connected attention, improve the encoder in the image feature detection model to obtain the improved image feature detection model, wherein the image feature detection model is composed of an encoder and two independent decoders;

[0021] Input the local rock micro-image after coarse registration into the improved image feature detection model, process it through the encoder and the decoder, extract the basic features and the description features in the local rock micro-image, and obtain the local image features of the local rock micro-image.

[0022] Further, the image feature matching model is used to perform feature matching on the local image features of the two groups of local rock ore microscopic images, to obtain a matching correspondence of the local image features, including:

[0023] Context information inside the basic features and the description features is aggregated through a self-attention mechanism, and feature point similarity between different local rock ore microscopic images is calculated through a cross-attention mechanism, to construct a global matching score matrix;

[0024] The matching problem is converted into a probability constraint optimization based on an iterative algorithm, and the matching probability is dynamically adjusted to eliminate low-confidence point pairs; in combination with a key point mask algorithm, an end-to-end training is realized through a differentiable matching layer, to obtain a matching result under geometric consistency constraints;

[0025] Matching connection lines between the local microscopic images are constructed based on the basic features, and a random sample consistency algorithm is used to eliminate incorrect matching point pairs in the matching result under geometric consistency constraints, to obtain the matching correspondence of the local image features.

[0026] Further, based on an image fusion algorithm of homography matrix and partial differential equation, in combination with the matching correspondence of the local image features, the local rock ore microscopic images are spliced and optimized to generate a large-view rock ore microscopic image, including:

[0027] A homography matrix between the matched local rock ore microscopic image and the adjacent local rock ore microscopic image to be spliced is calculated, and the matched local rock ore microscopic image is subjected to perspective transformation according to the homography matrix, so that the local rock ore microscopic image is aligned with the local rock ore microscopic image to be spliced;

[0028] An image sharpness recognition model is used to perform sharpness recognition on the local rock ore microscopic image after perspective transformation, and an image enhancement model is constructed to perform quality enhancement on the blurred local rock ore microscopic image;

[0029] Four corner point coordinates of the matched local rock ore microscopic image after transformation are calculated, and are combined with four corner point coordinates of the local rock ore microscopic image to be spliced, to calculate boundary values of the spliced rock ore microscopic image;

[0030] According to the boundary values of the spliced rock ore microscopic image, a translation amount between the matched local rock ore microscopic image and the adjacent local rock ore microscopic image to be spliced is calculated, and the canvas size is adjusted;

[0031] And through a sharpness evaluation function combined with a three-stage focusing strategy, the sharpness of the rock ore microscopic image is quantified.

[0032] Further, the image sharpness recognition model is used to recognize the sharpness of the local rock ore microscopic image after perspective transformation, and an image enhancement model is constructed to enhance the quality of the fuzzy local rock ore microscopic image, including:

[0033] The local rock ore microscopic image is recognized based on a multi-scale convolutional neural network, a three-dimensional attention mechanism is added, an improved multi-scale convolutional neural network is obtained, and the channel attention and spatial attention in the three-dimensional attention mechanism are used to optimize the local rock ore microscopic image region;

[0034] The two-dimensional layer operation and convolution operation in the channel attention and spatial attention are replaced by three-dimensional layer operation and convolution operation;

[0035] The three-dimensional layer operation and convolution operation are embedded into the sampling layer of the improved multi-scale convolutional neural network respectively, the channel attention and spatial attention layer are used to optimize the local rock ore microscopic image region, and the local rock ore microscopic image features are extracted therefrom;

[0036] Based on the extracted local rock ore microscopic image features, the quality of the fuzzy rock ore microscopic image is enhanced by using the spatial sharpening filter method.

[0037] Further, the sharpness of the rock ore microscopic image is quantified by combining the three-stage focusing strategy with the sharpness evaluation function, including:

[0038] The Sobel operator is used to calculate the gradient amplitude square sum of the rock ore microscopic image to form a sharpness evaluation curve of the rock ore microscopic image;

[0039] According to the sharpness evaluation curve of the rock ore microscopic image, the clear area of the rock ore microscopic image is located, and the best focus point is determined by combining the three-stage focusing strategy.

[0040] Further, based on the image fusion algorithm of homography matrix and partial differential equation, the local rock ore microscopic image is spliced and optimized by combining the matching corresponding relationship of local image features to generate a large field rock ore microscopic image, further including:

[0041] The diffusion coefficient of the reaction image space position is introduced to smooth the boundary value of the rock ore microscopic image;

[0042] The image fusion algorithm is improved by using the partial differential equation to obtain an improved image fusion algorithm, and the two aligned rock ore microscopic images are spliced to generate a large field rock ore microscopic image.

[0043] In a second aspect, the present application also provides a rock ore microscopic image splicing system based on deep learning, which comprises:

[0044] An image processing module is configured to acquire and preprocess a local rock and mineral micrograph of a rock and mineral thin section, and perform coarse registration on the preprocessed local rock and mineral micrograph by using a phase correlation method.

[0045] A feature extraction module is configured to extract basic features and description features in the local rock and mineral micrograph after coarse registration based on an improved image feature detection model, to obtain local image features of the local rock and mineral micrograph.

[0046] A feature matching module is configured to perform feature matching on the local image features of the two groups of local rock and mineral micrographs by using an image feature matching model, to obtain a matching corresponding relationship of the local image features.

[0047] An image generation module is configured to perform splicing and optimization on the local rock and mineral micrograph based on a homography matrix and an image fusion algorithm of partial differential equations, in combination with the matching corresponding relationship of the local image features, to generate a large-view rock and mineral micrograph.

[0048] The present application has the following advantages:

[0049] 1) The phase correlation method is used for coarse registration, and the calculation amount is reduced: there may be a large number of similar textures in the micro rock and mineral image, and direct feature matching is prone to mis-matching; the translation amount is calculated by using Fourier transform and phase difference, and coarse registration of the image overlapping area is quickly realized; the interference of irrelevant feature points is reduced, the calculation complexity of subsequent feature extraction and matching is reduced, the overall efficiency is improved, and good conditions are provided for subsequent fine registration.

[0050] 2) Fine registration based on the improved deep learning algorithm: in the micro rock and mineral image, the image may have problems such as illumination change and noise interference, and the lightweight SuperPoint algorithm can more efficiently extract feature points of the image. Compared with traditional feature extraction algorithms (such as SIFT and SURF), the network structure is optimized, the extraction speed is improved while the accuracy is maintained; the SuperGlue feature matching algorithm based on deep learning can learn feature descriptors and matching score functions, and significantly improve the matching accuracy. In the micro rock and mineral image splicing, complex transformations between images can be better handled, and accurate feature point matching can be realized.

[0051] 3) High-quality image splicing and fusion: based on the feature point matching result, the geometric transformation relationship between the images is calculated to obtain the homography matrix, accurate image registration is realized, the spliced image is ensured to be geometrically aligned, and the misplacement and blur phenomenon is reduced; compared with the traditional fusion algorithm, the partial differential equation improved image fusion algorithm can better handle the splicing gap and large-area color difference problem, the pixel value of the fusion area is optimized, the diffusion coefficient is introduced to realize smooth transition, the splicing trace is reduced, and the visual quality of the spliced image is improved. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 is a flow chart of a rock and mineral microscopic image splicing method based on deep learning according to an embodiment of the present application;

[0054] Figure 2 is a structural block diagram of a rock and mineral microscopic image splicing system based on deep learning according to an embodiment of the present application;

[0055] Figure 3 is a microscope automatic focusing image sharpness evaluation curve schematic diagram according to an embodiment of the present application;

[0056] Figure 4 is a microscope automatic focusing image sharpness evaluation curve schematic diagram;

[0057] Figure 5 is a schematic diagram of a microscopic local image in the present application;

[0058] Figure 6 is a schematic diagram of a microscopic panoramic image in the present application. DETAILED DESCRIPTION

[0059] In order to further illustrate the embodiments, the present application provides drawings which are part of the disclosure of the present application, mainly used to illustrate the embodiments, and can explain the operating principle of the embodiments in conjunction with the related description of the specification. Those skilled in the art should understand other possible embodiments and advantages of the present application by referring to these contents.

[0060] According to an embodiment of the present application, a rock and mineral microscopic image splicing method and system based on deep learning are provided.

[0061] The present application will be further described in conjunction with the drawings and specific embodiments, as shown in the drawings, the rock and mineral microscopic image splicing method based on deep learning according to an embodiment of the present application comprises: Figure 1

[0062] Step S1, obtaining a local rock and mineral microscopic image of a rock and mineral thin section and pre-processing, using phase correlation method to perform rough registration on the overlapping area of the pre-processed local rock and mineral microscopic image;

[0063] ​Step S2, based on the improved image feature detection (i.e. SuperPoint) model, extract the basic features (i.e. feature points) and description features (i.e. descriptors) in the local rock micrograph after rough registration, to obtain the local image features of the local rock micrograph;

[0064] Step S3, using the image feature matching (i.e. SuperGlue) model, the local image features of the two groups of local rock micrographs are matched to obtain the matching corresponding relationship of the local image features;

[0065] Step S4, based on the homography matrix and the image fusion algorithm of partial differential equation, combined with the matching corresponding relationship of the local image features, the local rock micrograph is spliced and optimized to generate a large field rock micrograph.

[0066] In this optional embodiment, the local rock micrograph of the rock slice is obtained and preprocessed, and the phase correlation method is used for rough registration of the overlapping area of the preprocessed local rock micrograph, which includes:

[0067] Step S11, using an optical microscope to perform micro-imaging on the rock slice, and scanning the collected continuous rock micrograph sequence row by row;

[0068] Step S12, the collected continuous rock micrograph sequence is processed by gray scale, denoising and contrast enhancement, and the phase correlation method is used for rough registration of the overlapping area, to obtain the rough registration of the rock micrograph.

[0069] It should be noted that before feature point extraction and matching, the input rock micrograph is first preprocessed. The preprocessing steps include image graying, denoising, contrast enhancement, etc. to improve the accuracy of subsequent feature extraction, and then the phase correlation method is used to realize the rough registration of the image overlapping area.

[0070] An optical microscope is used to perform micro-imaging on the rock slice, and by two-dimensional plane motion of the stage, a continuous micrograph sequence is scanned and collected row by row, ensuring that there is a certain overlapping area between every two adjacent images. Before collecting each image, the system performs focusing method, because the uneven or inclined surface of the rock slice may cause partial area out of focus. The collected images are roughly registered by the phase correlation method to obtain images closer to the overlapping area.

[0071] The sample is scanned in the X and Y axes by the stage, and three-dimensional imaging is performed by combining the microscope lens focusing (Z axis), to ensure that the complete observation area is covered, and the parameters such as exposure time, light source intensity and objective magnification are fixed to avoid differences in image brightness and contrast caused by changes in environmental light or equipment jitter.

[0072] In the optional embodiment, the phase correlation method is used for coarse registration of the overlapping area, and the coarse-registered rock and mineral microscopic images include:

[0073] In step S121, the Fourier transform formula is used for phase difference processing of the two coarse-registered rock and mineral microscopic images.

[0074] It should be noted that if there is only a translation relationship between the two images I1(x,y) and I2(x,y), i.e., I2(x,y) = I1(x-x0,y-y0), the Fourier transform formula is:

[0075]

[0076] In the formula, F[I2](u,v) and F[I1](u,v) represent the Fourier transform of the images I1(x,y) and I2(x,y), respectively; (u,v) represents the translation amount between the two images; represents the phase difference introduced by translation.

[0077] In the optional embodiment, based on the improved image feature detection model, the basic features and description features in the coarse-registered local rock and mineral microscopic images are extracted, and the local image features of the local rock and mineral microscopic images include:

[0078] In step S21, the VGG encoder in the image feature detection model is improved based on the lightweight convolutional neural network architecture (i.e., GhostNetV2) and the decoupled fully connected attention (i.e., DFC attention module), and an improved image feature detection model is obtained, wherein the image feature detection model is composed of an encoder and two independent decoders.

[0079] In step S22, the coarse-registered local rock and mineral microscopic images are input into the improved image feature detection model, and the basic features and description features in the local rock and mineral microscopic images are extracted through the encoder and the decoder, and the local image features of the local rock and mineral microscopic images are obtained.

[0080] It should be noted that the SuperPoint model is composed of an encoder and two independent decoders, wherein the encoder serves as a shared component and serves the entire network. The two decoders are respectively focused on feature point detection and descriptor generation, and by sharing the same encoder, the computational complexity is significantly reduced and the operation efficiency is improved.

[0081] The encoder replaces the original network structure VGG-like encoder with GhostNetV2, which adopts the structure of inverted residual bottleneck containing two Ghost modules, wherein the first module generates more channel number expand features, and the second module reduces the channel number to generate output features; the DFC attention module is introduced, which reduces the calculation complexity by decomposing the traditional full connection attention into horizontal and vertical full connection layers.

[0082] The GhostNetV2 is used to replace the original SuperPoint network structure VGG-like encoder to realize lightweight network deployment; the ECA module is used to replace the traditional SE module after the output of each Ghost module, the ECA module avoids the dimension reduction and complex calculation of the full connection layer in the SE module by designing the adaptive convolution kernel size, while maintaining the lightweight, and increases the expression of features between channels; sufficient feature points are extracted from the collected microscopic images, and the NMS (Non Maximum Suppression) non-maximum suppression is used to eliminate redundant feature points.

[0083] Meanwhile, the movement trajectory of the microscope stage can be determined, and the stitching path of the rock image is consistent with the acquisition path, which constitutes the basic prior information of the microscope operation. Due to the fact that the optical microscope and the stage cannot be completely parallel, during the movement of the stage, the image captured by the microscope each time will have a deviation in the x and y axes. In addition, during the movement of the stage, due to mechanical factors such as motor gear operation, a certain error will be generated, resulting in a displacement between adjacent fields of view that is not constant. Therefore, in order to integrate the image into a large field of view image, an image stitching algorithm must be used for processing.

[0084] The descriptor network layer combines multi-layer convolution and full connection layer architecture to extract 256-dimensional descriptors from the surrounding area of feature points (representing key positions in the image) for describing the local features of these feature points. These descriptors can efficiently evaluate the similarity between different feature points.

[0085] In this optional embodiment, the image feature matching model is used to perform feature matching on the local image features of the two groups of local rock and mineral microscopic images to obtain the matching corresponding relationship of the local image features, including:

[0086] Step S31, the context information inside the basic features and the description features is aggregated through the self-attention mechanism, and the feature point similarity between different local rock and mineral microscopic images is calculated through the cross-attention mechanism to construct a global matching score matrix;

[0087] Step S32, convert the matching problem into a probability constraint optimization based on an iterative algorithm, and dynamically adjust the matching probability to eliminate low-confidence point pairs; combine the key point masking algorithm, realize end-to-end training through the differentiable matching layer, and obtain the matching result under the geometric consistency constraint;

[0088] Step S33, construct the matching connection between local microscopic images according to the basic features, and use the random sample consistency algorithm (i.e. RANSAC) to eliminate the wrong matching point pairs in the matching result under the geometric consistency constraint, and obtain the matching corresponding relationship of the local image features.

[0089] It should be noted that the SuperGlue network mainly includes two components: an attention graph neural network (GNN) and an optimal matching layer. The network receives the input information provided by SuperPoint, i.e. the feature points and their corresponding descriptors of the two images, and outputs the matching corresponding relationship between the feature points.

[0090] The extracted feature points and descriptors are input into the SuperGlue network. The SuperGlue network learns the deeper feature representation of the image by optimizing the graph network structure, and shows stronger robustness, successfully coping with various image transformation scenarios. The network outputs the matching corresponding relationship between the feature points, including the matched feature point pairs and the corresponding matching scores.

[0091] After inputting the extracted feature points and descriptors into the SuperGlue matcher, the core mechanism realizes accurate matching through multi-modal attention fusion and optimal transport optimization.

[0092] SuperGlue aggregates context information within a single set of features through self-attention (Self-Attention), and then calculates feature point similarity across images using cross-attention (Cross-Attention) to construct a global matching score matrix. Subsequently, based on Sinkhorn iteration, the matching problem is converted into a probability constraint optimization, and the matching probability is dynamically adjusted to eliminate low-confidence point pairs. At the same time, key point masking (Keypoint Masking) is combined to suppress noise area response, and a differentiable matching layer is used to realize end-to-end training, finally outputting high-precision matching results under geometric consistency constraints.

[0093] According to the feature points, the matching connection between the microscopic images is constructed, and the RANSAC (Random Sample Consensus) algorithm is used to eliminate the wrong matching point pairs, improving the accuracy of the matching.

[0094] In the optional embodiment, based on the homography matrix and the partial differential equation image fusion algorithm, the matching corresponding relationship of the local image features is combined to splice and optimize the local rock mineral microscopic images, and generate large field rock mineral microscopic images, including:

[0095] In step S41, the homography matrix between the matched local rock mineral microscopic image and the adjacent local rock mineral microscopic image to be spliced is calculated, and the perspective transformation is performed on the matched local rock mineral microscopic image according to the homography matrix, so that the local rock mineral microscopic image is aligned with the local rock mineral microscopic image to be spliced.

[0096] In step S42, the image sharpness recognition model is used to recognize the sharpness of the perspective-transformed local rock mineral microscopic image, and the image enhancement model is constructed to enhance the quality of the fuzzy local rock mineral microscopic image.

[0097] In step S43, the four corner point coordinates of the matched local rock mineral microscopic image after transformation are calculated, combined with the four corner point coordinates of the local rock mineral microscopic image to be spliced, and the boundary value of the spliced rock mineral microscopic image is calculated.

[0098] In step S44, according to the boundary value of the spliced rock mineral microscopic image, the translation amount between the matched local rock mineral microscopic image and the adjacent local rock mineral microscopic image to be spliced is calculated and the canvas size is adjusted.

[0099] In step S45, the sharpness of the rock mineral microscopic image is quantified by the sharpness evaluation function combined with the three-stage focusing strategy.

[0100] It should be noted that according to the matching corresponding relationship output by the SuperGlue network, the splicing process includes image alignment, fusion and other steps to generate high-quality spliced images.

[0101] The homography matrix H between the adjacent two images is calculated, and the image A is perspective-transformed according to the homography matrix H, so as to align with the image B;

[0102] By calculating the four corner point coordinates of the transformed image A, and combining with the four corner point coordinates of the image B, the boundary of the spliced image is determined;

[0103] According to the calculated boundary value, the necessary translation amount is determined to correct the image position;

[0104] The canvas size is adjusted according to these boundary values to ensure that the spliced image is complete without missing;

[0105] Copy the image B to the corresponding position of the transformed image A to complete the splicing.

[0106] Image alignment: the images are geometrically aligned by calculating the geometric transformation matrix between the matching point pairs, and the homography matrix is used for geometric alignment, the core model of which can be realized by homogeneous coordinate matrix and parameterization expansion, and the general expression is formula:

[0107]

[0108] In the formula, a 11 , a 12 , a 21 and a 22 represent image rotation, scaling and shearing parameters respectively; a 13 and a 23 represent the corresponding horizontal and vertical translation amounts respectively; a 31 and a 32 represent the control of perspective distortion.

[0109] In this optional embodiment, the local rock micrograph image after perspective transformation is subjected to sharpness identification by using an image sharpness identification model, and an image enhancement model is constructed to enhance the quality of the blurred local rock micrograph image, including:

[0110] Step S421, identify the local rock micrograph image based on a multi-scale convolutional neural network (i.e. LVNet network), add a three-dimensional attention mechanism, obtain an improved multi-scale convolutional neural network (i.e. CS-LVNet network), and optimize the local rock micrograph image area by using the channel attention and spatial attention in the three-dimensional attention mechanism;

[0111] Step S422, replace the two-dimensional layer operation and convolution operation in the channel attention and spatial attention with three-dimensional layer operation and convolution operation;

[0112] Step S423, embed the three-dimensional layer operation and convolution (i.e. 3D CBAM module) operation into the sampling layer of the improved multi-scale convolutional neural network respectively, optimize the local rock micrograph image area by using the channel attention and spatial attention layer, and extract the local rock micrograph image features therefrom;

[0113] Step S424, based on the extracted local rock micrograph image features, use the spatial sharpening filter method to enhance the quality of the blurred rock micrograph image.

[0114] It should be noted that the CS-LVNet network is based on the LVNet network, and the improved attention mechanism is added to pay more attention to the rock and mineral micro image area during training, and better results are obtained than the LVNet network. And embed the 3DCBAM module in the model structure of the LVNet network, embed the 3DCBAM module after the convolution layer in the down-sampling stage, pay more attention to the rock and mineral micro image area from the channel and spatial level when extracting features, and significantly improve the network performance.

[0115] The comparison of the segmentation results trained by the LVNet network and the segmentation results of the CS-LVNet network shows that the LVNet network and the CS-LVNet network can predict the rock and mineral micro image area, but the CS-LVNet network can predict more rock and mineral micro image areas, which shows that the CS-LVNet network can more accurately predict the rock and mineral micro image area.

[0116] The spatial sharpening filter method is a technique that enhances the details of mineral grain edges, grain boundaries, pores and other features in blurred rock and mineral micro images by operating in the image spatial domain. The core principle is to highlight the regions with large changes in gray value in the image, offset the "smoothing effect" caused by blurring, and thus improve the image clarity.

[0117] In this optional embodiment, the clarity of the rock and mineral micro image is quantified by the clarity evaluation function (i.e. Tenengrad function) combined with the three-stage focusing strategy, which includes:

[0118] Step S451, calculating the square sum of the gradient amplitude of the rock and mineral micro image using the Sobel operator (i.e. Sobel operator), and constructing the clarity evaluation curve of the rock and mineral micro image;

[0119] Step S452, combining the three-stage focusing strategy, positioning the clear area of the rock and mineral micro image according to the clarity evaluation curve of the rock and mineral micro image, and determining the best focus point

[0120] It should be noted that the clarity of the image is quantified by the clarity evaluation function Tenengrad function, and the Tenengrad function uses the Sobel operator to calculate the image gradient and takes the square sum of the gradient amplitude as the clarity index; The three-stage focusing strategy is adopted, and according to the image clarity evaluation curve, first scan in a large step range to locate the possible clear area, and then perform fine scanning around the search result, and determine the best focus point according to the front and rear focus positions.

[0121] As Figure 3As shown, in the rock mineral microscopic image acquisition and processing scene, F point is defined as the precise focusing position, and the image captured at this position has the best clarity, which is manifested as sharp edges, complete details and no optical blur. On the contrary, F1 and F2 represent the defocused images collected on both sides of the focusing position. Because these two types of images are not in the best focal plane, they all have different degrees of blur effect, which is manifested as high-frequency information attenuation, edge diffusion and contrast reduction, and the blur degree is positively correlated with the defocus distance.

[0122] In this optional embodiment, the image fusion algorithm based on homography matrix and partial differential equation, combined with the matching corresponding relationship of local image features, is used to stitch and optimize the local rock mineral microscopic images to generate a large-view rock mineral microscopic image, which further includes:

[0123] Step S46, introduce the diffusion coefficient that can reflect the spatial position of the image to smooth the boundary value of the rock mineral microscopic image;

[0124] Step S47, improve the image fusion algorithm by using partial differential equation to obtain the improved image fusion algorithm, and stitch the two aligned rock mineral microscopic images to generate a large-view rock mineral microscopic image.

[0125] It should be noted that, as Figure 4 shown, the image fusion algorithm improved by partial differential equation introduces a diffusion coefficient that can reflect the spatial position of the image to smooth the image boundary, so that the edge detail features of the fusion area are preserved.

[0126] Image fusion: the image fusion algorithm improved by partial differential equation is used to stitch the aligned images together. Let the left image be f(x, y) and the right image be g(x, y). The expression of the fusion image h(x, y) obtained by using partial differential equation in the overlapping area of the two images is as follows:

[0127]

[0128] In the formula, h(x, y) represents the fusion image; f(x, y) represents the left image; g(x, y) represents the right image; represents the partial differential symbol; represents the second-order partial derivative; x represents the horizontal coordinate of the current pixel point; y represents the vertical coordinate of the current pixel point.

[0129] When the image is fused, a diffusion coefficient that can reflect the spatial position of the image is introduced, and its expression is as follows:

[0130]

[0131] In the formula, D(x,y) represents the diffusion coefficient reflecting the spatial location of the image; exp represents the exponential function sign; x c Indicates the horizontal coordinates of the center point of the merged region; y c σ represents the vertical coordinates of the center point of the fusion region; σ represents the spatial scale controlling the diffusion coefficient.

[0132] The expression for the fused image obtained after introducing a diffusion coefficient that reflects the spatial location of the image is as follows:

[0133]

[0134] In the formula, h(x,y) represents the fused image; f(x,y) represents the left image; g(x,y) represents the right image; D(x,y) represents the diffusion coefficient that reflects the spatial position of the image; σ represents the spatial scale that controls the diffusion coefficient; x represents the horizontal coordinate of the current pixel; and y represents the vertical coordinate of the current pixel.

[0135] This allows for the removal of seams and smoothing of joint areas, resulting in a seamless stitched image. The final stitched image retains detailed information from the rock and mineral microscopic image, and the stitching quality is significantly improved through optimization algorithms.

[0136] like Figure 2 As shown, according to another embodiment of the present invention, a deep learning-based rock and mineral microscopic image stitching system is also provided, the system comprising:

[0137] Image processing module 1 is used to acquire local rock and mineral microscopic images of rock and mineral thin sections and perform preprocessing, and to perform coarse registration of overlapping areas of the preprocessed local rock and mineral microscopic images using the phase correlation method.

[0138] Feature extraction module 2 is used to extract basic and descriptive features from local rock and mineral microscopic images after coarse registration based on an improved image feature detection model, so as to obtain local image features of the local rock and mineral microscopic images.

[0139] Feature matching module 3 is used to perform feature matching on local image features of two sets of local rock and mineral microscopic images using an image feature matching model, so as to obtain the matching correspondence of local image features;

[0140] Image generation module 4 is used to stitch and optimize local rock and mineral microscopic images by combining the image fusion algorithm based on homography matrix and partial differential equations and the matching correspondence of local image features, so as to generate large field-of-view rock and mineral microscopic images.

[0141] To sum up, by means of the above technical scheme of the present application, the present application aims at the problems of high-definition image processing and splicing of various kinds of rock and ore under a microscope, such as the problem that complex transformation between the reference image and the image to be spliced leads to difficulty in matching, the problem of splicing gap and large-area color difference, and the problem that the traditional algorithm is slow and difficult to meet the real-time requirement of industrial detection, etc. Firstly, after the image is extracted, the phase correlation method is used for coarse registration of the overlapping area to reduce the influence of irrelevant feature points on subsequent image registration. Then, the lightweight SuperPoint algorithm is used to extract image feature points, the NMS non-maximum suppression is used to eliminate redundant feature points, the extracted feature points and descriptors are input into the SuperGlue matcher, the RANSAC algorithm is used to eliminate incorrect matching point pairs, and the matching relationship between the feature points of the two images is output. Finally, in the image splicing and fusion stage, the homography matrix is used to calculate the positional relationship between the images, and the image splicing and fusion are completed by using the partial differential equation improved image fusion algorithm.

[0142] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based rock and mineral microscopic image stitching method, characterized in that, The method comprises the following steps: Obtain local rock and mineral micro images of rock and mineral thin sections and preprocess them, and use a phase correlation method to coarsely register the overlapping areas of the preprocessed local rock and mineral micro images; Based on an improved image feature detection model, extract the basic features and description features in the coarsely registered local rock and mineral micro images to obtain the local image features of the local rock and mineral micro images; Use an image feature matching model to match the local image features of the two groups of local rock and mineral micro images to obtain the matching corresponding relationship of the local image features; Based on the homography matrix and the image fusion algorithm of partial differential equation, combine the matching corresponding relationship of the local image features to stitch and optimize the local rock and mineral micro images to generate a large-view rock and mineral micro image.

2. The deep learning-based rock-mineral microscopic image stitching method according to claim 1, characterized in that, The method comprises the following steps: Use an optical microscope to perform micro imaging on the rock and mineral thin sections and scan the collected continuous rock and mineral micro image sequence row by row; Perform grayscale, denoising and contrast enhancement processing on the collected continuous rock and mineral micro image sequence, and use a phase correlation method to coarsely register the overlapping areas to obtain the coarsely registered rock and mineral micro images. 3.The deep learning-based rock-mineral microscopic image stitching method according to claim 2, characterized in that, The method comprises the following steps: Based on the Fourier transform formula, perform a phase difference processing on the coarsely registered same two rock and mineral micro images. 4.The deep learning-based rock-mineral microscopic image stitching method according to claim 1, characterized in that, The method comprises the following steps: Based on a lightweight convolutional neural network architecture and a decoupled fully connected attention, improve the encoder in the image feature detection model to obtain an improved image feature detection model, wherein the image feature detection model is composed of an encoder and two independent decoders; Input the coarsely registered local rock and mineral micro images into the improved image feature detection model, process them through the encoder and the decoder, extract the basic features and the description features in the local rock and mineral micro images, and obtain the local image features of the local rock and mineral micro images. 5.The deep learning-based rock-mineral microscopic image stitching method according to claim 1, wherein, The method comprises the following steps: Through a self-attention mechanism, aggregate the context information inside the basic features and the description features, and use a cross-attention mechanism to calculate the feature point similarity between different local rock and mineral micro images to construct a global matching score matrix; Based on an iterative algorithm, convert the matching problem into a probability constraint optimization, dynamically adjust the matching probability to eliminate low-confidence point pairs, combine a key point mask algorithm, realize end-to-end training through a differentiable matching layer, and obtain the matching results under the geometric consistency constraint; According to the basic features, construct matching connection lines between the local micro images, and use a random sample consistency algorithm to eliminate the incorrect matching point pairs in the matching results under the geometric consistency constraint to obtain the matching corresponding relationship of the local image features. 6.The deep learning-based rock-mineral microscopic image stitching method according to claim 1, wherein, The image fusion algorithm based on homography matrix and partial differential equation combines the matching corresponding relationship of local image features, splices and optimizes the local rock mineral micro images, and generates large-view rock mineral micro images, including: The homography matrix between the matched local rock mineral micro image and the adjacent local rock mineral micro image to be spliced is calculated, and the perspective transformation is performed on the matched local rock mineral micro image according to the homography matrix, so that the local rock mineral micro image is aligned with the local rock mineral micro image to be spliced; The local rock mineral micro image after perspective transformation is subjected to sharpness recognition by using an image sharpness recognition model, and an image enhancement model is constructed to enhance the quality of the fuzzy local rock mineral micro image; The four corner point coordinates of the matched local rock mineral micro image after transformation are calculated, and are combined with the four corner point coordinates of the local rock mineral micro image to be spliced to calculate the boundary value of the spliced rock mineral micro image; According to the boundary value of the spliced rock mineral micro image, the translation amount between the matched local rock mineral micro image and the adjacent local rock mineral micro image to be spliced is calculated, and the canvas size is adjusted; And through the sharpness evaluation function combined with the three-stage focusing strategy, the sharpness of the rock mineral micro image is quantified.

7. The deep learning-based rock-mineral microscopic image stitching method according to claim 6, characterized in that, The image sharpness recognition model is used to recognize the sharpness of the local rock mineral micro image after perspective transformation, and an image enhancement model is constructed to enhance the quality of the fuzzy local rock mineral micro image, including: Based on the multi-scale convolutional neural network, the local rock mineral micro image is recognized, a three-dimensional attention mechanism is added, an improved multi-scale convolutional neural network is obtained, and the channel attention and spatial attention in the three-dimensional attention mechanism are used to optimize the local rock mineral micro image region; The two-dimensional layer operation and convolution operation in the channel attention and spatial attention are replaced by three-dimensional layer operation and convolution operation; The three-dimensional layer operation and convolution operation are embedded into the sampling layer of the improved multi-scale convolutional neural network respectively, the local rock mineral micro image region is optimized by using the channel attention and spatial attention layer, and the local rock mineral micro image features are extracted from the local rock mineral micro image region; Based on the extracted local rock mineral micro image features, the quality of the fuzzy rock mineral micro image is enhanced by using the spatial sharpening filter method.

8. The deep learning-based rock-mineral microscopic image stitching method according to claim 7, characterized in that, The sharpness of the rock mineral micro image is quantified by using the sharpness evaluation function combined with the three-stage focusing strategy, including: The Sobel operator is used to calculate the gradient amplitude square sum of the rock mineral micro image to form a sharpness evaluation curve of the rock mineral micro image; Combined with the three-stage focusing strategy, the sharpness area of the rock mineral micro image is located according to the sharpness evaluation curve of the rock mineral micro image, and the best focus point is determined. 9.The deep learning-based rock-mineral microscopic image stitching method according to claim 8, characterized in that, The image fusion algorithm based on homography matrix and partial differential equation combines the matching corresponding relationship of local image features, splices and optimizes the local rock mineral micro images, and generates large-view rock mineral micro images, and also includes: The diffusion coefficient reflecting the spatial position of the image is introduced to smooth the boundary value of the rock mineral micro image; The image fusion algorithm is improved by using the partial differential equation to obtain an improved image fusion algorithm, and the two aligned rock mineral micro images are spliced to generate a large-view rock mineral micro image.

10. A deep learning-based rock and mineral microscopic image stitching system for implementing the deep learning-based rock and mineral microscopic image stitching method of any one of claims 1-9, characterized in that, Including: The image processing module is configured to acquire and preprocess a local rock and mineral micrograph of a rock and mineral slice, and perform coarse registration on the local rock and mineral micrograph after preprocessing by using a phase correlation method. The feature extraction module is configured to extract basic features and description features in the local rock and mineral micrograph after coarse registration based on an improved image feature detection model, and obtain local image features of the local rock and mineral micrograph. The feature matching module is configured to perform feature matching on the local image features of the two groups of local rock and mineral micrographs by using an image feature matching model, and obtain a matching corresponding relationship of the local image features. The image generation module is configured to perform splicing and optimization on the local rock and mineral micrographs based on a homography matrix and an image fusion algorithm of a partial differential equation, and generate a large-view rock and mineral micrograph in combination with the matching corresponding relationship of the local image features.

Images