Cave temple surface disease automatic identification method based on multi-modal imaging and visual segmentation
By combining a four-channel beam-splitting prism group and a low-rank adaptive module, multimodal imaging and visual segmentation of surface defects in grotto temples were achieved, solving the problems of insufficient single-modal detection and spatial misalignment, and improving the accuracy and grading effect of defect detection.
Patent Information
- Application Number
- CN202611241170.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-17
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2046-08-17
AI Technical Summary
Existing technologies for detecting surface defects in grottoes suffer from problems such as insufficient coverage of single-modal detection, spatial misalignment caused by time-division acquisition, dilution of information from simple fusion, and overfitting of segmentation models in small sample scenarios.
A four-channel beam-splitting prism group is used for multimodal imaging. Visible, near-infrared, mid-wave infrared and ultraviolet fluorescence images are acquired through synchronous exposure. Combined with a low-rank adaptive module and frequency domain mode selection, pixel-level alignment and feature fusion are achieved. Morphological skeletonization is used to extract and assist in disease classification.
It achieves pixel-level spatial alignment of multimodal images, fully utilizes the complementary advantages of different modalities, reduces the risk of overfitting, and improves the accuracy of disease detection and the degree of hierarchical matching.
Smart Images

Figure CN122780703B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing technology, specifically relating to an automatic identification method for surface defects in grotto temples based on multimodal imaging and visual segmentation. Background Technology
[0002] Grotto temples are important immovable cultural relics. Their statues, walls, and inscriptions are exposed to the natural environment for extended periods, suffering from various surface defects such as cracks, powdering, peeling, salt precipitation, contamination crusting, and biofouling due to multiple factors including temperature and humidity changes, moisture penetration, salt crystallization, and microbial erosion. Accurate identification and quantitative assessment of the type, extent, and severity of these defects are fundamental to developing scientific restoration plans. Traditional surveys of grotto temple surface defects rely primarily on on-site visual observation and manual mapping by cultural relic conservation professionals. This approach is inefficient, prone to subjective variations, and has limited coverage of high cliff faces and deep niches. In recent years, visible light digital photogrammetry, infrared thermal imaging, and ultraviolet fluorescence detection have been increasingly applied to the non-destructive testing of cultural relic surface defects. At the image analysis level, deep learning-based semantic segmentation methods have made significant progress in scenarios such as building crack detection and road damage identification. Encoder-decoder networks, represented by U-Net and its variants, can achieve pixel-level segmentation of defect areas. However, existing technologies still face the following challenges. At the data acquisition level, existing solutions typically employ single-modal or time-sharing acquisition of images from different modalities. Single-modal acquisition struggles to cover the effective detection needs of all disease types, while time-sharing acquisition suffers from spatial misalignment between different modal images due to inter-frame latency, impacting subsequent fusion accuracy. At the multimodal fusion level, existing methods often use simple channel stitching or pixel-by-pixel weighted averaging, failing to fully consider the differences in information contribution from different modalities at different spatial frequency scales. This results in redundant interference between modalities and dilution of effective information in the fusion results. At the segmentation and recognition level, the pixel-level labeled data for the vertical domain of grotto temple diseases is limited. Training a large segmentation network from scratch faces a serious risk of overfitting, while the computational cost and data requirements for full parameter fine-tuning of a pre-trained model are equally difficult to meet. Summary of the Invention
[0003] Therefore, the main objective of this invention is to provide an automatic identification method for surface defects of grotto temples based on multimodal imaging and visual segmentation, which solves the problems of insufficient coverage of single-modal detection, spatial misalignment of time-sharing acquisition, dilution of information from simple fusion, and overfitting of segmentation models in small sample scenarios in the prior art.
[0004] The technical solution adopted in this invention is as follows: An automatic identification method for surface defects in grotto temples based on multimodal imaging and visual segmentation includes the following steps: Step 1: The incident beam is split into four channels—visible light, near-infrared, mid-wave infrared, and ultraviolet fluorescence—using a four-channel beam splitter prism group. An area array sensor is installed at the end of each channel and four original images are simultaneously exposed to obtain the images. After distortion correction and cross-modal space registration, a pixel-level aligned four-modal registration image group is obtained. Step 2: Transform the four registered images to the frequency domain and divide them into multiple concentric ring frequency bands. Calculate the normalized frequency band energy of each mode in each concentric ring frequency band. Mark the mode with the largest normalized frequency band energy in each concentric ring frequency band as the active mode. In the frequency domain of each image, only retain the frequency band component of the current mode that is the active mode. After inverse transformation to restore it to the spatial domain, stitch it along the channel dimension to generate a 4-channel fused feature map. Step 3: Insert a low-rank adaptive module into the image encoder of the general segmentation base model, freeze the original pre-training parameters and fine-tune the training of the low-rank adaptive module, input the 4-channel fused feature map into the image encoder to obtain image embedding features, output a pixel-level segmentation mask by the mask decoder, output a disease type label by the disease type classification head, determine the disease level based on the pixel-level segmentation mask and disease type label, and generate a disease distribution map and disease classification results on the surface of the grotto temple.
[0005] Furthermore, in step 1, the four-channel beam splitter assembly includes a first dichroic beam splitter, a second dichroic beam splitter, and a third dichroic beam splitter arranged sequentially along the optical path propagation direction; the first dichroic beam splitter splits the incident beam into a transmitted visible light beam and a reflected non-visible light beam; the second dichroic beam splitter splits the non-visible light beam into a transmitted near-infrared beam and a reflected mixed ultraviolet and mid-infrared beam; the third dichroic beam splitter splits the mixed ultraviolet and mid-infrared beam into a transmitted mid-infrared beam and a reflected ultraviolet beam.
[0006] Furthermore, in step 1, an ultraviolet excitation light source is set around the acquisition lens to illuminate the surface of the grotto temple target with ultraviolet excitation. An excitation cutoff filter is set in the ultraviolet fluorescence channel to block the direct transmission of ultraviolet excitation light, allowing only the fluorescence emission signal generated by the excitation of the target surface to enter the area array sensor at the end of the ultraviolet fluorescence channel.
[0007] Furthermore, in step 1, a visible light area array sensor is installed at the end of the visible light channel, a near-infrared area array sensor is installed at the end of the near-infrared channel, a mid-wave infrared area array thermal imaging sensor is installed at the end of the mid-wave infrared channel, and an ultraviolet-enhanced area array sensor is installed at the end of the ultraviolet fluorescence channel; synchronous exposure trigger pulses are simultaneously sent to the four area array sensors through a field-programmable gate array.
[0008] Furthermore, in step 1, the cross-modal space registration method is as follows: using the pre-calibrated intrinsic parameters and distortion coefficients of each area array sensor, lens distortion correction is performed on the four original images respectively. Using the distortion-corrected visible light image as the reference image, based on the pre-calibrated extrinsic parameter mapping relationship between each modal sensor, the distortion-corrected near-infrared image, mid-wave infrared image, and ultraviolet fluorescence image are aligned to the pixel coordinate system of the visible light image through coordinate reprojection transformation.
[0009] Furthermore, in step 2, the transformation to the frequency domain is performed as follows: a 2D Fast Fourier Transform is performed on each registered image to obtain a frequency domain complex matrix, and the modulus of each element in the frequency domain complex matrix is taken to obtain the frequency domain amplitude spectrum. The concentric ring frequency bands are divided as follows: the geometric center of the frequency domain amplitude spectrum is taken as the center, the distance from the center to the short edge of the frequency domain amplitude spectrum is taken as the maximum radius, the maximum radius is divided into N equal segments, and concentric circles are drawn with the endpoints of each segment as the radius, dividing the area into N concentric ring frequency bands numbered sequentially from the center outwards as the first ring band to the Nth ring band. The normalized frequency band energy is calculated as follows: the amplitude of all values in each concentric ring frequency band region is accumulated to obtain the frequency band energy value, the total energy of the current mode is obtained by summing all N frequency band energy values of the same mode, and the normalized frequency band energy is obtained by dividing each frequency band energy value by the total energy of the current mode.
[0010] Furthermore, in step 2, after marking the active modes of all concentric ring frequency bands, a frequency band activation mapping table is generated. The frequency band activation mapping table contains N records, each recording stores one concentric ring frequency band number and the corresponding active mode identifier. Based on the frequency band activation mapping table, each frequency domain complex matrix is checked one by one from the first ring band to the Nth ring band. If the active mode identifier of the current ring band is the same as the mode corresponding to the current frequency domain complex matrix, all complex coefficients in the current ring band are retained; otherwise, all complex coefficients in the current ring band are set to zero.
[0011] Furthermore, in step 3, a channel-adaptive convolutional layer is set at the input of the image encoder. The channel-adaptive convolutional layer has 4 input channels and the number of output channels is the same as the number of input channels in the original pre-training of the image encoder. Low-rank adaptive modules are inserted in parallel next to the self-attention projection matrix of each transformer layer inside the image encoder. Each low-rank adaptive module is composed of a dimension-reduced linear mapping layer and a dimension-upgrading linear mapping layer connected in series. The output of the low-rank adaptive module is added element-wise to the output of the original self-attention projection matrix to obtain the final output of the current transformer layer. The parameters of the channel-adaptive convolutional layer and the parameters of all low-rank adaptive modules are set to a trainable state.
[0012] Furthermore, in step 3, uniform grid sampling is performed on the spatial dimension of the image embedding features according to preset row and column spacing to generate a set of grid cue points. The mask decoder combines the image embedding features and the set of grid cue points to output a pixel-level segmentation mask for each candidate disease region. For each disease region, the corresponding pixel-level segmentation mask is adjusted to the same spatial resolution as the image embedding features through spatial downsampling. Feature vectors of all spatial locations within the coverage area of the downsampled pixel-level segmentation mask are extracted from the image embedding features and global average pooling is performed to obtain a region pooling feature vector. The region pooling feature vector is input into the disease type classification head to output the disease type label.
[0013] Furthermore, in step 3, the method for determining the disease level is as follows: for disease areas labeled as cracks, morphological skeletonization is performed on the corresponding pixel-level segmentation mask to extract the crack centerline, and the total length and maximum connectivity length of the crack centerline are counted as morphological parameters. The disease level is determined by comparing the area ratio and morphological parameters with the corresponding crack grading threshold. For disease areas labeled as powdering, peeling, salting out, contamination crusting, or biological attachment, the disease level is determined by comparing the area ratio with the grading threshold of the corresponding type.
[0014] By employing the above technical solution, this invention achieves the following beneficial effects: A four-channel beam splitter prism group splits the incident beam from the acquisition lens into four independent optical paths: visible light, near-infrared, mid-wave infrared, and ultraviolet fluorescence. A field-programmable gate array (FPGA) drives four area array sensors to complete synchronous exposure simultaneously. This eliminates the spatial misalignment problem of multimodal images caused by inter-frame delay in time-division acquisition, ensuring the pixel-level spatial alignment required for subsequent fusion processing. In the multimodal fusion stage, this invention transforms the four registered images to the frequency domain and finely divides them into concentric ring frequency bands. By calculating the normalized energy of each mode in each frequency band and selecting the mode with the highest energy as the activated mode, selective fusion is achieved by adaptively selecting the optimal information source mode according to the spatial frequency scale. This fully utilizes the complementary advantages of different modes in different frequency ranges, avoiding mutual interference of redundant signals between modes and the dilution of effective information in simple channel stitching methods. In the segmentation and recognition stage, this invention uses a general segmentation model as the backbone network, and only inserts a small number of low-rank adaptive parameters for domain adaptation fine-tuning. While retaining the general visual feature extraction capabilities acquired through large-scale pre-training, it achieves effective transfer of grotto temple disease scenes with a minimal number of trainable parameters, reducing dependence on large-scale labeled data and the risk of overfitting. Furthermore, for crack-type diseases, this invention introduces morphological skeletonization to extract the centerline and calculates the total length and maximum connected length as auxiliary grading indicators. Combined with area proportion, it achieves differentiated disease grading judgment, improving the matching degree between the grading results and the actual degree of disease damage. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the optical path splitting principle of the four-channel beam splitting prism group provided in an embodiment of the present invention; Figure 2 The four-mode normalized frequency band energy distribution and activation mode selection curves are provided in the embodiments of the present invention. Detailed Implementation
[0016] An automatic identification method for surface defects in grotto temples based on multimodal imaging and visual segmentation includes the following steps: Step 1: The incident beam is split into four channels—visible light, near-infrared, mid-wave infrared, and ultraviolet fluorescence—using a four-channel beam splitter prism group. An area array sensor is installed at the end of each channel and four original images are simultaneously exposed to obtain the images. After distortion correction and cross-modal space registration, a pixel-level aligned four-modal registration image group is obtained. Step 2: Transform the four registered images to the frequency domain and divide them into multiple concentric ring frequency bands. Calculate the normalized frequency band energy of each mode in each concentric ring frequency band. Mark the mode with the largest normalized frequency band energy in each concentric ring frequency band as the active mode. In the frequency domain of each image, only retain the frequency band component of the current mode that is the active mode. After inverse transformation to restore it to the spatial domain, stitch it along the channel dimension to generate a 4-channel fused feature map. Step 3: Insert a low-rank adaptive module into the image encoder of the general segmentation base model, freeze the original pre-training parameters and fine-tune the training of the low-rank adaptive module, input the 4-channel fused feature map into the image encoder to obtain image embedding features, output a pixel-level segmentation mask by the mask decoder, output a disease type label by the disease type classification head, determine the disease level based on the pixel-level segmentation mask and disease type label, and generate a disease distribution map and disease classification results on the surface of the grotto temple.
[0017] The surface defects of grotto temples are diverse, including cracks, pulverization, flaking, salting out, contamination crusting, and biofouling. Different types of defects exhibit different response characteristics under different electromagnetic bands. For example, cracks show obvious grayscale jumps and shadow lines in the visible light band, while in the mid-infrared band, local temperature anomalies occur due to the difference in thermal conduction between the crack interior and the surrounding intact stone. Chlorophyll and metabolites in biofouling produce characteristic fluorescence emission under ultraviolet light excitation, but only show color difference in the visible light band, which is easily confused with the mineral background. Relying on image data of a single band is often insufficient to effectively detect all the above-mentioned defect types simultaneously. Therefore, this scheme adopts multimodal simultaneous imaging, acquiring image information of four bands—visible light, near-infrared, mid-infrared, and ultraviolet fluorescence—in a single exposure cycle, enhancing the separability of defects at the signal source level.
[0018] The core component of multimodal synchronous acquisition is a four-channel beam-splitter assembly. Its function is to split the broadband incident light collected by the acquisition lens into four independent optical paths according to wavelength range, ensuring that each optical path contains only light energy of a specific wavelength band. The four-channel beam-splitter assembly consists of a first dichroic beam-splitter, a second dichroic beam-splitter, and a third dichroic beam-splitter arranged sequentially along the optical path propagation direction. A dichroic beam-splitter is an optical element that utilizes the interference effect of multilayer dielectric thin films to achieve wavelength-selective transmission and reflection. By controlling the material, thickness, and number of thin film layers, the cutoff wavelength is set, causing light shorter than the cutoff wavelength to be reflected while light longer than the cutoff wavelength is directly transmitted, or vice versa.
[0019] The specific beam splitting process is as follows. The acquisition lens converges the light reflected and emitted from the surface of the grotto temple target into an incident beam, which first reaches the first dichroic beam splitter. The cutoff wavelength of the first dichroic beam splitter is set around 700 nm. Visible light beams with wavelengths in the range of 400 nm to 700 nm continue to propagate along the original optical axis after passing through the first dichroic beam splitter. Non-visible light beams with wavelengths greater than 700 nm and less than 400 nm are reflected on the reflective surface of the first dichroic beam splitter, deviating from the original optical axis and entering the lateral optical path. In an optional embodiment, the cutoff wavelength of the first dichroic beam splitter can be adjusted within the range of 680 nm to 720 nm according to the acquisition environment and the spectral characteristics of the target stone material.
[0020] The non-visible light beam reflected by the first dichroic beam splitter reaches the second dichroic beam splitter. The cutoff wavelength of the second dichroic beam splitter is set around 1000 nm. Near-infrared beams with wavelengths in the range of 700 nm to 1000 nm continue to propagate along the lateral optical path after passing through the second dichroic beam splitter. Mixed ultraviolet and mid-infrared beams with wavelengths greater than 1000 nm and less than 400 nm are reflected on the reflective surface of the second dichroic beam splitter and enter the second lateral optical path. In an optional embodiment, the upper limit of the operating wavelength of the near-infrared channel can be extended to 1100 nm to cover part of the short-wave infrared region and enhance the detection sensitivity of differences in the composition of stony minerals.
[0021] refer to Figure 1 ,like Figure 1As shown, the acquisition lens converges the light reflected and emitted from the surface of the grotto target into an incident beam, which propagates to the right along the optical axis of the acquisition lens. An ultraviolet excitation source with a center wavelength of 365 nm is placed around the acquisition lens, irradiating the surface of the grotto target with ultraviolet excitation light, exciting the surface material to produce fluorescence emission. The incident beam first reaches the first dichroic beam splitter, which is placed at a 45-degree angle on the optical axis, splitting the incident beam into a visible light beam transmitted along the original optical axis and a non-visible light beam reflected downwards. The wavelength range of the visible light beam is 400 nm to 700 nm. The visible light beam continues to propagate horizontally, reaching the visible light array sensor installed at the end of the visible light channel. The non-visible light beam propagates downwards to the second dichroic beam splitter, which splits it into a downward-transmitting near-infrared beam and a right-reflecting mixed ultraviolet and mid-infrared beam. The near-infrared beam has a wavelength range of 700 nm to 1000 nm. The near-infrared beam continues downwards to the near-infrared external array sensor. The mixed ultraviolet and mid-infrared beam propagates to the right to the third dichroic beam splitter, which splits it into a right-transmitting mid-infrared beam and a downward-reflecting ultraviolet beam. The mid-infrared beam has a wavelength range of 3000 nm to 5000 nm, and the ultraviolet beam has a wavelength range of 300 nm to 400 nm. The mid-infrared beam reaches the mid-infrared external array thermal imaging sensor, while the ultraviolet beam, after passing through an excitation cutoff filter in its downward propagation path, reaches the ultraviolet-enhanced area array sensor. The excitation cutoff filter has a cutoff wavelength set at 400 nm, blocking the reflected echo of ultraviolet excitation light from directly entering the sensor and allowing only fluorescence emission signals with wavelengths longer than 400 nm to pass through. Four area array sensors are synchronously triggered for exposure via a field-programmable gate array (FPGA). The FPGA sends trigger pulses to all four area array sensors simultaneously on the rising edge of the same clock, ensuring that all four images are acquired at the same time. Through this beam splitting structure, the single incident beam from the acquisition lens is split into four independent optical paths: a visible light channel, a near-infrared channel, a mid-infrared channel, and an ultraviolet fluorescence channel. These four channels share the same entrance pupil and optical axis origin, achieving natural spatial alignment of multimodal images at the optical level.
[0022] The mixed ultraviolet and mid-infrared beam reaches the third dichroic beam splitter. The third dichroic beam splitter employs a long-pass design with a cutoff wavelength set around 2500 nm. Mid-infrared beams with wavelengths in the 3000-5000 nm range pass through the third dichroic beam splitter, while ultraviolet beams with wavelengths in the 300-400 nm range are reflected at the reflective surface of the third dichroic beam splitter. Thus, the incident beam is completely split into four independent optical paths: a visible light path, a near-infrared path, a mid-infrared path, and an ultraviolet fluorescence path. Each independent optical path contains only the light energy of its corresponding wavelength band.
[0023] In an optional implementation, the arrangement order of the three dichroic beam splitters can be adjusted according to the optical design requirements. For example, the mid-infrared channel can be separated first, followed by the near-infrared and ultraviolet fluorescence channels, with the remaining visible light beam directly entering the visible light channel. Different arrangement orders do not change the final result of forming four independent optical paths, but they will affect the optical path length and light energy distribution ratio of each channel. In engineering implementation, optimization is required based on the sensitivity requirements of the sensors at the end of each channel.
[0024] At the ends of the four independent optical paths are band-matched area array sensors. The visible light channel ends with a visible light area array sensor, a silicon-based complementary metal-oxide-semiconductor (CMOS) image sensor with a wavelength range of 400 nm to 700 nm, a pixel size of 3.45 μm, and an array size of 2048 pixels by 2048 pixels. The near-infrared channel ends with a near-infrared area array sensor, using a silicon-based area array sensor or an indium gallium arsenide (IGaAs) area array sensor with enhanced near-infrared response, a wavelength range of 700 nm to 1000 nm, a pixel size of 5 μm, and an array size of 1024 pixels by 1024 pixels. The mid-infrared channel ends with a mid-infrared area array thermal imaging sensor, an uncooled focal plane array made of vanadium oxide or amorphous silicon thin film, a wavelength range of 3000 nm to 5000 nm, a pixel size of 17 μm, and an array size of 640 pixels by 512 pixels. An ultraviolet-enhanced area array sensor is installed at the end of the ultraviolet fluorescence channel. This sensor can be a back-illuminated thin-film silicon-based area array sensor or a silicon carbide-based ultraviolet sensor, covering a wavelength range of 300 nm to 400 nm. The pixel size is 4.8 μm, and the array size is 1280 pixels by 1024 pixels. The aforementioned pixel size, array size, and sensor type are preferred configurations; in actual implementation, they can be replaced based on spatial resolution requirements, cost constraints, and available commercial sensor specifications.
[0025] Ultraviolet fluorescence imaging is an active excitation imaging method, unlike the passive imaging methods of the other three channels. The surface of the grotto temple itself does not actively emit ultraviolet fluorescence; it requires an external light source to excite the surface materials to produce fluorescence. An ultraviolet excitation source is set around the acquisition lens, using an array of ultraviolet light-emitting diodes or an ultraviolet mercury lamp with a center wavelength of 365 nm, to irradiate the surface of the grotto temple with ultraviolet excitation light. After absorbing the ultraviolet excitation light, the electrons of the surface-attached organic matter, biofilm, certain mineral salts, and remediation materials transition from the ground state to the excited state, and then return to the ground state through radiation, emitting fluorescence with a wavelength longer than the excitation wavelength. The fluorescence peak wavelength of chlorophyll in the biofilm is concentrated around 685 nm, while the fluorescence emission band of some acrylic resins used for remediation is concentrated in the range of 420 nm to 480 nm. There is a significant Stokes shift between the wavelengths of these fluorescence signals and the excitation light wavelength, which is used for filtering and separation.
[0026] An excitation cutoff filter is incorporated into the ultraviolet fluorescence channel. This filter is a long-pass optical filter with a cutoff wavelength set at 400 nm. It effectively blocks the direct transmission of ultraviolet excitation light with wavelengths shorter than 400 nm, while allowing fluorescence emission signals with wavelengths longer than 400 nm to pass through. The purpose of this filtering structure is to eliminate the drowning effect of excitation light echo on the fluorescence signal: without the excitation cutoff filter, the ultraviolet excitation light, after reflection from the target surface, enters the sensor at an intensity much higher than the fluorescence signal, causing the sensor response to be dominated by the excitation light reflection component, thus masking the effective fluorescence information. In an optional embodiment, the excitation cutoff filter can be replaced with a bandpass filter, allowing only signals in a specific fluorescence emission band to pass through, thereby further improving the signal-to-noise ratio of fluorescence imaging. However, the passband range needs to be designed according to the fluorescence emission band corresponding to the type of surface damage on the target surface.
[0027] The exposure timing of the four area array sensors is uniformly controlled by a field-programmable gate array (FPGA). The FPGA is connected to the external trigger input ports of the four area array sensors via four independent trigger signal lines. During each acquisition cycle, the FPGA simultaneously sends a 10-microsecond pulse-width level trigger signal to the four trigger signal lines on the rising edge of the same clock cycle. Upon receiving the trigger pulse, the four area array sensors synchronously begin their respective exposure processes. The significance of synchronous exposure lies in the fact that the identification of surface defects in grotto temples requires that the corresponding pixels in the multimodal images reflect the physical state of the target surface at the same location at the same time. If the four sensors are exposed at different times, and the acquisition platform experiences slight movement or vibration during this period, or the illumination conditions of the target surface change, inconsistencies will arise between the different modal images, introducing false information into subsequent fusion processing. The exposure time of each channel can be set independently; for example, the exposure time for the visible light channel is set to 5 milliseconds, the near-infrared channel to 8 milliseconds, the mid-wave infrared channel to 16 milliseconds, and the ultraviolet fluorescence channel to 20 milliseconds. The difference in exposure time between channels is determined by the sensitivity of each modal sensor and the radiation intensity of the target surface in the corresponding wavelength band. A field-programmable gate array (FPGA) ensures synchronization of the exposure start times for the four channels, rather than the same exposure duration. In alternative implementations, the FPGA can be replaced with a dedicated multi-channel synchronization trigger controller or an industrial camera synchronization trigger board.
[0028] After a single synchronous exposure, each of the four area array sensors outputs a digital image: a raw visible light image, a raw near-infrared image, a raw mid-infrared image, and a raw ultraviolet fluorescence image. Since the four area array sensors are located at different ends of the optical path and image through different optical elements, each sensor's optical path suffers from radial distortion due to lens surface shape deviations and tangential distortion due to lens misalignment. These distortions cause deviations between the geometric shape of the target in the image and its actual shape. Before the acquisition equipment leaves the factory or is deployed in the field, each area array sensor is individually calibrated using a standard checkerboard calibration board to obtain the intrinsic parameter matrix and distortion coefficients for each sensor. The intrinsic parameter matrix includes the focal length and principal point coordinates, describing the ideal projection geometry of the sensor; the distortion coefficients include radial and tangential distortion coefficients, describing the degree to which the actual image deviates from the ideal projection. The calibration process employs the Zhang Zhengyou calibration method, which involves capturing images of the calibration board at different positions and angles, and solving for the intrinsic parameter matrix and distortion coefficients based on corner detection and nonlinear optimization. Using the intrinsic parameter matrix and distortion coefficients obtained from calibration, distortion correction was performed on four original images. The pixel coordinates in the distorted images were then corrected to the coordinate system of the distortion-free images using inverse distortion mapping, resulting in four distortion-corrected images. The mathematical description of distortion correction mapping is: for any pixel coordinate point in the distorted image... ,in The x-coordinate is the pixel coordinate. For pixel ordinates, first convert the pixel coordinates to normalized coordinates with the principal point as the origin. ,in For the normalized x-axis, For the normalized ordinate, calculate the squared distance from the normalized coordinate to the origin. ,in The radial distance from the normalized coordinates to the principal point is then determined based on the radial distortion coefficient. , , and tangential distortion coefficient , Calculate the corrected distortion-free normalized coordinates ,in To correct and normalize the x-axis, To correct the normalized ordinate, the normalized coordinates are finally converted back to pixel coordinates using the intrinsic parameter matrix, thus completing the distortion correction.
[0029] Distortion correction eliminates the geometric distortion of the image caused by the optical path of a single sensor. However, the four sensors are located in different optical paths, and their optical axis directions, image plane positions, and magnifications are not entirely consistent. Even with a common aperture optical path formed by the same beam splitter group, there will inevitably be cross-modal spatial displacement caused by deviations in the beam splitter angle, differences in optical path length, and image plane mounting errors. Without cross-modal spatial registration, the physical positions of the target surface corresponding to pixels with the same spatial position in the four distortion-corrected images will not be consistent, and subsequent fusion processing will incorrectly superimpose information from different physical positions. The purpose of cross-modal spatial registration is to establish a precise pixel-level spatial correspondence between the four images, so that each pixel position after fusion strictly corresponds to the same physical point on the target surface.
[0030] Using the distortion-corrected visible light image as the reference image, the other three distortion-corrected images were registered to the pixel coordinate system of the visible light image. The visible light image was chosen as the reference image because it typically has the highest spatial resolution and the richest texture details, thus preserving the most spatial detail information. Furthermore, the visible light image aligns with human visual perception, facilitating the subsequent visualization of disease distribution maps.
[0031] The registration process is based on the pre-calibrated extrinsic parameter mapping relationship between the various modal sensors. The calibration method for the extrinsic parameter mapping relationship is as follows: A multispectral calibration board containing target points of various materials is placed in front of the acquisition device. The surface of the calibration board has target points that can be clearly imaged in four bands: visible light, near-infrared, mid-infrared, and ultraviolet fluorescence. The target points are circular markers with coatings of different reflectivities and emissivities. The center position of the target points in the four bands can be accurately located through image processing. After the four area array sensors simultaneously expose and acquire four images of the calibration board, the pixel coordinates of all target points are extracted in each image, forming four sets of pixel coordinate points. Using the pixel coordinates of the target points in the visible light image as the reference point set and the pixel coordinates of the corresponding target points in the near-infrared image as the target point set, an affine transformation matrix from the target point set to the reference point set is fitted using the least squares method. The affine transformation matrix is a 3x3 homogeneous transformation matrix that includes rotation, translation, and scaling parameters, and can describe the spatial geometric mapping relationship between the two sensors. Similarly, the affine transformation matrices for the mid-infrared image to visible light image and the ultraviolet fluorescence image to visible light image are calibrated respectively. During the on-site data acquisition at the grotto temple, for each frame of the four distortion-corrected images acquired, the coordinates of each pixel in the near-infrared, mid-infrared, and ultraviolet fluorescence images are reprojected using their respective affine transformation matrices. The corresponding position in the visible light image coordinate system is calculated, and the pixel value at the reprojected position is obtained through bilinear interpolation. After the coordinate reprojection transformation, the pixel coordinate systems of the four images are completely unified to the visible light image coordinate system, forming a pixel-level spatially aligned four-modal registration image group.
[0032] In an optional implementation, to address the spatial resolution differences caused by the varying array sizes and pixel dimensions of the four sensors, the low-resolution modality image can be spatially upsampled using bicubic interpolation before the coordinate reprojection transformation. This ensures the image has the same pixel array size as the visible light image before performing the coordinate reprojection transformation. Alternatively, instead of using an affine transformation matrix for registration, a unified stereo calibration model can be established for the four sensors. The rotation matrix and translation vector between the four sensors can then be calculated, and sub-pixel-level registration accuracy can be achieved through epipolar correction and parallax compensation.
[0033] The four-modal registration image set includes four images: a visible light registration image, a near-infrared registration image, a mid-wave infrared registration image, and an ultraviolet fluorescence registration image. The four images have the same pixel coordinate system, the same spatial resolution, and the same array size. Each pixel position corresponds to the response value of the same physical point on the surface of the grotto temple under four different electromagnetic bands, providing a spatially aligned multimodal data foundation for subsequent frequency domain fusion processing.
[0034] The four-modal registration image set obtained after step 1 contains four pixel-level spatially aligned images, each carrying the response information of the target surface under the corresponding electromagnetic band. Directly stacking the four images along the channel dimension and feeding them into the subsequent segmentation network is the most basic multimodal fusion method. However, this simple stacking approach ignores a crucial fact: different bands have different response sensitivities to disease features within different spatial frequency ranges. Spatial frequency intuitively reflects the rate of brightness change in an image. Low spatial frequencies correspond to large, gradually changing areas, such as the overall tone or macroscopic trend of temperature distribution on a complete stone surface; high spatial frequencies correspond to drastically changing local details, such as sharp gray-scale jumps at crack edges or the microscopic texture and granularity of pulverized areas. Mid-wave infrared images record the thermal radiation distribution of the target surface. The heat conduction process itself has spatial low-pass filtering properties, and thermal anomaly signals are mainly concentrated in the low and mid-frequency regions. The effective information carried by mid-wave infrared images in the high-frequency region is relatively limited, with the high-frequency components consisting mostly of sensor thermal noise. Conversely, visible light images are extremely sensitive to the response of crack edges and surface microtextures in the high-frequency region, but are easily affected by interference factors such as uneven illumination and large areas of shadow in the low-frequency region. The signal distribution of ultraviolet fluorescence images depends on the distribution pattern of surface fluorescent substances: the fluorescence of biological membranes is usually distributed in a patchy diffuse pattern, which is prominent in the mid-to-low frequency region; while the point-like distribution of mineral salt precipitation fluorescence has a strong response in the mid-to-high frequency region.
[0035] Based on the aforementioned physical facts, this proposal suggests a frequency-domain energy distribution-guided selective mode activation (SMO) fusion method. The core idea is to avoid global fusion of the entire image in the spatial domain. Instead, in the frequency domain, based on finely divided spatial frequencies, each frequency sub-interval retains only the frequency domain component of the mode with the richest information within that interval, removing the frequency domain components of other modes within that interval. The processed frequency domain data is then inversely transformed back to the spatial domain and finally stitched together to form the fused image. The effect of this approach is that the low-frequency component of the fused image may primarily originate from the thermal distribution information of the mid-wave infrared image, the mid-frequency component may primarily originate from the fluorescence patch distribution of the ultraviolet fluorescence image, and the high-frequency component may primarily originate from the texture edge details of the visible light image. The mode contribution with the highest information content is selected at each frequency level, thereby maximizing the complementary advantages of each mode without introducing redundant noise.
[0036] The specific implementation process is as follows. Two-dimensional discrete Fourier transforms are performed on the four registered images in the four-modal registered image group. The two-dimensional discrete Fourier transform decomposes the image signal in the spatial domain into a linear superposition of sinusoidal basis functions with different spatial frequencies and directions. The transformation result is a complex matrix with the same size as the original image. Each element in the matrix corresponds to a complex coefficient of a specific spatial frequency component. The magnitude of the complex coefficient reflects the amplitude of that frequency component, and the argument of the complex coefficient reflects the phase information of that frequency component. In actual calculations, the two-dimensional fast Fourier transform algorithm is used to complete the numerical computation. For one image with a size of... grayscale image ,in The number of rows in the image. The number of columns in the image. For pixel row index, Indexed by pixel column, pixel coordinates The gray value at that position is transformed into a frequency domain complex matrix by performing a 2D Fast Fourier Transform. ,in For frequency domain row index, For frequency domain column index, Frequency coordinates The complex coefficients at the specified points. After the transformation, the frequency domain complex matrix is center-shifted, moving the zero-frequency component from the upper left corner of the matrix to its geometric center. This concentrates low-frequency components in the central region and distributes high-frequency components at the edges, facilitating subsequent frequency band division based on radial distance. After performing the above transformation on the four registered images, four frequency domain complex matrices are obtained.
[0037] Next, construct the corresponding frequency domain amplitude spectrum for each frequency domain complex matrix. For each element in the array, calculate its modulus. The modulus is equal to the square root of the sum of the squares of the real and imaginary parts of the complex number. The matrix formed by all the moduli is the frequency domain amplitude spectrum. The value at each position in the amplitude spectrum reflects the energy intensity of the corresponding spatial frequency component in the image. Four frequency domain complex matrices correspond to four frequency domain amplitude spectra.
[0038] To achieve fine-grained mode selection in the frequency dimension, the frequency domain amplitude spectrum needs to be divided into multiple non-overlapping sub-regions. The division method employs a concentric ring-shaped frequency band structure, with the geometric center of the frequency domain amplitude spectrum as the center. The geometric center is the location of the zero-frequency component after centering and translation, with pixel coordinates as follows: The maximum radius is defined as the distance from the center of the circle to the short edge of the frequency domain amplitude spectrum. When the number of rows in the image Less than the number of columns hour, equal Conversely equals The maximum radius Divided into equal parts Section, among which The preset number of frequency band divisions, each segment being [length missing]. Using the center of the circle as the common center, respectively with... , , Until Draw for radius The frequency domain amplitude spectrum is divided into concentric circles. There are three concentric ring frequency bands. Numbered sequentially from the center outwards as ring 1 to ring 2. Rings, the first ring has a radius from 0 to The disk region contains the lowest frequency component, the first... The ring is the radius from arrive The annular region contains the highest frequency components. In a preferred embodiment, A value of 16 provides a suitable frequency range for each ring band. This avoids both excessively fine division leading to too few sampling points within a single ring band and unstable energy statistics, and excessively coarse division resulting in insufficient frequency selection granularity. In an optional implementation, It can take any integer value in the range of 8 to 32, and the specific value can be adjusted according to the image size and application scenario.
[0039] The fundamental reason for using concentric ring bands instead of rectangular grids or other forms of division is that all positions equidistant from the geometric center in the frequency domain correspond to the same spatial frequency, even if their directions differ. Grouping components of the same frequency magnitude into the same ring band for statistical analysis allows for obtaining direction-independent frequency energy distribution characteristics, avoiding the introduction of unnecessary directional bias. The orientation of surface defects in grotto temples is typically random, and cracks can be horizontal, vertical, or arbitrarily inclined; therefore, isotropic ring division is more suitable for this scenario.
[0040] After completing the frequency band division, the frequency band energy of each mode in each concentric ring frequency band is calculated. Specifically, for each frequency domain amplitude spectrum, iterate through each one... For each of the four concentric ring frequency bands, the amplitude values of all pixels within that region are summed sequentially. The summation result is the frequency band energy value for that ring band. Each of the four frequency domain amplitude spectrum diagrams generates... The energy values of each frequency band are assigned according to the ring band number, from ring 1 to ring 2. The rings are arranged sequentially to form a frequency band energy sequence.
[0041] Because the four modes differ in sensor type, quantization depth, and dynamic range, the absolute energy values between different bands within the same mode can span multiple orders of magnitude. Directly comparing the absolute band energy values of different modes within the same band will result in the outcome being dominated by a mode with higher overall radiation intensity, failing to truly reflect the relative information distribution characteristics of each mode across different frequency sub-intervals. Therefore, intra-mode normalization is necessary before comparison. The normalization method is as follows: for the band energy sequence of each mode, first normalize all... The total energy of the current mode is obtained by summing the energy values of each frequency band. ,in This is the sum of the energies of the current mode across all frequency bands, then the energy value of each frequency band is divided by... The normalized frequency band energy values are obtained. After normalization, the sum of all normalized frequency band energy values for each mode equals 1. The physical meaning of the normalized frequency band energy value becomes: the proportion of the current frequency band in the total energy of the current mode. If the proportion of normalized energy of a certain mode in a certain ring band is high, it indicates that the effective information of the current mode is concentrated in this frequency range. After the above normalization process, four sets of normalized frequency band energy sequences are obtained, corresponding to the four modes of visible light, near-infrared, mid-wave infrared, and ultraviolet fluorescence, respectively.
[0042] like Figure 2As shown, the horizontal axis represents the concentric ring band numbers, arranged sequentially from the 1st ring to the NbN_bNb ring. Smaller numbers correspond to lower spatial frequencies, while larger numbers correspond to higher spatial frequencies. The vertical axis represents the normalized band energy, indicating the proportion of the current band's energy in the total energy of the current mode across the entire frequency band. The four curves in the figure correspond to the four modes: visible light, near-infrared, mid-infrared, and ultraviolet fluorescence, distinguished by different line types and symbols. The mid-infrared mode curve shows a significant peak in the low-frequency ring region, indicating that the effective signal energy of the mid-infrared image is concentrated in the low-frequency region. This is consistent with the inherent spatial low-pass characteristics of heat conduction, where thermal anomaly signals have the strongest information contribution in the low-frequency band. The ultraviolet fluorescence mode curve shows a peak in the mid-to-low frequency ring region, corresponding to the medium spatial scale characteristics of the fluorescence patch distribution. The fluorescence response of biofilms and organic pollutants is mostly diffusely distributed, with its spatial variation frequency falling between large-area thermal distribution and microscopic texture. The near-infrared mode curve peaks in the mid-frequency region, reflecting the near-infrared band's sensitive response to differences in mineral composition and medium-scale reflectance variations. The visible light mode curve is prominent in the high-frequency annular region, reflecting the visible light image's sensitive response to high-frequency details such as sharp gray-scale jumps at fracture edges and the granular texture of the surface in pulverized areas. Each of the four curves has its advantages in different frequency bands and exhibits significant overlap; this differentiated frequency domain energy distribution characteristic is the physical basis for the frequency domain-guided selective mode activation fusion method. For each concentric annular frequency band number, the normalized frequency band energy values of the four curves at the current number position are read, and the largest value is selected. The mode to which the largest value belongs is marked as the active mode of the current annular band. The vertical background markers at each annular band position visually indicate which active mode the current annular band is assigned to. This reveals that the low-frequency band is mainly activated by the mid-infrared mode, the mid-low-frequency band by the ultraviolet fluorescence mode, the mid-frequency band by the near-infrared mode, and the high-frequency band by the visible light mode. This distribution pattern, dominated by different modes from low to high frequency, reflects the complementary characteristics of multimodal images in the frequency domain. The subsequent selective retention operation performs ring-band retention or zeroing processing on each frequency domain complex matrix based on the above activation relationship.
[0043] Based on the normalized frequency band energy sequence, the frequency bands are numbered from the first ring band to the second ring band. The ring bands are compared cross-mode one by one. For each ring band number, the four normalized frequency band energy values at the corresponding positions are read from the normalized frequency band energy sequence of the four modes. The four values are compared, and the mode with the largest value is selected as the active mode of the current ring band. The reason for selecting the mode with the largest normalized energy as the active mode is that the higher the proportion of normalized energy, the richer the signal structure of the current mode in this frequency range, and the greater the amount of effective information it carries. For all After completing the above comparisons and markings for each concentric ring frequency band, a frequency band activation mapping table is generated. The frequency band activation mapping table is a table containing... The lookup table structure consists of records, each record comprising a ring band number and an activation mode identifier. The significance of the frequency band activation mapping table lies in its establishment of a correspondence between spatial frequencies and the optimal information source modes; subsequent selective retention operations are entirely driven by this table.
[0044] In an optional implementation, when two or more modes have equal normalized band energy values in the same ring band, they can be selected according to a preset mode priority order. For example, the visible light mode is selected first, followed by the near-infrared mode, then the ultraviolet fluorescence mode, and finally the mid-wave infrared mode. This priority order can be adjusted according to the contribution of each mode to the discrimination of the main disease types in the specific application scenario.
[0045] Selective retention is performed on the four frequency domain complex matrices according to the frequency band activation mapping table. This operation is performed as follows: take one frequency domain complex matrix, determine its mode identifier, and then transfer it from the first ring to the second ring. The loop checks the activation mode identifier corresponding to the loop number in the frequency band activation mapping table one by one. If the activation mode identifier of the current loop is the same as the mode to which the current frequency domain complex matrix belongs, it indicates that the current mode is the richest source of information in this frequency range. In this case, all complex coefficients in the current frequency domain complex matrix that fall within this loop region are retained without any modification. If the activation mode identifier of the current loop is different from the mode to which the current frequency domain complex matrix belongs, it indicates that another mode carries richer information in this frequency range, and the component of the current mode in this frequency range is a redundant or non-dominant signal. In this case, the real and imaginary parts of all complex coefficients in the current frequency domain complex matrix that fall within this loop region are set to zero. This determines a specific frequency domain position. Whether it falls into the first The ring method involves calculating the Euclidean distance from the frequency domain location to the geometric center. ,in The radial distance from the frequency domain location to the geometric center. For frequency domain row index, For frequency domain column index, Let be the row number of the frequency domain matrix. Let be the number of columns in the frequency domain matrix, if If true, then the frequency domain position Belongs to the Circular zone.
[0046] After selectively preserving the complex coefficients of the four frequency domain complex matrices, each matrix retains only the complex coefficients within the bands marked as the active mode, while the complex coefficients of the remaining bands are set to zero. Next, a 2D inverse fast Fourier transform is performed on each of the four frequency domain complex matrices to restore the frequency domain data to the spatial domain, resulting in four selectively activated spatial domain images. The inverse transform is the inverse operation of the forward transform, where the retained frequency domain coefficients are superimposed according to the sinusoidal basis functions of the corresponding frequencies to reconstruct the spatial domain signal. Because the complex coefficients of some bands are set to zero, the reconstructed spatial domain image only contains the spatial information components of the current mode within the activated frequency band, which is equivalent to applying an adaptive bandpass filter defined by the frequency band activation map to the original registered image.
[0047] Four selectively activated spatial domain images are stitched together along the channel dimension to generate a 4-channel fused feature map. The stitching method is the same as the organization of the multi-channel images: the first channel of the fused feature map contains the visible light selectively activated spatial domain image, the second channel contains the near-infrared selectively activated spatial domain image, the third channel contains the mid-wave infrared selectively activated spatial domain image, and the fourth channel contains the ultraviolet fluorescence selectively activated spatial domain image. The spatial size of the 4-channel fused feature map is consistent with the size of a single image in the four-modal registration image group, but each channel only retains the spatial component of the current mode in the information-dominant frequency band, realizing cross-modal selective information fusion guided by frequency domain energy distribution.
[0048] In an optional implementation, the frequency band division can also employ a non-equidistant spacing scheme. Considering that frequency variations in the low-frequency region have a more significant impact on image content while the frequency resolution requirements in the high-frequency region are relatively lower, a logarithmic spacing method can be used. This results in a narrower ring band width in the low-frequency region for more precise frequency selection, and a wider ring band width in the high-frequency region to ensure sufficient sampling points for energy statistics within each ring band. Another optional implementation is to use sector-shaped regions instead of ring bands, dividing the frequency domain simultaneously radially and angularly to form a grid structure with intersecting ring and sector shapes. Activation mode selection is then performed independently within each sub-region. This method introduces directional sensitivity and may have stronger adaptive capabilities for crack-like diseases with obvious directional characteristics, but the corresponding computational complexity will also increase.
[0049] The 4-channel fused feature map output from step 2 serves as the input data for subsequent disease segmentation and classification. Traditional semantic segmentation networks are trained from scratch for specific tasks or pre-trained on general datasets and then fine-tuned for the target domain. Both approaches face challenges in the scenario of identifying diseases in grotto temples. Training from scratch requires a large number of grotto temple disease images with pixel-level annotations, but obtaining such annotation data is extremely costly, requiring professionals in the field of cultural relic protection to label disease types and boundaries pixel by pixel. Currently, the publicly available grotto temple disease annotation datasets are limited in size. Fine-tuning, on the other hand, carries the risk of overfitting: when the training sample size is small, performing full gradient updates on millions or even hundreds of millions of parameters of a large segmentation network can easily lead to the network over-memorizing on limited training samples and losing its generalization ability.
[0050] The strategy employed in this approach is as follows: a pre-trained general segmentation backbone model is used as the main segmentation network, all original parameters learned on a large-scale general dataset are frozen, and domain adaptation is performed only by inserting a small number of trainable low-rank adaptive parameters. The general segmentation backbone model is a large-scale visual segmentation model trained on over 10 million images and over 1 billion mask annotations, possessing extremely strong image feature extraction capabilities and zero-shot segmentation generalization. Retaining its original parameters means preserving this general visual knowledge. The role of the low-rank adaptive parameters is to introduce specific preferences related to the disease domain of the grotto temple on top of this general knowledge. This approach requires only 1% to 5% of the total number of trainable parameters in the original model, significantly reducing the need for labeled data and the risk of overfitting.
[0051] The general segmentation model consists of two parts: an image encoder and a mask decoder. The image encoder is responsible for converting the input image into a high-dimensional feature representation, while the mask decoder is responsible for generating a pixel-level segmentation mask based on the feature representation and prompts. The image encoder adopts a visual transformer architecture, consisting of multiple cascaded transformer layers. Each transformer layer contains a multi-head self-attention mechanism and a feedforward network. The core operation of the multi-head self-attention mechanism involves three linear mapping processes: query projection, key projection, and value projection. These processes map the input features to the query space, key space, and value space, respectively. Then, the attention distribution is calculated using the inner product of the query and the key. Finally, the features in the value space are weighted and aggregated using the attention distribution as coefficients.
[0052] The original design of the image encoder in the general segmentation base model accepts 3-channel image input, while the 4-channel fused feature map output in step 2 of this scheme contains 4 channels. To address the mismatch in the number of input channels, a channel-adaptive convolutional layer is added to the input of the image encoder. The channel-adaptive convolutional layer is a 2D convolutional layer with a kernel size of 1 pixel by 1 pixel, 4 input channels, and 3 output channels. Its function is to linearly combine the 4-channel input to a 3-channel output in the channel dimension, ensuring that the mapped data dimension matches the original pre-trained input specifications of the image encoder. Choosing a kernel size of 1 pixel by 1 pixel means that the channel-adaptive convolutional layer only performs linear mixing in the channel dimension, without changing the information distribution in the spatial dimension. The parameters of the channel-adaptive convolutional layer are updated with gradients along with the low-rank adaptive parameters during the fine-tuning training phase. In an optional implementation, the kernel size of the channel-adaptive convolutional layer can be set to 3 pixels by 3 pixels or 7 pixels by 7 pixels to introduce local spatial context information while mapping the channels.
[0053] The low-rank adaptive parameter is inserted next to the self-attention projection matrix of each transformer layer within the image encoder. For the query projection, key projection, and value projection processes in each transformer layer, a low-rank adaptive branch is inserted in parallel next to the original projection matrix. Each low-rank adaptive branch consists of a cascaded dimensionality reduction linear mapping layer and a dimensionality increase linear mapping layer. The dimensionality reduction linear mapping layer transforms the input features from the original dimension... Mapping to low-rank intermediate dimension The dimension-upgrading linear mapping layer moves features from a low-rank intermediate dimension. Mapping back to the original dimension ,in The feature dimension of the original projection matrix in the transformer layer. For the preset low-rank intermediate dimension and much smaller In a preferred embodiment, It is 1024. It is 4. Low-rank intermediate dimension. The significance of taking the minimum value is that the two-step linear mapping of dimensionality reduction followed by dimensionality increase is equivalent to applying a rank of to the original projection matrix. The incremental matrix correction only requires a limited number of trainable parameters. rather than the original projection matrix This significantly reduces the number of parameters that need to be trained. The output of the low-rank adaptive branch is element-wise added to the output of the original self-attention projection matrix in the same transformer layer, and the sum is used as the final output of the self-attention projection of the current transformer layer. The parameters of the original projection matrix remain frozen and do not participate in gradient updates; only the parameters of the dimensionality-reduced linear mapping layer and the dimensionality-increasing linear mapping layer in the low-rank adaptive branch receive gradient updates. At the start of fine-tuning training, the parameters of the dimensionality-reduced linear mapping layer are initialized with random Gaussian initialization, and all parameters of the dimensionality-increasing linear mapping layer are initialized to zero. This ensures that the output of the low-rank adaptive branch is a zero vector at the beginning of training, without interfering with the initial behavior of the original pre-trained model.
[0054] In an alternative implementation, the insertion position of the low-rank adaptive branch is not limited to the self-attention projection matrix, but can also be extended to the side of the linear mapping layer of the feedforward network in each transformer layer. Increasing the insertion position can enhance the domain adaptation capability, but correspondingly increases the number of trainable parameters. Low-rank intermediate dimension It can take integer values in the range of 2 to 16. The larger the value, the stronger the adaptability, but the risk of overfitting also increases accordingly.
[0055] Fine-tuning training was performed using the Shikusi (Stone Cave Temple) disease labeling dataset. Each sample in the disease labeling dataset contains one 4-channel fused feature map processed in step 2 and one corresponding pixel-level label mask. The label value of each pixel in the mask is an integer between 0 and 6, where 0 represents the background (i.e., a normal stone surface without disease), and 1 to 6 correspond to six types of disease: cracks, pulverization, flaking, salting out, contamination crust, and biofouling, respectively. During training, only the parameters of the channel-adaptive convolutional layers and all parameters in the low-rank adaptive branches participate in gradient updates; all other original pre-trained parameters of the general segmentation base model remain frozen. Training employs a stochastic gradient descent optimizer or an adaptive moment estimation optimizer, with an initial learning rate of 0.001, a batch size of 8, and 50 training iterations. The loss function is a weighted combination of cross-entropy loss and dice loss. In an optional implementation, the disease labeling dataset can also be augmented using data enhancement techniques such as random cropping, random horizontal flipping, random rotation, and color jittering to increase the number of effective training samples.
[0056] After fine-tuning the training, the inference phase begins. The 4-channel fused feature map generated in step 2 is mapped to a 3-channel input tensor through a channel-adaptive convolutional layer and then fed into the image encoder. The self-attention projection process of each transformer layer in the image encoder is jointly executed by the original pre-trained parameters and the fine-tuned low-rank adaptive parameters, extracting image features layer by layer, and finally outputting the image embedding feature. The image embedding feature is a 3D tensor with a spatial resolution lower than that of the original input image. In a preferred embodiment, when the input image size is 1024 pixels by 1024 pixels, the spatial resolution of the image embedding feature is 64 by 64, and the number of feature channels is 256.
[0057] The mask decoder needs to receive prompts to guide the generation of segmented regions. In automated disease identification scenarios, there is no opportunity for manual input of prompts; therefore, an automatic prompt generation strategy is adopted. In the spatial dimension of the image embedding features, uniform grid sampling is performed according to preset row and column spacing to generate a set of grid prompt points covering the entire image. In a preferred embodiment, both row and column spacing are set to one-eighth of the spatial size of the image embedding features, i.e., one sampling point is taken every 8 positions in a 64x64 spatial dimension, generating 64 grid prompt points (8x8). Each grid prompt point is treated as a foreground prompt point and input into the mask decoder. The mask decoder combines the image embedding features and each grid prompt point to generate a candidate disease region mask centered on that prompt point. When candidate masks generated from multiple prompt points overlap, a non-maximum suppression strategy is adopted: the cross-union ratio (CUI) between overlapping masks is calculated; when the CUI exceeds a preset threshold of 0.7, the candidate mask with the higher confidence score is retained, and the candidate mask with the lower confidence score is deleted. After non-maximum suppression, each remaining candidate disease region corresponds to a pixel-level segmentation mask. Pixels with a value of 1 in the mask indicate that they belong to the current disease region, while pixels with a value of 0 indicate that they do not belong to the current disease region.
[0058] In an alternative implementation, the spacing of the grid cue points can be set to a value ranging from one-quarter to one-sixteenth of the image embedding feature space size. A smaller spacing results in denser cue points and more comprehensive coverage, but also increases the computational cost. Another alternative approach is to employ a two-stage cueing strategy: the first stage uses a sparse grid to generate coarse candidate regions, and the second stage supplements each coarse candidate region with dense cue points at its center and boundaries to generate a fine-grained segmentation mask.
[0059] A lesion type classification header is added to the output of the mask decoder to determine the type of each segmented lesion region. For each lesion region corresponding to a pixel-level segmentation mask, the spatial resolution of the pixel-level segmentation mask is first adjusted to the same spatial size as the image embedding feature. Since the spatial resolution of the pixel-level segmentation mask is usually consistent with the original input image (1024 pixels by 1024 pixels), while the spatial resolution of the image embedding feature is 64 by 64, a spatial downsampling operation needs to be performed on the pixel-level segmentation mask. The downsampling method is nearest neighbor interpolation, reducing the mask size from 1024 pixels by 1024 pixels to 64 by 64. The downsampled mask and the image embedding feature have the same spatial grid correspondence; the positions with a value of 1 in the mask are the spatial locations where features need to be extracted. On the image embedding feature, feature vectors of all spatial locations within the coverage area of the downsampled mask are extracted. A global average pooling operation is performed on these feature vectors, that is, the arithmetic mean of the values of each feature channel at all spatial locations within the mask is calculated to obtain a region pooling feature vector with a dimension equal to the number of feature channels. Regional pooling feature vectors compress feature information of all spatial locations within the current disease area, forming a global representation of the current disease area.
[0060] The region pooling feature vector is input into the disease type classification head. The disease type classification head consists of a fully connected layer and a normalized exponential function layer concatenated. The fully connected layer maps the region pooling feature vector from the feature channel dimension to a 6-dimensional output vector, where each component of the 6-dimensional output vector corresponds to one disease type. The normalized exponential function layer converts each component in the 6-dimensional output vector into a probability value between 0 and 1, and the sum of all probability values equals 1. The category corresponding to the highest probability value is the disease type label for the current disease area. The six categories are, in order: fissure, pulverization, exfoliation, salting out, contamination crust, and biofouling.
[0061] After obtaining the pixel-level segmentation mask and disease type label for each diseased area, disease classification is performed. The classification process employs a differentiated judgment strategy based on the disease type. Based on the pixel-level segmentation mask for each diseased area, the ratio of the total number of pixels with a value of 1 in the mask to the total number of pixels in the entire image is calculated to obtain the area proportion. Simultaneously, the coordinates of the top-left and bottom-right corners of the smallest bounding rectangle of the pixels with a value of 1 in the mask are calculated in the image coordinate system and used as spatial distribution coordinates to mark the geographical location of the diseased area on the surface of the grotto temple.
[0062] For disease areas labeled as "cracks," the area percentage is insufficient to fully characterize their severity, as the hazard of cracks largely depends on their length and connectivity: a single, long, penetrating crack may pose a greater structural safety hazard than a localized, sheet-like spalling area of equal area. Therefore, morphological parameters are introduced as auxiliary classification indicators for crack-type disease areas. Morphological skeletonization is performed on the pixel-level segmentation mask of the crack area. Morphological skeletonization is an iterative refinement process that gradually removes edge pixels from binary regions until each connected region is refined to a single-pixel-width centerline, while maintaining the original region's topological connectivity. The extracted crack centerlines reflect the crack's direction, branching, and intersection structures. The total number of pixels in the crack centerline is counted, and multiplied by the corresponding physical size of each pixel to obtain the total length of the crack centerline. Connectivity component analysis is performed on the crack centerline to identify the number of pixels contained in the largest connected component, which is then converted into a physical length to obtain the maximum connected length. The maximum connected length reflects the maximum extension range of a single, uninterrupted crack in the crack network and is a key indicator for assessing the risk of crack penetration. The severity of the crack is determined by comparing its area percentage and morphological parameters with corresponding crack classification thresholds: when the area percentage exceeds a preset severe area threshold or the maximum connectivity length exceeds a preset severe length threshold, the current crack area is marked as severely damaged; when both the area percentage and the maximum connectivity length are below the severe threshold but at least one exceeds the moderate threshold, it is marked as moderately damaged; when both the area percentage and the maximum connectivity length are below the moderate threshold, it is marked as lightly damaged. In a preferred embodiment, the severe area threshold is set to 5%, the moderate area threshold is set to 1%, the severe length threshold is set to 500 mm, and the moderate length threshold is set to 100 mm. These thresholds can be adjusted according to the specific cultural relic protection requirements of the grotto temple and industry technical specifications.
[0063] For areas tagged with five disease types—pulverization, spalling, salting out, contamination crusting, or biofouling—the severity is primarily related to the area affected. These diseases are typically distributed in patches or clumps, lacking the linear extension characteristic of cracks; therefore, only the area percentage is used for grading. The disease level is determined by comparing the area percentage with the corresponding grading threshold: a percentage exceeding a preset severe threshold is marked as severe disease; a percentage between the moderate and severe thresholds is marked as moderate disease; and a percentage below the moderate threshold is marked as mild disease. Different grading thresholds can be set for different types of diseases. For example, spalling diseases, which directly lead to the loss of the stone matrix, can have their severe threshold set lower than that of contamination crusting, reflecting a higher level of vigilance for material loss-related diseases. In a preferred embodiment, the threshold for severe area of peeling is set to 3%, and the threshold for moderate area is set to 0.5%; the threshold for severe area of salting out is set to 8%, and the threshold for moderate area is set to 2%; the threshold for severe area of pulverization is set to 10%, and the threshold for moderate area is set to 3%; the threshold for severe area of contamination crust is set to 15%, and the threshold for moderate area is set to 5%; and the threshold for severe area of bioattachment is set to 10%, and the threshold for moderate area is set to 3%.
[0064] The pixel-level segmentation mask, disease type label, area proportion, spatial distribution coordinates, and disease level of all diseased areas are integrated to generate a disease distribution map and disease classification results for the grotto temple surface. The disease distribution map is a pseudo-color labeled image with the same size as the original input image, where different colors represent different disease types, and the boundary of each colored area is determined by the outline of the pixel-level segmentation mask. The disease classification results are output in the form of a structured data table, where each row corresponds to an independent diseased area, recording its disease type label, area proportion, spatial distribution coordinates, morphological parameters (only crack-type diseases include this item), and disease level. In an optional implementation, the disease distribution map can be overlaid on a visible light registered image for visualization with transparency blending, allowing operators to intuitively compare the disease markings with the actual surface morphology.
[0065] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these specific embodiments are merely illustrative. Those skilled in the art can omit, substitute, and modify the details of the above methods and systems in various ways without departing from the principles and essence of the present invention. For example, combining the above method steps to perform substantially the same function and achieve substantially the same result according to substantially the same method falls within the scope of the present invention. Therefore, the scope of the present invention is defined only by the appended claims.
Claims
1. An automatic identification method for surface defects in grotto temples based on multimodal imaging and visual segmentation, characterized in that, Includes the following steps: Step 1: The incident beam is split into four channels—visible light, near-infrared, mid-wave infrared, and ultraviolet fluorescence—using a four-channel beam splitter prism group. An area array sensor is installed at the end of each channel and four original images are simultaneously exposed to obtain the images. After distortion correction and cross-modal space registration, a pixel-level aligned four-modal registered image group is obtained. Step 2: Transform the four registered images to the frequency domain and divide them into multiple concentric ring frequency bands. Calculate the normalized frequency band energy of each mode in each concentric ring frequency band. Mark the mode with the largest normalized frequency band energy in each concentric ring frequency band as the active mode. In the frequency domain of each image, only retain the frequency band component of the current mode that is the active mode. After inverse transformation to restore it to the spatial domain, stitch it along the channel dimension to generate a 4-channel fused feature map. Step 3: Insert a low-rank adaptive module into the image encoder of the general segmentation base model, freeze the original pre-training parameters and fine-tune the training of the low-rank adaptive module, input the 4-channel fused feature map into the image encoder to obtain image embedding features, output a pixel-level segmentation mask by the mask decoder, output a disease type label by the disease type classification head, determine the disease level based on the pixel-level segmentation mask and disease type label, and generate a disease distribution map and disease classification results on the surface of the grotto temple.
2. The method according to claim 1, characterized in that, In step 1, the four-channel beam splitter assembly includes a first dichroic beam splitter, a second dichroic beam splitter, and a third dichroic beam splitter arranged sequentially along the optical path propagation direction. The first dichroic beam splitter splits the incident beam into a transmitted visible light beam and a reflected non-visible light beam. The second dichroic beam splitter splits the non-visible light beam into a transmitted near-infrared beam and a reflected mixed ultraviolet and mid-infrared beam. The third dichroic beam splitter splits the mixed ultraviolet and mid-infrared beam into a transmitted mid-infrared beam and a reflected ultraviolet beam.
3. The method according to claim 1, characterized in that, In step 1, an ultraviolet excitation light source is set around the acquisition lens to illuminate the surface of the grotto temple target with ultraviolet excitation. An excitation cutoff filter is set in the ultraviolet fluorescence channel to block the direct transmission of ultraviolet excitation light, allowing only the fluorescence emission signal generated by the target surface to enter the area array sensor at the end of the ultraviolet fluorescence channel.
4. The method according to claim 1, characterized in that, In step 1, a visible light area array sensor is installed at the end of the visible light channel, a near-infrared area array sensor is installed at the end of the near-infrared channel, a mid-wave infrared area array thermal imaging sensor is installed at the end of the mid-wave infrared channel, and an ultraviolet-enhanced area array sensor is installed at the end of the ultraviolet fluorescence channel; synchronous exposure trigger pulses are sent to the four area array sensors simultaneously through a field-programmable gate array.
5. The method according to claim 1, characterized in that, In step 1, the cross-modal space registration method is as follows: the lens distortion correction of the four original images is performed by using the pre-calibrated intrinsic parameters and distortion coefficients of each area array sensor. The distortion-corrected visible light image is used as the reference image. Based on the pre-calibrated extrinsic parameter mapping relationship between each modal sensor, the distortion-corrected near-infrared image, mid-wave infrared image and ultraviolet fluorescence image are aligned to the pixel coordinate system of the visible light image by coordinate reprojection transformation.
6. The method according to claim 1, characterized in that, In step 2, the transformation to the frequency domain is performed as follows: a 2D Fast Fourier Transform is performed on each registered image to obtain a frequency domain complex matrix. The modulus of each element in the frequency domain complex matrix is taken to obtain the frequency domain amplitude spectrum. The concentric ring frequency bands are divided as follows: the geometric center of the frequency domain amplitude spectrum is taken as the center, and the distance from the center to the short edge of the frequency domain amplitude spectrum is taken as the maximum radius. The maximum radius is divided into N equal segments, and concentric circles are drawn with the endpoints of each segment as the radius. The N concentric ring frequency bands are numbered sequentially from the center outwards as the first ring band to the Nth ring band. The normalized frequency band energy is calculated as follows: the amplitude of all values in each concentric ring frequency band region is accumulated to obtain the frequency band energy value. The total energy of the current mode is obtained by summing the energy values of all N frequency bands of the same mode. The normalized frequency band energy is obtained by dividing each frequency band energy value by the total energy of the current mode.
7. The method according to claim 6, characterized in that, In step 2, after marking the active modes of all concentric ring frequency bands, a frequency band activation mapping table is generated. The frequency band activation mapping table contains N records, and each record stores one concentric ring frequency band number and the corresponding active mode identifier. Based on the frequency band activation mapping table, each frequency domain complex matrix is checked one by one from the 1st ring band to the Nth ring band. If the activation mode identifier of the current ring band is the same as the mode corresponding to the current frequency domain complex matrix, all complex coefficients in the current ring band are retained; otherwise, all complex coefficients in the current ring band are set to zero.
8. The method according to claim 1, characterized in that, In step 3, a channel-adaptive convolutional layer is set at the input of the image encoder. The channel-adaptive convolutional layer has 4 input channels and the number of output channels is the same as the number of input channels in the original pre-training of the image encoder. Low-rank adaptive modules are inserted in parallel next to the self-attention projection matrix of each transformer layer inside the image encoder. Each low-rank adaptive module is composed of a cascaded dimension-reduced linear mapping layer and a dimension-upgraded linear mapping layer. The output of the low-rank adaptive module is added element-wise to the output of the original self-attention projection matrix to obtain the final output of the current transformer layer. The parameters of the channel-adaptive convolutional layer and the parameters of all low-rank adaptive modules are set to a trainable state.
9. The method according to claim 1, characterized in that, In step 3, uniform grid sampling is performed on the spatial dimension of the image embedding features according to preset row and column spacing to generate a set of grid cue points. The mask decoder combines the image embedding features and the set of grid cue points to output a pixel-level segmentation mask for each candidate disease region. For each disease region, the corresponding pixel-level segmentation mask is adjusted to the same spatial resolution as the image embedding features through spatial downsampling. Feature vectors of all spatial locations within the coverage area of the downsampled pixel-level segmentation mask are extracted from the image embedding features and global average pooling is performed to obtain a region pooling feature vector. The region pooling feature vector is input into the disease type classification head to output the disease type label.
10. The method according to claim 1, characterized in that, In step 3, the disease level is determined as follows: For disease areas labeled as cracks, morphological skeletonization is performed on the corresponding pixel-level segmentation mask to extract the crack centerline. The total length and maximum connectivity length of the crack centerline are counted as morphological parameters. The disease level is determined by comparing the area ratio and morphological parameters with the corresponding crack grading threshold. For disease areas labeled as powdering, peeling, salting out, contamination crusting, or biological attachment, the disease level is determined by comparing the area ratio with the grading threshold of the corresponding type.
Citation Information
Patent Citations
Crack detection method, device and equipment
CN116188348A
Unmovable cultural relic damage identification and early warning system
CN122200146A
