Method, device and equipment for automatically extracting cracking form of ancient ceramic glaze and medium

Abstract

The invention relates to an automatic extraction method, device and equipment for an ancient ceramic glaze splitting form and a medium. The method comprises the following steps: acquiring a hyperspectral image cube and three-dimensional point cloud data of an ancient ceramic glaze, and performing spatial alignment processing on the hyperspectral image cube and the three-dimensional point cloud data to obtain a hyperspectral data block and a three-dimensional point cloud data block; inputting the hyperspectral data block and the three-dimensional point cloud data block into a pre-trained double-flow coding cross-modal fusion model to obtain an initial three-dimensional point cloud slicing segmentation prediction result and a two-dimensional chemical anomaly confidence prediction map; on the basis of the two-dimensional chemical anomaly confidence prediction map, arbitration optimization is carried out on prediction points with fuzzy or isolated boundaries in the initial three-dimensional point cloud slicing segmentation prediction result, and three-dimensional point cloud optimization segmentation labels are obtained; and performing three-dimensional space smoothing processing on the three-dimensional point cloud optimization segmentation label to obtain an ancient ceramic glaze three-dimensional cracking network form. By adopting the method, the end-to-end automatic extraction of the three-dimensional slicing network form can be refined.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and in particular relates to an automatic extraction method, device, equipment and medium for the crackle pattern of ancient ceramic glaze. Background Technology

[0002] Crackles on the glaze of ancient ceramics are a network of microscopic cracks that are generated inside the ceramic glaze layer due to stress release. Their morphological characteristics and the filling or precipitation of substances within the cracks are key scientific evidence for identifying the age, craftsmanship, authenticity, and preservation status of cultural relics.

[0003] In traditional techniques, the research and detection of glaze cracks on ancient ceramics mainly employs methods such as microscopic observation, two-dimensional image analysis, or single-modal data acquisition to conduct chemical composition analysis and morphological structure analysis of glaze cracks. The identification and segmentation of glaze crack networks mainly rely on the experience judgment of professionals, and crack areas with fine cracks and low contrast between the filling material and the glaze background color are identified and segmented manually.

[0004] However, the information obtained by the above methods is fragmented. Due to differences in instruments, time, and sampling points, it is difficult to establish a precise correspondence between chemical information and geometric structure information in the same microscopic area. Crack network identification and segmentation based on expert experience is not only inefficient when dealing with fine cracks and low-contrast crazing areas, but also has poor repeatability of test results, making it difficult to meet the scientific needs of identifying and studying crazing on ancient ceramic glazes. Summary of the Invention

[0005] Therefore, it is necessary to provide an automatic extraction method, device, equipment, and medium for ancient ceramic glaze crack patterns that can accurately integrate hyperspectral chemical information and three-dimensional microscopic morphology information to address the aforementioned technical problems.

[0006] Firstly, this application provides an automatic method for extracting the crazing patterns of ancient ceramic glazes, including:

[0007] The hyperspectral image cube and 3D point cloud data of the glaze surface of ancient ceramics were acquired, and the hyperspectral image cube and 3D point cloud data were spatially aligned to obtain hyperspectral data blocks and 3D point cloud data blocks.

[0008] The hyperspectral data block and the 3D point cloud data block are input into a pre-trained dual-stream coded cross-modal fusion model to obtain the initial 3D point cloud slicing prediction results and the 2D chemical anomaly confidence prediction map.

[0009] Based on the two-dimensional chemical anomaly confidence prediction map, arbitration optimization is performed on the prediction points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results to obtain the three-dimensional point cloud optimized segmentation label.

[0010] The three-dimensional point cloud segmentation labels are optimized and smoothed in three-dimensional space to obtain the three-dimensional crackle network morphology of ancient ceramic glaze.

[0011] In one embodiment, the dual-stream coded cross-modal fusion model obtains the initial 3D point cloud patching prediction results and the 2D chemical anomaly confidence prediction map through the following methods:

[0012] Multi-scale spectral features of hyperspectral data blocks are extracted by a spectral flow 3D convolutional encoder, and multi-scale geometric features of 3D point cloud data blocks are extracted by a geometric flow point cloud network encoder.

[0013] By using a bidirectional cross-modal attention mechanism, the multi-scale spectral features and multi-scale geometric features of each encoder layer are fused at the feature level to obtain the geometric features and spectral features of each encoder layer that are enhanced by chemical information and guided by geometric information.

[0014] The geometric decoder is used to upsample and decode the chemically enhanced geometric features of all encoder layers to obtain the initial 3D point cloud slicing prediction results.

[0015] Using a chemical decoder, the spectral features of all encoder layers enhanced by geometric information are upsampled and decoded to obtain a two-dimensional chemical anomaly confidence prediction map.

[0016] In one embodiment, a bidirectional cross-modal attention mechanism is used to perform feature-level fusion of multi-scale spectral features and multi-scale geometric features of each encoder layer, resulting in chemically enhanced geometric features and geometrically enhanced spectral features of each encoder layer, including:

[0017] Channel adjustment is performed on the multi-scale spectral features to obtain a spectral feature map, and then the spectral feature map is spatially projected to obtain a spectral feature vector;

[0018] Using the spectral feature vector as the query vector and the multi-scale geometric features of each 3D point as the key vector and value vector, the first attention weight is calculated, and the value vectors of the multi-scale geometric features are weighted and summed based on the first attention weight to obtain the attention features of each 3D point.

[0019] The attention features of each 3D point are added element-wise with the multi-scale geometric features, and then processed by a fully connected layer and a nonlinear activation function to generate the chemically enhanced geometric features of each 3D point in the current encoder layer.

[0020] The geometric features of all 3D points in the current encoder layer, enhanced by chemical information, are projected onto each pixel of the spectral feature map through an inverse distance weighted average to obtain the geometric feature vector corresponding to each pixel.

[0021] Using the geometric feature vector as the query vector and the spectral feature vector of each pixel as the key vector and value vector, the second attention weight is calculated, and the value vector of the spectral feature vector is weighted and summed using the second attention weight to obtain the attention features of each pixel;

[0022] The attention features of each pixel are added element-wise with the multi-scale spectral features, and then processed by a fully connected layer and a nonlinear activation function to generate the spectral features of each pixel in the current encoder layer that are enhanced by geometric information.

[0023] In one embodiment, the two-stream coded cross-modal fusion model is obtained through end-to-end training using a joint loss function; the joint loss function is constructed using the following method:

[0024] Based on the predicted probability and the real point-level geometric labels of the initial 3D point cloud patch segmentation results, the classification loss is calculated through the focal loss function, and the geometric segmentation loss is obtained by combining the Dice loss between the predicted segmentation region and the real region of the initial 3D point cloud patch segmentation results.

[0025] Based on the two-dimensional chemical anomaly confidence prediction map and the real pixel-level chemical anomaly labels, the chemical anomaly detection loss is calculated using the binary cross-entropy loss function.

[0026] Based on the spatial correspondence between the hyperspectral data block and the 3D point cloud data block, the predicted probability of the crack category in the initial 3D point cloud flaking segmentation result is projected onto the 2D image plane to obtain the geometric segmentation probability projection map.

[0027] The confidence prediction map of two-dimensional chemical anomalies is transformed by a convolutional neural network, and the L2 norm loss of the two-dimensional confidence prediction map of chemical anomalies after feature transformation and the geometric segmentation probability projection map are calculated to obtain the cooperative consistency loss.

[0028] The geometric segmentation loss, chemical anomaly detection loss, and cooperative consistency loss are linearly combined according to preset weights to obtain the joint loss function.

[0029] In one embodiment, channel adjustment is performed on the multi-scale spectral features to obtain a spectral feature map, and the spectral feature map is spatially projected to obtain a spectral feature vector, including:

[0030] Based on each encoder layer, channel transformation of multi-scale spectral features is performed through three-dimensional convolution to obtain a spectral feature map with the same number of channels as the multi-scale geometric features of the same encoder layer.

[0031] Based on the spatial correspondence between hyperspectral data blocks and 3D point cloud data blocks, the spatial coordinates of all 3D points in the 3D point cloud data block are mapped to the 2D image coordinate system where the spectral feature map is located through affine transformation, so as to obtain the 2D coordinates corresponding to each 3D point.

[0032] Based on the two-dimensional coordinates corresponding to each three-dimensional point, four pixel grid points adjacent to the two-dimensional coordinates are found on the spectral feature map. Bilinear interpolation is then performed based on the relative positions of the two-dimensional coordinates and the four pixel grid points to obtain the spectral feature vector of each three-dimensional point.

[0033] In one embodiment, the chemically enhanced geometric features of all 3D points in the current encoder layer are projected onto each pixel of the spectral feature map using an inverse distance-weighted average to obtain the geometric feature vector corresponding to each pixel, including:

[0034] The position coordinates of each pixel on the spectral feature map in the two-dimensional image coordinate system are reversed and mapped back to the three-dimensional space to obtain the three-dimensional reference point corresponding to each pixel.

[0035] Divide the three-dimensional spherical space neighborhood with the three-dimensional reference point as the center, and find all three-dimensional points belonging to the current encoder layer within the three-dimensional spherical space neighborhood as neighboring points;

[0036] Calculate the Euclidean distance from each neighboring point to the 3D reference point, and calculate the inverse distance weight of each neighboring point based on the Euclidean distance;

[0037] The weighted features are obtained by multiplying the enhanced geometric features of each neighboring point guided by chemical information with the inverse distance weights. The weighted features of all neighboring points are then fused to obtain the geometric feature vector of the corresponding pixel.

[0038] In one embodiment, based on a two-dimensional chemical anomaly confidence prediction map, arbitration optimization is performed on predicted points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results to obtain optimized three-dimensional point cloud segmentation labels, including:

[0039] Based on the spatial correspondence between hyperspectral data blocks and 3D point cloud data blocks, the two-dimensional chemical anomaly confidence prediction map is mapped onto the 3D point cloud of the 3D point cloud block through bilinear interpolation to obtain the chemical anomaly confidence value of each 3D point.

[0040] Select the three-dimensional points that belong to the crack category in the initial three-dimensional point cloud flaking segmentation prediction results and whose prediction probabilities are between the preset lower threshold and the preset upper threshold to obtain the boundary fuzzy point set;

[0041] For each three-dimensional point in the boundary ambiguity point set, determine whether the chemical anomaly confidence value of the three-dimensional point is greater than the preset chemical arbitration threshold. If the chemical anomaly confidence value is greater than the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as a crack category. If the chemical anomaly confidence value is less than or equal to the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as a non-crack category.

[0042] All three-dimensional points whose predicted probability is lower than the preset lower threshold and whose chemical anomaly confidence value is higher than the average value of the background area are identified as isolated point groups.

[0043] If the area of ​​the spatially connected region of an isolated point group is greater than the preset minimum area threshold, the final label of the isolated point group will be arbitrated as a crack category.

[0044] The final labels of all 3D points are used to obtain the optimized segmentation labels for the 3D point cloud.

[0045] Secondly, this application also provides an automatic extraction device for the crazing patterns of ancient ceramic glazes, comprising:

[0046] The data module is used to acquire hyperspectral image cubes and 3D point cloud data of ancient ceramic glazes, and to perform spatial alignment processing on the hyperspectral image cubes and 3D point cloud data to obtain hyperspectral data blocks and 3D point cloud data blocks.

[0047] The prediction module is used to input hyperspectral data blocks and 3D point cloud data blocks into a pre-trained dual-stream coded cross-modal fusion model to obtain the initial 3D point cloud slicing prediction results and the 2D chemical anomaly confidence prediction map.

[0048] The optimization module is used to arbitrate and optimize the prediction points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results based on the two-dimensional chemical anomaly confidence prediction map, so as to obtain the three-dimensional point cloud optimized segmentation label.

[0049] The crack morphology module is used to perform three-dimensional spatial smoothing on the optimized segmentation labels of the three-dimensional point cloud to obtain the three-dimensional crack network morphology of the ancient ceramic glaze.

[0050] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-mentioned automatic extraction method for the crackle morphology of ancient ceramic glaze.

[0051] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the above-mentioned automatic extraction method for the crazing patterns of any of the ancient ceramic glazes.

[0052] The aforementioned automatic extraction method, apparatus, equipment, and medium for the crazing morphology of ancient ceramic glazes achieves this by spatially aligning hyperspectral image cubes and 3D point cloud data of the ancient ceramic glaze surface to obtain registered multimodal data blocks. These blocks are then input into a pre-trained dual-stream coded cross-modal fusion model. This model deeply extracts and fuses chemical spectral features and 3D geometric features to generate preliminary crazing segmentation results and chemical anomaly maps. Chemical anomaly confidence information is used to intelligently arbitrate and optimize geometric prediction points with blurred or isolated boundaries in the initial segmentation. The optimized segmentation labels undergo 3D spatial smoothing, thus achieving end-to-end automatic extraction from heterogeneous cross-scale data to refined 3D crazing network morphology. This effectively solves the problem of deep fusion between chemical fingerprints and microscopic morphology information, significantly improving the accuracy and robustness of segmenting complex and weak crazing, and simultaneously providing chemical composition anomaly information in the crazing region. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the automatic extraction method for the crazing patterns of ancient ceramic glazes according to the present invention.

[0055] Figure 2 This is a structural diagram of the automatic extraction device for the crackle morphology of ancient ceramic glaze according to the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] In one embodiment, such as Figure 1 As shown, an automatic extraction method for the crazing patterns of ancient ceramic glazes is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] S101. Obtain the hyperspectral image cube and three-dimensional point cloud data of the ancient ceramic glaze, and perform spatial alignment processing on the hyperspectral image cube and three-dimensional point cloud data to obtain hyperspectral data blocks and three-dimensional point cloud data blocks.

[0059] In a schematic manner, for the ancient ceramic glaze sample to be analyzed, a region containing typical crackle networks was selected as the target analysis area. This area was scanned and acquired using a hyperspectral imaging device, resulting in a hyperspectral image cube. This data contains comprehensive information in both spatial and spectral dimensions, reflecting the chemical composition characteristics of different locations on the glaze surface. Simultaneously, a three-dimensional surface scanning device was used to scan the same target analysis area, acquiring three-dimensional point cloud data. This data records the spatial location information and local surface morphology features of each sampling point on the glaze surface. After data acquisition, spatial alignment processing was performed to establish the spatial correspondence between the two datasets. Specifically, the hyperspectral image cube and the three-dimensional point cloud data were initially correlated in coordinate systems. Common stable features in both datasets were identified. Stable features possess the characteristics of fixed spatial location and high recognizability. An initial rigid transformation matrix was calculated using these stable features. This matrix describes the rotation and translation relationship between the hyperspectral image coordinate system and the three-dimensional point cloud coordinate system. ,in This is a rotation matrix used to represent the rotational angular relationship between two coordinate systems. The translation vector is used to represent the positional offset between two coordinate systems. The zero vector, superscript This represents the transpose operation of a matrix. This initial rigid transformation matrix achieves a preliminary mapping from the hyperspectral image coordinate system to the 3D point cloud coordinate system. Based on this initial mapping, a data cropping operation is performed: the boundary of the target analysis region in the hyperspectral image is mapped to the 3D point cloud coordinate system using the initial rigid transformation matrix. The point cloud region enclosed by this mapped boundary is then cropped from the 3D point cloud, and the corresponding target analysis region image cube is cropped from the hyperspectral image. To ensure consistency in subsequent processing, the spatial resolution of the cropped hyperspectral data and 3D point cloud data is adjusted using an interpolation algorithm to ensure that the spatial resolution of the hyperspectral data matches the density of the 3D point cloud data on the 2D projection plane. This results in spatially preliminarily aligned hyperspectral data blocks and 3D point cloud data blocks.

[0060] S102. Input the hyperspectral data block and the 3D point cloud data block into the pre-trained dual-stream coded cross-modal fusion model to obtain the initial 3D point cloud slicing prediction results and the 2D chemical anomaly confidence prediction map.

[0061] Furthermore, the obtained hyperspectral data blocks and 3D point cloud data blocks undergo preprocessing operations, including data augmentation and normalization, to improve the robustness and generalization ability of the model. After preprocessing, the two types of data blocks are input into a pre-trained dual-stream coding cross-modal fusion model. This model can simultaneously process the spectral features of hyperspectral data and the geometric features of 3D point cloud data, and achieve complementarity and synergy between the two features through a cross-modal fusion mechanism. Specifically, the model extracts features from the hyperspectral data blocks and 3D point cloud data blocks through the encoding module, performs deep fusion of the extracted multi-scale features, and then outputs the initial 3D point cloud sectional segmentation prediction results and the 2D chemical anomaly confidence prediction map through the decoding module. The initial 3D point cloud sectional segmentation prediction results reflect the probability distribution of each point in the 3D point cloud belonging to the sectional-related category; the value of each position in the 2D chemical anomaly confidence prediction map is used to characterize the probability of the presence of chemical anomalies at that position. Chemical anomalies are directly related to the filling material, precipitates, etc., in the sectional area.

[0062] S103. Based on the two-dimensional chemical anomaly confidence prediction map, arbitration optimization is performed on the prediction points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results to obtain the three-dimensional point cloud optimized segmentation label.

[0063] In the initial 3D point cloud crazing segmentation prediction results, some predicted points exhibit uncertain category classifications, i.e., predicted points with blurred boundaries. The category probability distribution of these points is relatively uniform, making direct classification difficult. Simultaneously, some isolated predicted points lack effective spatial connectivity with other predicted points, and the authenticity of these points requires further verification. Optionally, the high specificity of the 2D chemical anomaly confidence prediction map can be used to perform arbitration optimization on these two types of predicted points. Specifically, a spatial mapping relationship is established between the 2D chemical anomaly confidence prediction map and the 3D point cloud data, ensuring that each 3D predicted point corresponds to a specific location in the 2D chemical anomaly confidence prediction map, thereby obtaining the chemical anomaly confidence score for each 3D predicted point. Based on the characteristics of chemical anomaly confidence scores, crazing areas typically correspond to higher chemical anomaly confidence scores, while non-crazing background glaze areas correspond to lower chemical anomaly confidence scores. Based on this pattern, for prediction points with ambiguous boundaries, their true category is determined according to their corresponding chemical anomaly confidence level; for isolated prediction points, their true category is verified by combining chemical anomaly confidence level and spatial distribution characteristics. Through the above arbitration process, misjudged points in the initial segmentation prediction results are corrected, and optimized segmentation labels for 3D point clouds are obtained.

[0064] S104. Perform three-dimensional spatial smoothing on the optimized segmentation labels of the three-dimensional point cloud to obtain the three-dimensional crackling network morphology of the ancient ceramic glaze.

[0065] Optionally, while optimized 3D point cloud segmentation labels correct some misclassified points, noise interference during data acquisition or the complexity of local features may lead to discontinuous label distribution, affecting the integrity and accuracy of the crackle network morphology. Therefore, 3D spatial smoothing can be applied to the optimized 3D point cloud segmentation labels. This process prioritizes the consistency of features between adjacent points in 3D space, considering spatial distances between 3D points, differences in local surface normal vectors, and the correlation of segmentation labels. For example, a spatial smoothing algorithm optimizes and adjusts the segmentation label for each 3D point, ensuring that adjacent points with similar local features have consistent segmentation labels, eliminating isolated abnormal labels, and filling in minor gaps in the label distribution. After smoothing, all points labeled as crackle-related categories will form a continuous and complete 3D network structure. This structure accurately reflects the 3D morphological characteristics of ancient ceramic glaze cracks, including the direction, spatial distribution, and interconnections of the cracks, ultimately yielding the 3D crackle network morphology of ancient ceramic glazes.

[0066] In the aforementioned automatic extraction method for the crazing patterns of ancient ceramic glazes, hyperspectral image cubes and 3D point cloud data of the ancient ceramic glaze are acquired and spatially aligned to obtain matching hyperspectral data blocks and 3D point cloud data blocks. These data blocks are then input into a pre-trained dual-stream coding cross-modal fusion model, which simultaneously outputs initial 3D point cloud crazing segmentation prediction results and a 2D chemical anomaly confidence prediction map. The 2D chemical anomaly confidence prediction map is used to arbitrate and optimize the prediction points with blurred or isolated boundaries in the initial segmentation results to obtain accurate 3D point cloud optimized segmentation labels. These optimized segmentation labels are then smoothed in 3D space, thus achieving automatic extraction of the 3D crazing network morphology of ancient ceramic glazes. This method effectively solves the problems of information fragmentation and insufficient accuracy and repeatability caused by reliance on human experience in traditional methods, improving the accuracy and robustness of crazing pattern extraction, and enhancing the automation and standardization of the analysis process. It provides reliable technical support for the dating and technological research related to ancient ceramic crazing.

[0067] In one embodiment, the dual-stream coded cross-modal fusion model obtains the initial 3D point cloud patching prediction results and the 2D chemical anomaly confidence prediction map through the following methods:

[0068] S11. Multi-scale spectral features of hyperspectral data blocks are extracted by a spectral flow 3D convolutional encoder, and multi-scale geometric features of 3D point cloud data blocks are extracted by a geometric flow point cloud network encoder.

[0069] Indicatively, the spectral flow 3D convolutional encoder takes preprocessed hyperspectral data blocks as input. This encoder employs a hierarchical 3D convolutional neural network structure, progressively extracting multi-scale spectral features through multiple downsampling stages. Each downsampling stage includes a 3D convolutional layer, a batch normalization layer, an activation function layer, and a 3D max-pooling layer. The 3D convolutional layer, by setting a convolutional kernel of a specific size, simultaneously captures neighborhood information from both the spatial and spectral dimensions of the hyperspectral data, achieving joint extraction of spectral and spatial features. The batch normalization layer standardizes the feature map output by the convolution, reducing the impact of data distribution differences on feature extraction and accelerating model training convergence. The activation function layer uses a non-linear activation function to introduce non-linear expressive power of features, enhancing the model's ability to fit complex features. The 3D max-pooling layer, by setting a pooling kernel of a specific size, performs downsampling operations on the feature map, reducing the spatial resolution and computational cost of the feature map while preserving key feature information. After a series of downsampling phases, the hyperspectral data blocks are gradually encoded into a series of spectral feature maps with different spatial resolutions, forming multi-scale spectral features. These features contain feature information from low to high levels. Low-level features focus on local spectral differences in detail, while high-level features reflect the abstract global spectral distribution patterns.

[0070] Similarly, the geometric flow point cloud network encoder takes preprocessed 3D point cloud data blocks as input. Based on a hierarchical point cloud processing architecture, this encoder performs hierarchical downsampling of the input point cloud using a farthest point sampling algorithm. Each downsampling process selects a subset of points from the current point set that can uniformly cover the entire point cloud space, ensuring the representativeness of the downsampled subset. For each downsampled subset, a ball query or K-nearest neighbor query algorithm is used to obtain the local neighborhood point set of each sub-point. The features of the neighborhood point set are then fused and calculated through a local feature aggregation module. During the local feature aggregation process, multi-scale geometric context information is explicitly introduced. Point features within different radius neighborhoods are aggregated for each center point, and local features of different scales are concatenated to form a local feature vector containing multi-scale geometric information. Through multiple rounds of downsampling and local feature aggregation operations, the 3D point cloud data block is encoded into a series of downsampled subsets and their corresponding multi-scale geometric features. Low-level features reflect local surface details of the point cloud, such as normal vector changes and local curvature, while high-level features characterize the global geometric structure of the point cloud, such as the overall direction of cracks and regional morphology.

[0071] S12. By using a bidirectional cross-modal attention mechanism, the multi-scale spectral features and multi-scale geometric features of each encoder layer are fused at the feature level to obtain the geometric features and spectral features of each encoder layer that are enhanced by chemical information and guided by geometric information.

[0072] Furthermore, for the multi-scale spectral and geometric features output from each encoder level, a bidirectional cross-modal attention mechanism is employed for feature-level deep fusion. The core of this mechanism lies in establishing a dynamic correlation between the two modal features, enabling mutual guidance and enhancement of information. For example, for each encoder level, the spectral and geometric features of that level undergo feature adaptation processing to ensure that the two features are compatible in terms of channel dimension and feature scale. Through a chemistry-to-geometric attention transfer process, the chemical information contained in the spectral features is used to specifically enhance the geometric features, making them more focused on the geometric structures corresponding to regions of chemical composition anomalies. Simultaneously, through a geometry-to-chemistry attention transfer process, the morphological information contained in the geometric features is used to optimize and enhance the spectral features, making them more focused on the spectral differences in regions of abrupt geometric morphological changes. Through bidirectional attention interaction, the complementarity and synergy of the two modal features are achieved, ultimately outputting geometric features enhanced by chemical information and spectral features enhanced by geometric information at each encoder level.

[0073] S13. The geometric decoder is used to upsample and decode the enhanced geometric features of all encoder layers guided by chemical information to obtain the initial 3D point cloud segmentation prediction results.

[0074] Optionally, the geometric decoder adopts a hierarchical structure symmetrical to the geometric flow encoder. It gradually recovers the number of points in the point cloud through multi-stage upsampling operations, ultimately matching the number of points in the input 3D point cloud data block. During upsampling, a distance-based interpolation algorithm or a combination of transposed convolution and nearest-neighbor upsampling is used. For each subset of points in each upsampling stage, the enhanced geometric features of the lower level are interpolated and propagated to the current level's point set based on their spatial coordinates and the spatial correspondence with the lower-level point set, achieving cross-scale feature transfer. Simultaneously, a skip connection mechanism is introduced to directly transfer the chemically enhanced geometric features output from each encoder level to the corresponding level of the decoder, where they are concatenated and fused with the upsampled features to supplement lower-level detailed feature information and avoid feature loss during decoding.

[0075] After feature fusion at each decoding level is completed, the fused features of each point are mapped and transformed through a fully connected layer, converting the feature vector into a probability distribution vector for the corresponding category. The dimension of this vector is consistent with the preset number of geometric segmentation categories, and the value of each dimension represents the probability that the point belongs to the corresponding category. The division of geometric segmentation categories is set according to the crack morphology features, covering core categories such as background glaze, crack interior, and crack edge fracture area. The final output of the initial 3D point cloud crack segmentation prediction result is the set of category probability distributions corresponding to each point.

[0076] S14. Using a chemical decoder, the spectral features of all encoder layers that are enhanced by geometric information are upsampled and decoded to obtain a two-dimensional chemical anomaly confidence prediction map.

[0077] Optionally, the chemical decoder employs a 3D transposed convolutional network structure, which gradually recovers the spatial dimensions of spectral features through multi-stage upsampling operations, ultimately maintaining consistency with the spatial dimensions of the input hyperspectral data block. Each upsampling stage consists of a 3D transposed convolutional layer, a batch normalization layer, and an activation function layer. The 3D transposed convolutional layer performs deconvolution operations on the input enhanced spectral feature map by setting a specific kernel size and stride, thereby amplifying the spatial dimension. The batch normalization layer standardizes the feature map output by the transposed convolution, maintaining the stability of the feature distribution. The activation function layer introduces nonlinear transformations to improve the model's ability to fit complex chemical anomalies.

[0078] The decoding process also incorporates a skip connection mechanism, concatenating and fusing the spectral features enhanced by geometric information from each level of the encoder with the upsampled features from the corresponding level of the decoder. This fully utilizes multi-scale spectral feature information to improve the accuracy of chemical anomaly detection. After upsampling and feature fusion at all decoding levels, a two-dimensional convolutional layer maps the three-dimensional spectral feature map into a two-dimensional feature map. Then, a sigmoid activation function normalizes each pixel value in the two-dimensional feature map to obtain a two-dimensional chemical anomaly confidence prediction map. The value of each pixel in this prediction map falls within a specific range, and the magnitude of the value directly represents the probability of a chemical anomaly at that location. Chemical anomalies mainly correspond to substances such as fillers and precipitates within the cracked area that differ from the chemical composition of the background glaze.

[0079] In one embodiment, a bidirectional cross-modal attention mechanism is used to perform feature-level fusion of multi-scale spectral features and multi-scale geometric features of each encoder layer, resulting in chemically enhanced geometric features and geometrically enhanced spectral features of each encoder layer, including:

[0080] S21. Channel adjustment is performed on the multi-scale spectral features to obtain a spectral feature map, and the spectral feature map is spatially projected to obtain a spectral feature vector.

[0081] Indicatively, the channel dimensions of the multi-scale spectral features output from the current encoder level are adapted and adjusted. Specifically, a convolution operation is used to convert the number of channels in the multi-scale spectral features to match the number of channels in the multi-scale geometric features at the same level, eliminating the difference in channel dimensions between the two types of features and obtaining a dimensionally matched spectral feature map. This spectral feature map retains the spatial dimension and spectral information association of the original multi-scale spectral features. Furthermore, based on the previously established spatial correspondence between hyperspectral data and 3D point cloud data, the spectral feature map is projected from a regular grid structure to an irregular point set structure of the 3D point cloud. Each 3D point is assigned corresponding spectral feature information, ultimately forming a unique spectral feature vector for each 3D point, achieving a spatial mapping of spectral features from the grid domain to the point cloud domain.

[0082] S22. Using the spectral feature vector as the query vector and the multi-scale geometric features of each three-dimensional point as the key vector and value vector, calculate the first attention weight, and perform a weighted summation of the value vectors of the multi-scale geometric features based on the first attention weight to obtain the attention features of each three-dimensional point.

[0083] Specifically, let the spectral feature vector of each three-dimensional point be the query vector. This vector contains information about the chemical composition at the corresponding location; the multi-scale geometric features of each three-dimensional point serve as bond vectors. Sum value vector The key vector is used to establish the association and matching between the query vector and the geometric features, while the value vector carries the geometric feature information to be enhanced. Attention weights are calculated using a scaled dot product attention mechanism, i.e. ,in, As the first attention weight, This represents the dot product operation between the transpose of the query vector and the key vector, used to calculate the similarity between the two. Key vector The similarity results are scaled by adjusting the dimension of the vector to avoid the gradient vanishing problem caused by excessively high vector dimensions. The function is used to normalize the similarity results to the [0,1] interval, ensuring that the sum of each weight is 1, thus realizing the probabilistic allocation of weights.

[0084] Furthermore, based on the calculated first attention weights, the value vector is weighted and summed, i.e. ,in, The attention feature for each 3D point is guided by the chemical composition information, i.e. the query vector, and selectively weights the value vector of the geometric feature, so that the geometric features associated with the abnormal chemical composition area receive higher weight, thereby highlighting the geometric structure information related to the cleavage.

[0085] S23. Add the attention features and multi-scale geometric features of each 3D point element by element, and process them through a fully connected layer and a nonlinear activation function to generate the chemically enhanced geometric features of each 3D point in the current encoder layer.

[0086] Optionally, a residual connection mechanism is used to add the attention features to the original multi-scale geometric features element-wise, i.e. ,in The geometric features are those obtained after residual fusion. The original multi-scale geometric features are used. This residual connection design preserves the basic information of the original geometric features while incorporating chemically guided information selected by the attention mechanism, avoiding the loss of original geometric structure information during feature enhancement. After residual fusion, the features are input into a fully connected layer for nonlinear mapping of the feature dimensions. The fully connected layer transforms the input feature vector into higher-dimensional abstract features through a learnable weight matrix, enhancing the expressive power of the features. Subsequently, a nonlinear activation function is used to transform the mapped features, introducing nonlinear correlations among the features and improving the model's ability to fit complex cross-modal feature relationships. After the above processing, the output is the chemically guided enhanced geometric features of each 3D point in the current encoder level. These features retain the details of the original geometric structure and, through chemical guidance, focus on key regions related to the flaking.

[0087] S24. Project the enhanced geometric features of all 3D points in the current encoder layer, guided by chemical information, onto each pixel of the spectral feature map using an inverse distance weighted average to obtain the geometric feature vector corresponding to each pixel.

[0088] Specifically, to achieve the reverse mapping of geometric features from the point cloud domain to the mesh domain, an inverse distance weighted averaging algorithm is used to project the chemically enhanced geometric features onto each pixel of the spectral feature map. By establishing a spatial correspondence between the 3D point cloud and the pixels of the spectral feature map, the corresponding region of each pixel in 3D space is determined. Weights are assigned based on the spatial distance between the 3D point and the corresponding region of the pixel; the closer the 3D point, the greater its weight, and the farther away, the smaller its weight. This weighting method ensures that the projected geometric features accurately reflect the geometric characteristics of the spatial region corresponding to the pixel. A unique geometric feature vector for each pixel is calculated through weighted averaging, realizing the spatial transformation of geometric features from the point cloud domain to the mesh domain.

[0089] S25. Using the geometric feature vector as the query vector and the spectral feature vector of each pixel as the key vector and value vector, calculate the second attention weight, and use the second attention weight to perform a weighted summation of the value vector of the spectral feature vector to obtain the attention features of each pixel.

[0090] Schematic representation: the geometric feature vector of each cell serves as the query vector. This vector carries the geometric morphological information of the corresponding pixel spatial region, especially features related to cladding, such as abrupt changes in geometric shape; the spectral feature vectors of each pixel serve as key vectors. Sum value vector The key vector is used to establish similarity matching between geometric and spectral features, while the value vector carries the spectral feature information to be enhanced. The calculation of the second attention weight also employs a scaled dot product attention mechanism, i.e. ,in As the second attention weight, This represents the dot product of the query vector and the transpose of the key vector, used to measure the correlation between geometric features and spectral features; Key vector The dimension is used to scale the similarity results and avoid calculation bias caused by the dimensionality effect. The function normalizes the similarity results into probability weights, ensuring a reasonable allocation of weights.

[0091] The attention features of each pixel are obtained by weighted summation of the value vector based on the second attention weight. ,in The attention feature for pixels is guided by geometric morphology information, i.e., query vector, to highlight the spectral feature differences corresponding to geometrically abrupt regions, providing support for distinguishing the spectral signals of the segmented area from the background glaze.

[0092] S26. Add the attention features and multi-scale spectral features of each pixel element by element, and process them through a fully connected layer and a nonlinear activation function to generate the spectral features of each pixel in the current encoder layer that are enhanced by geometric information.

[0093] Continuing the residual connectivity design, the attention features are added element-wise to the original multi-scale spectral features, i.e. ,in The spectral characteristics are those after residual fusion. This method preserves the original multi-scale spectral features. It incorporates geometrically guided key features while maintaining the integrity of the original spectral data's chemical information, thus avoiding information distortion during the enhancement process.

[0094] Furthermore, the residual-fused features are input into a fully connected layer, where learnable parameters are used to perform dimensionality transformation and abstract representation of the features, strengthening the correlation information between cross-modal features. A nonlinear activation function is then used to introduce nonlinear interactions between the features, enhancing the model's ability to capture complex spectral-geometric correlation patterns. Finally, geometrically enhanced spectral features for each pixel at the current encoder level are generated, possessing both the specificity of chemical information and the directionality of geometric information.

[0095] In one embodiment, the two-stream coded cross-modal fusion model is obtained through end-to-end training using a joint loss function; the joint loss function is constructed using the following method:

[0096] S31. Based on the predicted probability and the true point-level geometric labels of the initial 3D point cloud patch segmentation results, the classification loss is calculated using the focal loss function, and the geometric segmentation loss is obtained by combining the Dice loss between the predicted segmented region and the true region of the initial 3D point cloud patch segmentation results.

[0097] This diagram illustrates the definition of labels for geometric segmentation tasks. The ground truth point-level geometric labels are the category identifiers corresponding to each point in the 3D point cloud. These identifiers correspond one-to-one with pre-defined geometric segmentation categories, covering categories such as background glaze, crack interior, and crack edge fracture zones. The predicted probabilities of the initial 3D point cloud crazing segmentation results are the probability distribution vectors of each point belonging to each geometric segmentation category output by the model. The focus loss function is used to address the class imbalance problem in geometric segmentation tasks. ,in, For the focal loss value, This represents the total number of points in the 3D point cloud. The number of geometric segmentation categories, For the first The point corresponds to the th point The true label of the category, if the first The point belongs to the th point Class Otherwise, it is 0. For the first The point belongs to the th point Predicted probability of the category This is a focusing parameter used to adjust the degree of attention given to difficult-to-differentiate samples. The function is the natural logarithm. By assigning low weights to easily segmented samples with predicted probabilities close to 1 and high weights to difficult samples with predicted probabilities close to 0, the training bias caused by class imbalance is effectively mitigated. The Dice loss is used to directly optimize the overlap between the predicted segmentation region and the ground truth region. ,in, This is the Dice loss value. For the first The point belongs to the th point The predicted label for a category is determined by a threshold applied to the predicted probability; a probability greater than the threshold is assigned a value of 1, otherwise 0. The numerator is twice the sum of the intersections of the true and predicted labels, and the denominator is the sum of the squares of the true and predicted labels. A smaller loss value indicates a higher degree of overlap between the predicted and true regions. The geometric segmentation loss is obtained through a weighted combination of the focal loss and the Dice loss. ,in, For geometric segmentation loss, and These are preset weighting coefficients used to balance the contribution ratios of focus loss and Dice loss in the total loss, ensuring that the model addresses both class imbalance and the accuracy of segmentation regions.

[0098] S32. Based on the two-dimensional chemical anomaly confidence prediction map and the real pixel-level chemical anomaly labels, the chemical anomaly detection loss is calculated using the binary cross-entropy loss function.

[0099] Specifically, the chemical anomaly detection task is a binary classification task. The true pixel-level chemical anomaly labels are two-dimensional images, with each pixel having a label value of 0 or 1. A 1 indicates the presence of a chemical anomaly in the corresponding region, such as crack filling or precipitates, while a 0 indicates the region is background glaze with no chemical anomalies. The value of each pixel in the two-dimensional chemical anomaly confidence prediction image represents the model's prediction of the presence of a chemical anomaly at that location, ranging from [0,1]. The binary cross-entropy loss function is used to measure the difference between the predicted probability and the true binary classification label. ,in, Loss due to chemical anomaly detection and These represent the height and width of the two-dimensional chemical anomaly confidence prediction map and the actual pixel-level chemical anomaly labels, respectively. For the first Line number The true chemical anomaly label of the column pixels, For the first Line number Predicted probability of chemical anomalies in column pixels The loss function is the natural logarithm. This loss reflects the overall accuracy of the model in detecting chemical anomalies by averaging the loss values ​​of all pixels. The smaller the loss value, the higher the consistency between the predicted results and the true labels.

[0100] S33. Based on the spatial correspondence between the hyperspectral data block and the three-dimensional point cloud data block, the predicted probability of the crack category in the initial three-dimensional point cloud flaking segmentation result is projected onto the two-dimensional image plane to obtain the geometric segmentation probability projection map.

[0101] Optionally, the spatial correspondence between the previously established hyperspectral data blocks and 3D point cloud data blocks, including the coordinate system mapping matrix and spatial resolution matching parameters, is utilized to achieve the projection transformation from 3D point cloud data to a 2D image plane. For the initial 3D point cloud slicing results, the predicted probability of each 3D point belonging to all crack-related categories is extracted, including categories such as crack interior and crack edge fracture zone. The maximum value among the probabilities of each category is taken as the comprehensive crack category prediction probability of that point. This probability value represents the confidence level that the point belongs to the crack region.

[0102] The comprehensive crack category prediction probability of each 3D point is assigned to the corresponding pixel position in the 2D image plane according to the mapping relationship between its spatial coordinates and the 2D image plane. For cases where multiple 3D points are mapped to the same 2D pixel, average pooling or max pooling is used to calculate the final probability value of that pixel, ensuring that each pixel corresponds to only one probability value. After the above projection and aggregation processing, a geometric segmentation probability projection map with the same spatial size as the 2D chemical anomaly confidence prediction map is obtained. The value of each pixel in this map represents the comprehensive prediction probability of the corresponding spatial region belonging to the crack category.

[0103] S34. Perform feature transformation on the two-dimensional chemical anomaly confidence prediction map through a convolutional neural network, and calculate the L2 norm loss of the two-dimensional chemical anomaly confidence prediction map after feature transformation and the geometric segmentation probability projection map to obtain the collaborative consistency loss.

[0104] For example, a lightweight convolutional neural network is constructed as a feature adapter. This network consists of multiple convolutional layers, batch normalization layers, and activation function layers. The input is a two-dimensional chemical anomaly confidence prediction map, and the output is a transformed feature map with the same feature dimensions as the geometric segmentation probability projection map. The convolutional layers extract features and transform the dimensions of the input confidence map by setting convolutional kernels of a specific size; the batch normalization layers standardize the convolutional output features to maintain the stability of the feature distribution; and the activation function layers introduce nonlinear transformations to enhance the fitting ability of the feature adapter. Through this feature adapter, the two-dimensional chemical anomaly confidence prediction map is transformed from the confidence space to the feature space, ensuring that it is feature-comparable to the geometric segmentation probability projection map. The collaborative consistency loss is obtained by calculating the L2 norm loss between the feature-transformed two-dimensional chemical anomaly confidence prediction map and the geometric segmentation probability projection map. ,in, For the loss of coordination and consistency, and For the height and width of the image, This represents the feature transformation operation of the feature adapter. The first in the two-dimensional chemical anomaly confidence prediction graph Line number Confidence values ​​of the columns of pixels, For the geometric segmentation probability projection diagram, the first Line number The probability value of a column cell. This represents the square operation of the L2 norm. This loss forces the chemical anomaly detection results and geometric segmentation results to be spatially consistent, achieving synergistic optimization of the two tasks and avoiding contradictory prediction results.

[0105] S35. The geometric segmentation loss, chemical anomaly detection loss, and cooperative consistency loss are linearly combined according to preset weights to obtain the joint loss function.

[0106] Specifically, the joint loss function is constructed by weighted linear combination of geometric segmentation loss, chemical anomaly detection loss, and collaborative consistency loss, i.e. ,in, The total loss value of the joint loss function. For geometric segmentation loss, Loss due to chemical anomaly detection For the loss of coordination and consistency, The preset weighting coefficient for the collaborative consistency loss is used to adjust the degree of influence of the collaborative consistency loss in the total loss, ensuring that the model can maintain the spatial consistency of the prediction results of the two tasks while focusing on optimizing their respective tasks.

[0107] Optionally, the joint loss function integrates the losses of the three dimensions into a unified optimization objective. During model training, the gradient of the total loss with respect to all learnable parameters of the model is calculated through the backpropagation algorithm, and the parameters are updated based on the gradient descent optimization strategy. This enables end-to-end training of the dual-stream coding cross-modal fusion model, allowing the model to simultaneously learn accurate geometric segmentation capabilities and chemical anomaly detection capabilities, providing reliable model support for subsequent automatic extraction of morphology.

[0108] In one embodiment, channel adjustment is performed on the multi-scale spectral features to obtain a spectral feature map, and the spectral feature map is spatially projected to obtain a spectral feature vector, including:

[0109] S41. Based on each encoder layer, channel transformation is performed on the multi-scale spectral features through three-dimensional convolution to obtain a spectral feature map with the same number of channels as the multi-scale geometric features of the same encoder layer.

[0110] For the multi-scale spectral features output by each encoder level, the number of channels differs from the number of channels in the multi-scale geometric features of the same level. This channel dimension mismatch prevents effective computation during subsequent cross-modal feature fusion. Specifically, channel transformation is performed through 3D convolution. The core function of the 3D convolution layer is to compress or expand the channel dimension without changing the spectral feature space dimension or the spectral dimension, achieving precise matching of the number of channels. For example, the calculation process of 3D convolution follows the basic principles of convolution operations. ,in This is the spectral feature map after channel adjustment. Represents a three-dimensional convolution operation. For original multi-scale spectral features, This is the weight matrix of the 3D convolution kernel. The size of the convolution kernel in the spatial and spectral dimensions is set according to the local neighborhood information range of the features, and the size in the channel dimension is related to the number of input and output channels. The bias vector for the 3D convolution is used to adjust the baseline value of the convolution output. Through this 3D convolution operation, the number of channels of the multi-scale spectral features is precisely adjusted to be the same as the number of channels of the multi-scale geometric features in the same encoder layer, resulting in a spectral feature map with matched channel dimensions. This feature map preserves the spatial-spectral correlation information of the original multi-scale spectral features, laying the foundation for subsequent spatial projection.

[0111] S42. Based on the spatial correspondence between the hyperspectral data block and the 3D point cloud data block, the spatial coordinates of all 3D points in the 3D point cloud data block are mapped to the 2D image coordinate system where the spectral feature map is located through affine transformation to obtain the 2D coordinates corresponding to each 3D point.

[0112] In the preprocessing stage, an initial spatial correspondence is established between the hyperspectral data block and the 3D point cloud data block. This correspondence includes transformation parameters such as rotation and translation between coordinate systems. Based on this correspondence, an affine transformation can be used to map the 3D point coordinates to the 2D image coordinate system. The affine transformation preserves the parallelism between spatial points, ensuring that the mapped 2D coordinates accurately reflect the relative positions of the 3D points on the spectral feature map. For example, the calculation formula for the affine transformation is as follows: ,in, For the first The two-dimensional coordinates of a 3D point mapped to a 2D image coordinate system For the first The original 3D coordinates of a 3D point in a 3D point cloud coordinate system The affine transformation matrix, derived from the previously established spatial correspondence, includes transformation parameters such as rotation, translation, and scaling. It is used to achieve a linear transformation from the 3D coordinate system to the 2D image coordinate system. The matrix has a dimension of 3×4 to ensure efficient multiplication between the 3D coordinate vector and the transformation matrix. By performing this affine transformation on each 3D point in the 3D point cloud data block, the corresponding 2D coordinates of each 3D point in the 2D image coordinate system containing the spectral feature map are obtained, establishing the positional association between the starting point cloud domain and the mesh domain.

[0113] S43. Based on the two-dimensional coordinates corresponding to each three-dimensional point, find four pixel grid points that are adjacent to the two-dimensional coordinates on the spectral feature map, and perform bilinear interpolation according to the relative positions of the two-dimensional coordinates and the four pixel grid points to obtain the spectral feature vector of each three-dimensional point.

[0114] Furthermore, since the two-dimensional coordinates obtained from the mapping of three-dimensional points usually do not completely coincide with the pixel grid points of the spectral feature map, directly taking the features of the nearest pixel will lead to information loss or errors. Therefore, a bilinear interpolation algorithm is used, which utilizes the spectral features of the four nearest pixels for weighted calculation to obtain accurate interpolated features. Specifically, the two-dimensional coordinates corresponding to each three-dimensional point are determined. In the grid cell of the spectral feature map, find the pixel grid points corresponding to the four vertices of that grid cell, and denote them as follows: , , , ,in and These represent the integer parts of the two-dimensional coordinates in the horizontal and vertical directions of the spectral feature map, respectively, indicating the starting coordinates of the grid cell. The relative offset between the two-dimensional coordinates and the four pixel grid points is calculated. , ,in, This is the relative offset in the horizontal direction, and its value range is... , This is the relative offset in the vertical direction, and its value range is also [value range missing]. The offset is used to determine the weighting coefficients for the features of the four neighboring pixels. Furthermore, the formula for bilinear interpolation is as follows: ,in, For the first Spectral feature vectors of three-dimensional points , , , These are the spectral feature vectors of four neighboring pixel grid points, , , , The weighted coefficients for the four pixel features sum to 1, ensuring the reasonableness of the interpolation result. Through this bilinear interpolation operation, the grid domain features of the spectral feature map are accurately mapped to each 3D point, resulting in a unique spectral feature vector for each 3D point. This vector has the same number of channels as the geometric feature vector at the same level.

[0115] In one embodiment, the chemically enhanced geometric features of all 3D points in the current encoder layer are projected onto each pixel of the spectral feature map using an inverse distance-weighted average to obtain the geometric feature vector corresponding to each pixel, including:

[0116] S51. Reverse map the position coordinates of each pixel on the spectral feature map in the two-dimensional image coordinate system back to the three-dimensional space to obtain the three-dimensional reference point corresponding to each pixel.

[0117] Indicatively, each pixel in the spectral feature map has fixed position coordinates in a two-dimensional image coordinate system, which correspond one-to-one with the spatial acquisition location of the hyperspectral data. To establish the spatial association between pixels and the three-dimensional point cloud, based on the previously established spatial correspondence between the hyperspectral data and the three-dimensional point cloud, the two-dimensional pixel coordinates are reversed and mapped to three-dimensional space through an inverse affine transformation to obtain a three-dimensional reference point corresponding to each pixel. This reference point is used to characterize the center position of the pixel in three-dimensional space. The formula for calculating the inverse affine transformation is as follows: ,in The coordinates of the three-dimensional reference point corresponding to the two-dimensional pixel. The initial forward affine transformation matrix The inverse matrix, obtained through matrix inversion, is used to perform an inverse linear transformation from a two-dimensional image coordinate system to a three-dimensional point cloud coordinate system. Given the position coordinates of a pixel in a two-dimensional image coordinate system, we extend them to homogeneous coordinates. Then, a multiplication operation is performed with the inverse transformation matrix to obtain the homogeneous coordinates of the three-dimensional reference point. After removing the homogeneous term, the spatial coordinates of the three-dimensional reference point are obtained. By performing this inverse mapping operation on each pixel in the spectral feature map, a one-to-one correspondence between grid domain pixels and three-dimensional spatial reference points is established.

[0118] S52. Divide the three-dimensional spherical space neighborhood around the three-dimensional reference point, and find all three-dimensional points belonging to the current encoder layer within the three-dimensional spherical space neighborhood as neighboring points.

[0119] Furthermore, the 3D reference point is merely an abstract center position of the pixel in 3D space. Neighborhood partitioning is used to obtain the real 3D point cloud data surrounding this position, ensuring that the projected features reflect the true geometric characteristics of the spatial region corresponding to the pixel. A spherical space with a fixed radius is defined as the neighborhood range, centered on each 3D reference point. The neighborhood radius must balance point cloud density and feature representation requirements, ensuring sufficient coverage of 3D points to reflect local geometric features while avoiding the introduction of irrelevant geometric information due to an excessively large range. After neighborhood partitioning, spatial retrieval is performed on the 3D point cloud data of the current encoder layer to determine whether the spatial coordinates of each 3D point fall within the spherical neighborhood. Spatial retrieval is achieved by calculating the relationship between the straight-line distance from the 3D point to the 3D reference point and the neighborhood radius. If the distance is less than or equal to the neighborhood radius, the 3D point belongs to the spherical neighborhood. All 3D points meeting the conditions are selected as the set of neighboring points corresponding to the pixel. Each point in the neighboring point set carries geometric features enhanced by chemical information, providing a data foundation for subsequent weighted fusion.

[0120] S53. Calculate the Euclidean distance from each neighboring point to the three-dimensional reference point, and calculate the inverse distance weight of each neighboring point based on the Euclidean distance.

[0121] Specifically, Euclidean distance measures the spatial proximity between neighboring points and a 3D reference point; a smaller distance indicates a greater contribution of neighboring points to the pixel's geometric features. The formula for calculating Euclidean distance is: ,in For the first Euclidean distance from each neighboring point to the 3D reference point For the first Spatial coordinates of neighboring points Given the spatial coordinates of a three-dimensional reference point, the straight-line distance between two points is obtained by taking the square root of the sum of the squares of the coordinate differences.

[0122] The inverse distance weight is calculated based on the reciprocal of the Euclidean distance, achieving differentiated weighting of features of neighboring points. ,in, For the first The inverse distance weights of the nearest neighbors For the first Euclidean distance between neighboring points The total number of points in the neighbor set. For the first The Euclidean distance between the n nearest neighbors. The numerator is the nth... The weights of all neighboring points are calculated by taking the reciprocal of their distances and using the sum of the reciprocals of their distances in the denominator. This normalization ensures that the sum of the weights of all neighboring points is 1, thus guaranteeing the rationality of the weighted fusion result and achieving a differentiated allocation where closer neighboring points have greater weights.

[0123] S54. Multiply the enhanced geometric features of each neighboring point guided by chemical information with the inverse distance weight to obtain the weighted features, and fuse the weighted features of all neighboring points to obtain the geometric feature vector of the corresponding pixel.

[0124] Optionally, each neighboring point carries a chemically enhanced geometric feature vector, which contains chemically optimized geometric structure information. The feature vector of each neighboring point is element-wise multiplied with its corresponding inverse distance weight to achieve a weighted adjustment of the features. ,in, For the first The weighted feature vector of the nearest neighbor points For the first The inverse distance weights of the nearest neighbors For the first The geometric feature vectors of the neighboring points are enhanced by chemical information. Through weighted operations, the enhanced geometric features of neighboring points closer to the 3D reference point receive a higher contribution, while the feature contribution of neighboring points farther away decreases accordingly.

[0125] The weighted feature vectors of all neighboring points are summed and fused element-by-element to obtain the geometric feature vector of the corresponding pixel, i.e. ,in, The geometric feature vector corresponding to the pixel. The total number of points in the neighbor set. For the first The weighted feature vectors of the neighboring points. The summation and fusion process integrates the enhanced geometric features of all neighboring points within the 3D spatial region corresponding to the pixel, and achieves differentiated aggregation of features through inverse distance weights. The final geometric feature vector not only retains the key information of the local geometric structure, but also highlights the feature contribution of the core region through weight allocation. It corresponds one-to-one with the pixels of the spectral feature map and can be directly used for subsequent cross-modal attention fusion calculations.

[0126] In one embodiment, based on a two-dimensional chemical anomaly confidence prediction map, arbitration optimization is performed on predicted points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results to obtain optimized three-dimensional point cloud segmentation labels, including:

[0127] S61. Based on the spatial correspondence between the hyperspectral data block and the three-dimensional point cloud data block, the two-dimensional chemical anomaly confidence prediction map is mapped onto the three-dimensional point cloud of the three-dimensional point cloud block through bilinear interpolation to obtain the chemical anomaly confidence value of each three-dimensional point.

[0128] Indicatively, a stable spatial correspondence has been established between the hyperspectral data block and the 3D point cloud data block in the early stages. This relationship includes coordinate system mapping rules and spatial resolution matching parameters, providing a foundation for cross-domain mapping from 2D data to 3D point clouds. The 2D chemical anomaly confidence prediction map has a regular grid structure, while the 3D point cloud is an irregular point set. Direct mapping would result in some 3D points lacking corresponding confidence information; therefore, a bilinear interpolation algorithm is used to achieve accurate mapping. Specifically, based on the spatial correspondence, the 3D coordinates of each 3D point are mapped to the 2D image coordinate system where the 2D chemical anomaly confidence prediction map resides, obtaining the corresponding 2D mapped coordinates for each 3D point. Subsequently, the chemical anomaly confidence value corresponding to the two-dimensional mapped coordinates was calculated using bilinear interpolation, i.e. ,in, For the first Chemical anomaly confidence values ​​for three-dimensional points Two-dimensional mapped coordinates The coordinates of the starting cell of the grid cell. , These are the horizontal and vertical offsets of the two-dimensional mapped coordinates relative to the starting pixel, with values ​​ranging from [0,1). , , , These represent the chemical anomaly confidence values ​​for the four vertex pixels of the grid unit. Through this bilinear interpolation operation, the confidence information of the two-dimensional regular grid is accurately transferred to the three-dimensional irregular point cloud, so that each three-dimensional point obtains a corresponding chemical anomaly confidence value, providing data support for subsequent arbitration judgment.

[0129] S62. Select the three-dimensional points in the initial three-dimensional point cloud flaking segmentation prediction results that belong to the crack category and whose prediction probabilities are between the preset lower threshold and the preset upper threshold to obtain the boundary fuzzy point set.

[0130] In the initial 3D point cloud segmentation prediction results, each 3D point contains a predicted probability of belonging to each classification category. The crack category includes categories related to cracking, such as crack interior and crack edge fracturing. A preset lower threshold and a preset upper threshold constitute a probability ambiguity interval. This interval is determined based on the probability distribution characteristics predicted by the model and is used to define 3D points with uncertain category classification. For example, the screening process uses the crack category prediction probability of each 3D point as the criterion. If the probability value is greater than the preset lower threshold and less than the preset upper threshold, it indicates that the model cannot clearly determine the category classification of the point; such points are considered boundary ambiguity points. All 3D points that meet this condition are filtered out to form a boundary ambiguity point set. Points in this set may be boundary points of real cracks or background points misclassified by the model, requiring further arbitration using chemical anomaly confidence information.

[0131] S63. For each three-dimensional point in the boundary ambiguity point set, determine whether the chemical anomaly confidence value of the three-dimensional point is greater than the preset chemical arbitration threshold. If the chemical anomaly confidence value is greater than the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as the crack category. If the chemical anomaly confidence value is less than or equal to the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as the non-crack category.

[0132] Optionally, a preset chemical arbitration threshold is determined based on the statistical distribution of chemical anomaly confidence scores from a large number of labeled samples. This threshold is used to distinguish the chemical characteristics of the crazing area from the background area—the crazing area typically has a higher chemical anomaly confidence score than the background glaze area due to the presence of fillers, precipitates, and other substances. For each 3D point in the set of blurred boundary points, its corresponding chemical anomaly confidence score is extracted. and with chemical arbitration threshold Compare. If This indicates a significant chemical anomaly at the spatial location corresponding to this point, which highly matches the chemical characteristics of the flaking area. Therefore, the final label arbitration for this point is a crack category; if This indicates that the spatial location corresponding to this point has no obvious chemical anomalies and is more consistent with the chemical characteristics of the background glaze. Therefore, the final label arbitration for this point is classified as non-crack. The high specificity of chemical features enables accurate determination of the category of points with ambiguous boundaries.

[0133] S64. All three-dimensional points whose predicted probability is lower than the preset lower limit threshold and whose chemical anomaly confidence value is higher than the average value of the background area are identified as an isolated point group.

[0134] Optionally, the preset lower threshold is a low threshold for the predicted probability of the crack category. 3D points with a predicted probability below this threshold are typically classified as non-crack points in the initial segmentation results. The background region average is the statistical average of the chemical anomaly confidence values ​​of all clearly identified background glaze points in the 3D point cloud, i.e. ,in, This is the average value of the background area. To clearly determine the total number of three-dimensional points on the background glaze surface, For the first The chemical anomaly confidence value of each background glaze point. This average value reflects the baseline level of chemical anomalies in the background glaze area. During the screening process, the criteria are simultaneously met: the crack type prediction probability is below a preset lower threshold, and the chemical anomaly confidence value is [value missing]. These points were initially misclassified as non-cracks, but they possess the chemical characteristics of the flaking region. They may be isolated crack points that were missed due to their unclear geometric features, and their spatial distribution characteristics need to be further verified.

[0135] S65. If the area of ​​the spatially connected region of the isolated point group is greater than the preset minimum area threshold, the final label of the isolated point group will be arbitrated as the crack category.

[0136] Specifically, the area of ​​the spatially connected region is used to measure the spatial clustering of isolated point groups. Based on the spatial coordinates of the 3D point cloud, a connectivity analysis algorithm is used to determine whether the points in the isolated point group meet the spatial proximity condition, i.e., the Euclidean distance between two points is less than a set connectivity distance threshold. Points that meet the proximity condition are grouped into the same connected region, and the number of 3D points contained in the region or the corresponding spatial projection area is counted, which is the area of ​​the spatially connected region. The preset minimum area threshold is determined based on the minimum scale statistics of the real cracked area and is used to filter out false chemical anomalies caused by noise. If the area of ​​the spatially connected region of the isolated point group is greater than the minimum area threshold, it indicates that the point group is not random noise, but a real cracked area with a certain spatial scale, and the final label of the isolated point group is arbitrated as crack category; if the area of ​​the spatially connected region is less than or equal to the minimum area threshold, it is judged as noise point, and its non-crack category label is maintained.

[0137] S66. The final labels of all 3D points are used to obtain the optimized segmentation labels for the 3D point cloud.

[0138] After the arbitration optimization steps described above, each point in the 3D point cloud obtains a clear final label. This includes the original labels for 3D points with clear categories (i.e., crack points with predicted probabilities higher than a preset upper threshold) and background points with predicted probabilities lower than a preset lower threshold and chemical anomaly confidence levels lower than the average of the background region; labels for points with blurred boundaries after chemical arbitration; and labels for isolated point groups after spatial connectivity verification. All the final labels of the 3D points are then aggregated according to their index positions in the 3D point cloud, forming a label set that corresponds one-to-one with the original 3D point cloud. This set constitutes the optimized 3D point cloud segmentation label. This label set corrects misjudged and missed points in the initial segmentation, achieving accurate division between cracked and non-cracked regions, and providing a reliable label foundation for the subsequent reconstruction of the 3D cracked network morphology.

[0139] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0140] Based on the same inventive concept, this application also provides an automatic extraction device for the crackle morphology of ancient ceramic glazes, which is used to implement the automatic extraction method for the crackle morphology of ancient ceramic glazes described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the automatic extraction device for the crackle morphology of ancient ceramic glazes provided below can be found in the limitations of the automatic extraction method for the crackle morphology of ancient ceramic glazes described above, and will not be repeated here.

[0141] In one exemplary embodiment, such as Figure 2 As shown, an automatic extraction device for the crazing patterns of ancient ceramic glazes is provided, comprising:

[0142] Data module 201 is used to acquire hyperspectral image cubes and three-dimensional point cloud data of ancient ceramic glazes, and to perform spatial alignment processing on hyperspectral image cubes and three-dimensional point cloud data to obtain hyperspectral data blocks and three-dimensional point cloud data blocks.

[0143] Prediction module 202 is used to input hyperspectral data blocks and three-dimensional point cloud data blocks into a pre-trained dual-stream coded cross-modal fusion model to obtain initial three-dimensional point cloud slicing prediction results and two-dimensional chemical anomaly confidence prediction maps.

[0144] Optimization module 203 is used to arbitrate and optimize the prediction points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results based on the two-dimensional chemical anomaly confidence prediction map, so as to obtain the three-dimensional point cloud optimized segmentation label.

[0145] The crack morphology module 204 is used to perform three-dimensional spatial smoothing on the optimized segmentation labels of the three-dimensional point cloud to obtain the three-dimensional crack network morphology of the ancient ceramic glaze.

[0146] In one embodiment, the prediction module 202 is further configured to:

[0147] Multi-scale spectral features of hyperspectral data blocks are extracted by a spectral flow 3D convolutional encoder, and multi-scale geometric features of 3D point cloud data blocks are extracted by a geometric flow point cloud network encoder.

[0148] By using a bidirectional cross-modal attention mechanism, the multi-scale spectral features and multi-scale geometric features of each encoder layer are fused at the feature level to obtain the geometric features and spectral features of each encoder layer that are enhanced by chemical information and guided by geometric information.

[0149] The geometric decoder is used to upsample and decode the chemically enhanced geometric features of all encoder layers to obtain the initial 3D point cloud slicing prediction results.

[0150] Using a chemical decoder, the spectral features of all encoder layers enhanced by geometric information are upsampled and decoded to obtain a two-dimensional chemical anomaly confidence prediction map.

[0151] In one embodiment, a feature-guided fusion module is further included, for:

[0152] Channel adjustment is performed on the multi-scale spectral features to obtain a spectral feature map, and then the spectral feature map is spatially projected to obtain a spectral feature vector;

[0153] Using the spectral feature vector as the query vector and the multi-scale geometric features of each 3D point as the key vector and value vector, the first attention weight is calculated, and the value vectors of the multi-scale geometric features are weighted and summed based on the first attention weight to obtain the attention features of each 3D point.

[0154] The attention features of each 3D point are added element-wise with the multi-scale geometric features, and then processed by a fully connected layer and a nonlinear activation function to generate the chemically enhanced geometric features of each 3D point in the current encoder layer.

[0155] The geometric features of all 3D points in the current encoder layer, enhanced by chemical information, are projected onto each pixel of the spectral feature map through an inverse distance weighted average to obtain the geometric feature vector corresponding to each pixel.

[0156] Using the geometric feature vector as the query vector and the spectral feature vector of each pixel as the key vector and value vector, the second attention weight is calculated, and the value vector of the spectral feature vector is weighted and summed using the second attention weight to obtain the attention features of each pixel;

[0157] The attention features of each pixel are added element-wise with the multi-scale spectral features, and then processed by a fully connected layer and a nonlinear activation function to generate the spectral features of each pixel in the current encoder layer that are enhanced by geometric information.

[0158] In one embodiment, a model training module is also included, for:

[0159] Based on the predicted probability and the real point-level geometric labels of the initial 3D point cloud patch segmentation results, the classification loss is calculated through the focal loss function, and the geometric segmentation loss is obtained by combining the Dice loss between the predicted segmentation region and the real region of the initial 3D point cloud patch segmentation results.

[0160] Based on the two-dimensional chemical anomaly confidence prediction map and the real pixel-level chemical anomaly labels, the chemical anomaly detection loss is calculated using the binary cross-entropy loss function.

[0161] Based on the spatial correspondence between the hyperspectral data block and the 3D point cloud data block, the predicted probability of the crack category in the initial 3D point cloud flaking segmentation result is projected onto the 2D image plane to obtain the geometric segmentation probability projection map.

[0162] The confidence prediction map of two-dimensional chemical anomalies is transformed by a convolutional neural network, and the L2 norm loss of the two-dimensional confidence prediction map of chemical anomalies after feature transformation and the geometric segmentation probability projection map are calculated to obtain the cooperative consistency loss.

[0163] The geometric segmentation loss, chemical anomaly detection loss, and cooperative consistency loss are linearly combined according to preset weights to obtain the joint loss function.

[0164] In one embodiment, a feature transformation module is further included, for:

[0165] Based on each encoder layer, channel transformation of multi-scale spectral features is performed through three-dimensional convolution to obtain a spectral feature map with the same number of channels as the multi-scale geometric features of the same encoder layer.

[0166] Based on the spatial correspondence between hyperspectral data blocks and 3D point cloud data blocks, the spatial coordinates of all 3D points in the 3D point cloud data block are mapped to the 2D image coordinate system where the spectral feature map is located through affine transformation, so as to obtain the 2D coordinates corresponding to each 3D point.

[0167] Based on the two-dimensional coordinates corresponding to each three-dimensional point, four pixel grid points adjacent to the two-dimensional coordinates are found on the spectral feature map. Bilinear interpolation is then performed based on the relative positions of the two-dimensional coordinates and the four pixel grid points to obtain the spectral feature vector of each three-dimensional point.

[0168] In one embodiment, a projection module is also included, for:

[0169] The position coordinates of each pixel on the spectral feature map in the two-dimensional image coordinate system are reversed and mapped back to the three-dimensional space to obtain the three-dimensional reference point corresponding to each pixel.

[0170] Divide the three-dimensional spherical space neighborhood with the three-dimensional reference point as the center, and find all three-dimensional points belonging to the current encoder layer within the three-dimensional spherical space neighborhood as neighboring points;

[0171] Calculate the Euclidean distance from each neighboring point to the 3D reference point, and calculate the inverse distance weight of each neighboring point based on the Euclidean distance;

[0172] The weighted features are obtained by multiplying the enhanced geometric features of each neighboring point guided by chemical information with the inverse distance weights. The weighted features of all neighboring points are then fused to obtain the geometric feature vector of the corresponding pixel.

[0173] In one embodiment, the optimization module 203 is further configured to:

[0174] Based on the spatial correspondence between hyperspectral data blocks and 3D point cloud data blocks, the two-dimensional chemical anomaly confidence prediction map is mapped onto the 3D point cloud of the 3D point cloud block through bilinear interpolation to obtain the chemical anomaly confidence value of each 3D point.

[0175] Select the three-dimensional points that belong to the crack category in the initial three-dimensional point cloud flaking segmentation prediction results and whose prediction probabilities are between the preset lower threshold and the preset upper threshold to obtain the boundary fuzzy point set;

[0176] For each three-dimensional point in the boundary ambiguity point set, determine whether the chemical anomaly confidence value of the three-dimensional point is greater than the preset chemical arbitration threshold. If the chemical anomaly confidence value is greater than the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as a crack category. If the chemical anomaly confidence value is less than or equal to the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as a non-crack category.

[0177] All three-dimensional points whose predicted probability is lower than the preset lower threshold and whose chemical anomaly confidence value is higher than the average value of the background area are identified as isolated point groups.

[0178] If the area of ​​the spatially connected region of an isolated point group is greater than the preset minimum area threshold, the final label of the isolated point group will be arbitrated as a crack category.

[0179] The final labels of all 3D points are used to obtain the optimized segmentation labels for the 3D point cloud.

[0180] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps in the above method embodiments.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0182] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0183] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. An automatic method for extracting the crazing patterns of ancient ceramic glazes, characterized in that, The method includes: The hyperspectral image cube and three-dimensional point cloud data of the glaze surface of ancient ceramics are acquired, and the hyperspectral image cube and the three-dimensional point cloud data are spatially aligned to obtain hyperspectral data blocks and three-dimensional point cloud data blocks. The hyperspectral data block and the three-dimensional point cloud data block are input into a pre-trained dual-stream coded cross-modal fusion model to obtain the initial three-dimensional point cloud slicing prediction results and the two-dimensional chemical anomaly confidence prediction map. Based on the two-dimensional chemical anomaly confidence prediction map, arbitration optimization is performed on the prediction points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results to obtain the three-dimensional point cloud optimized segmentation label. The three-dimensional point cloud segmentation labels are subjected to three-dimensional spatial smoothing to obtain the three-dimensional crackle network morphology of the ancient ceramic glaze.

2. The method according to claim 1, characterized in that, The dual-stream coded cross-modal fusion model obtains the initial 3D point cloud patching prediction result and the 2D chemical anomaly confidence prediction map through the following methods: The multi-scale spectral features of the hyperspectral data block are extracted by a spectral flow 3D convolutional encoder, and the multi-scale geometric features of the 3D point cloud data block are extracted by a geometric flow point cloud network encoder. By using a bidirectional cross-modal attention mechanism, the multi-scale spectral features and multi-scale geometric features of each encoder layer are fused at the feature level to obtain the geometric features and spectral features of each encoder layer that are enhanced by chemical information and guided by geometric information. The geometric decoder is used to upsample and decode the chemically enhanced geometric features of all encoder layers to obtain the initial 3D point cloud slicing prediction results; Using a chemical decoder, the spectral features enhanced by geometric information in all encoder layers are upsampled and decoded to obtain a two-dimensional chemical anomaly confidence prediction map.

3. The method according to claim 2, characterized in that, The multi-scale spectral features and multi-scale geometric features of each encoder layer are fused at the feature level through a bidirectional cross-modal attention mechanism to obtain the chemically enhanced geometric features and geometrically enhanced spectral features of each encoder layer, including: Channel adjustment is performed on the multi-scale spectral features to obtain a spectral feature map, and the spectral feature map is spatially projected to obtain a spectral feature vector; Using the spectral feature vector as the query vector and the multi-scale geometric features of each three-dimensional point as the key vector and value vector, a first attention weight is calculated, and the value vectors of the multi-scale geometric features are weighted and summed based on the first attention weight to obtain the attention features of each three-dimensional point. The attention features of each of the three-dimensional points are added element-wise to the multi-scale geometric features, and then processed through a fully connected layer and a nonlinear activation function to generate the chemically enhanced geometric features of each of the three-dimensional points in the current encoder layer. The chemically enhanced geometric features of all 3D points in the current encoder layer are projected onto each pixel of the spectral feature map by inverse distance weighted averaging to obtain the geometric feature vector corresponding to each pixel. Using the geometric feature vector as the query vector and the spectral feature vector of each pixel as the key vector and value vector, a second attention weight is calculated, and the value vector of the spectral feature vector is weighted and summed using the second attention weight to obtain the attention features of each pixel; The attention features of each pixel are added element-wise to the multi-scale spectral features, and then processed through a fully connected layer and a nonlinear activation function to generate the geometrically enhanced spectral features of each pixel in the current encoder layer.

4. The method according to claim 2, characterized in that: The dual-stream encoded cross-modal fusion model is obtained through end-to-end training using a joint loss function; the joint loss function is constructed using the following method: Based on the predicted probability and the true point-level geometric labels of the initial 3D point cloud patch segmentation results, the classification loss is calculated using the focus loss function, and the geometric segmentation loss is obtained by combining the Dice loss between the predicted segmentation region and the true region of the initial 3D point cloud patch segmentation results. Based on the two-dimensional chemical anomaly confidence prediction map and the real pixel-level chemical anomaly labels, the chemical anomaly detection loss is calculated using the binary cross-entropy loss function. Based on the spatial correspondence between the hyperspectral data block and the three-dimensional point cloud data block, the predicted probability of the crack category in the initial three-dimensional point cloud flaking segmentation result is projected onto the two-dimensional image plane to obtain a geometric segmentation probability projection map. The two-dimensional chemical anomaly confidence prediction map is transformed using a convolutional neural network, and the L2 norm loss of the transformed two-dimensional chemical anomaly confidence prediction map and the geometric segmentation probability projection map is calculated to obtain the collaborative consistency loss. The geometric segmentation loss, the chemical anomaly detection loss, and the cooperative consistency loss are linearly combined according to preset weights to obtain the joint loss function.

5. The method according to claim 3, characterized in that, The process of channel-adjusting the multi-scale spectral features to obtain a spectral feature map, and spatially projecting the spectral feature map to obtain a spectral feature vector, includes: Based on each encoder layer, the multi-scale spectral features are transformed by three-dimensional convolution to obtain a spectral feature map with the same number of channels as the multi-scale geometric features of the same encoder layer. Based on the spatial correspondence between the hyperspectral data block and the three-dimensional point cloud data block, the spatial coordinates of all three-dimensional points in the three-dimensional point cloud data block are mapped to the two-dimensional image coordinate system where the spectral feature map is located through affine transformation to obtain the two-dimensional coordinates corresponding to each three-dimensional point. Based on the two-dimensional coordinates corresponding to each three-dimensional point, four pixel grid points adjacent to the two-dimensional coordinates are found on the spectral feature map, and bilinear interpolation is performed according to the relative positions of the two-dimensional coordinates and the four pixel grid points to obtain the spectral feature vector of each three-dimensional point.

6. The method according to claim 5, characterized in that, The step of projecting the chemically enhanced geometric features of all 3D points in the current encoder layer onto each pixel of the spectral feature map using an inverse distance weighted average to obtain the geometric feature vector corresponding to each pixel includes: The position coordinates of each pixel on the spectral feature map in the two-dimensional image coordinate system are reversed and mapped back to the three-dimensional space to obtain the three-dimensional reference point corresponding to each pixel. Divide a three-dimensional spherical spatial neighborhood around the three-dimensional reference point, and find all three-dimensional points belonging to the current encoder layer within the three-dimensional spherical spatial neighborhood as neighboring points; Calculate the Euclidean distance from each of the neighboring points to the three-dimensional reference point, and calculate the inverse distance weight of each of the neighboring points based on the Euclidean distance; The chemically enhanced geometric features of each neighboring point are multiplied by the inverse distance weight to obtain a weighted feature, and the weighted features of all neighboring points are fused to obtain the geometric feature vector of the corresponding pixel.

7. The method according to claim 1, characterized in that, The arbitration optimization of predicted points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results, based on the two-dimensional chemical anomaly confidence prediction map, yields optimized three-dimensional point cloud segmentation labels, including: Based on the spatial correspondence between the hyperspectral data block and the three-dimensional point cloud data block, the two-dimensional chemical anomaly confidence prediction map is mapped onto the three-dimensional point cloud of the three-dimensional point cloud block through bilinear interpolation to obtain the chemical anomaly confidence value of each three-dimensional point. Filter out the three-dimensional points in the initial three-dimensional point cloud flaking segmentation prediction results that belong to the crack category and whose prediction probabilities are between a preset lower threshold and a preset upper threshold to obtain a boundary fuzzy point set; For each of the three-dimensional points in the boundary ambiguity point set, it is determined whether the chemical anomaly confidence value of the three-dimensional point is greater than a preset chemical arbitration threshold. If the chemical anomaly confidence value is greater than the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as a crack category. If the chemical anomaly confidence value is less than or equal to the preset chemical arbitration threshold, the final label of the three-dimensional point is arbitrated as a non-crack category. All three-dimensional points whose predicted probability is lower than the preset lower threshold and whose chemical anomaly confidence value is higher than the average value of the background area are identified as an isolated point group. If the area of ​​the spatially connected region of the isolated point group is greater than the preset minimum area threshold, the final label of the isolated point group will be arbitrated as a crack category. The final labels of all the three-dimensional points are used to obtain the optimized segmentation labels for the three-dimensional point cloud.

8. An automatic extraction device for the crackle pattern of ancient ceramic glaze, characterized in that, The device includes: The data module is used to acquire hyperspectral image cubes and three-dimensional point cloud data of ancient ceramic glazes, and to perform spatial alignment processing on the hyperspectral image cubes and the three-dimensional point cloud data to obtain hyperspectral data blocks and three-dimensional point cloud data blocks. The prediction module is used to input the hyperspectral data block and the three-dimensional point cloud data block into a pre-trained dual-stream coded cross-modal fusion model to obtain the initial three-dimensional point cloud slicing prediction result and the two-dimensional chemical anomaly confidence prediction map. The optimization module is used to perform arbitration optimization on the prediction points with blurred or isolated boundaries in the initial three-dimensional point cloud slicing prediction results based on the two-dimensional chemical anomaly confidence prediction map, so as to obtain the three-dimensional point cloud optimized segmentation label. The crack morphology module is used to perform three-dimensional spatial smoothing on the optimized segmentation labels of the three-dimensional point cloud to obtain the three-dimensional crack network morphology of the ancient ceramic glaze.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images