Intelligent inference method and system for cultural relic material parameters based on multi-modal large model

Abstract

The application relates to an intelligent relic material parameter reasoning method and system based on a multi-modal large model, the method comprising the following steps: acquiring a high-resolution digital image and structured archive text data of a to-be-tested relic, performing standardization preprocessing on the image, and performing word segmentation, stop word removal and word embedding coding on the structured archive text data; adopting a visual encoder and a text encoder of a multi-modal pre-training large model to respectively extract a visual feature vector and a text feature vector; realizing semantic alignment of the two types of features through a cross-attention mechanism to generate a cross-modal fusion feature vector; inputting the fusion feature into a material parameter prediction head network constructed by a multi-layer perception mechanism to obtain SVBRDF material parameters; and outputting SVBRDF material parameter reasoning results through parameter analysis and physical constraint processing. The method can fuse visual and semantic information of the relic, and realize efficient and accurate SVBRDF material parameter reasoning in a non-contact manner, thereby supporting digital protection and high-fidelity virtual display of large-scale relics.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and digital cultural relic protection technology, and in particular relates to an intelligent reasoning method and system for cultural relic material parameters based on a multimodal large model. Background Technology

[0002] With the deep integration and development of computer vision and digital cultural heritage protection technologies, image material parameter inversion technology based on deep learning has emerged. This technology can infer the physical material properties of the surface of cultural relics in a non-contact manner by analyzing two-dimensional digital images of cultural relics, providing a new technical path to replace the traditional physical measurement paradigm.

[0003] In traditional techniques, obtaining material parameters of cultural relics mainly relies on precision physical measurement instruments such as spectrophotometers, white light interferometers, and laser confocal microscopes. Operated by professionals, these instruments scan the surface of the relics point by point, measuring parameters such as spectral reflectance, surface microstructure, and roughness. Data fitting is then used to obtain spatially varying bidirectional reflectance distribution function parameters, including diffuse reflectance, normal distribution, roughness, and specular coefficient. Some studies have also attempted to use convolutional neural networks to directly predict material parameters from a single image, but these rely solely on end-to-end mapping learning based on visual texture information.

[0004] However, current methods for obtaining material parameters of cultural relics have the following significant problems. First, traditional physical measurement methods rely on precision instruments such as spectrophotometers and laser confocal microscopes, requiring professionals to scan point by point. Measuring the complete parameters of a single cultural relic can take hours or even days, resulting in extremely low efficiency. Moreover, the measurement process requires close contact or high-intensity light exposure, posing irreversible safety risks to fragile and sensitive unearthed cultural relics, and is unsuitable for the digitization needs of large-scale museum collections. Second, existing deep learning-based material prediction methods rely solely on end-to-end mapping from single-view images. On the one hand, they face the ill-conditioned problem of retrieving high-dimensional physical properties from two-dimensional images. Pixel color is the result of multiple couplings of lighting, geometry, and material, making the model prone to physical ambiguity. On the other hand, they completely ignore the unique semantic information of cultural relics, such as historical archives and archaeological records, and lack the ability to model professional terminology and material characteristics in the field of cultural relics. General models are difficult to directly adapt to professional scenarios for predicting the material properties of cultural relics, and the prediction accuracy cannot meet the needs of practical applications. Summary of the Invention

[0005] Therefore, it is necessary to provide a technical solution that can deeply integrate the visual information and historical semantic information of cultural relics to achieve efficient and accurate material parameter reasoning, in order to overcome the shortcomings of traditional methods such as low efficiency, high cost and high risk. This solution is based on a multimodal large model for intelligent reasoning of cultural relic material parameters.

[0006] Firstly, this application provides an intelligent reasoning method for cultural relic material parameters based on a multimodal large model, including:

[0007] S1. Acquire high-resolution digital images and structured archival text data of the cultural relic to be tested, and preprocess the high-resolution digital images to obtain preprocessed image data.

[0008] S2. Perform word segmentation, stop word removal, and word embedding encoding on the structured archive text data to obtain the encoded text data;

[0009] S3. A multimodal pre-trained large model is used as a feature extractor. Visual features are extracted from the preprocessed image data by a visual encoder to obtain visual feature vectors. Text semantic features are extracted from the encoded text data by a text encoder to obtain text feature vectors.

[0010] S4. Semantically align the visual feature vector and text feature vector through a cross-attention mechanism to obtain a cross-modal fusion feature vector;

[0011] S5. Input the cross-modal fusion feature vector into the material parameter prediction head network built on a multilayer perceptron to obtain the SVBRDF material parameters;

[0012] S6. Perform parameter analysis and physical constraint processing on the SVBRDF material parameters to obtain the inference results of the SVBRDF material parameters of the artifact to be tested.

[0013] In one embodiment, S2 includes:

[0014] S21. Clean the text content in the structured archive text data to obtain the cleaned text content;

[0015] S22. Perform Chinese word segmentation on the cleaned text content to obtain the segmentation results, and remove function words and punctuation marks from the segmentation results by comparing with the preset stop word list to obtain a valid word sequence;

[0016] S23. Map each word in the effective vocabulary sequence to a fixed-dimensional word vector, and aggregate the word vectors through average pooling to obtain the encoded text data.

[0017] In one embodiment, S4 includes:

[0018] S41. Based on the visual feature vector, a visual feature matrix is ​​constructed; based on the text feature vector, a text feature matrix is ​​constructed.

[0019] S42. Perform cross-modal similarity measurement on the visual feature matrix and the text feature matrix to obtain the cross-modal similarity matrix;

[0020] S43. Using a preset temperature scaling factor, normalize the cross-modal similarity matrix to obtain the cross-modal attention weight matrix;

[0021] S44. Based on the cross-modal attention weight matrix, the visual feature vector and the text feature vector are weighted and fused to obtain the cross-modal fused feature vector; wherein, the cross-modal fused feature vector includes detailed information and text semantic prior.

[0022] In one embodiment, the expression for the cross-modal attention weight matrix is:

[0023]

[0024] In the formula, This represents the cross-modal attention weight matrix. Represents the visual feature matrix. This represents the transpose of the visual feature matrix. Represents the text feature matrix. The feature dimensions represent the visual feature vector and the text feature vector.

[0025] In one embodiment, the training method of the material parameter prediction head network includes:

[0026] S51. Obtain a dataset of no less than 50 cultural relics with labeled ground truth values ​​of measured SVBRDF parameters; wherein, the cultural relics sample dataset includes corresponding high-resolution images, archival texts, and ground truth values ​​of measured SVBRDF parameters obtained through physical measurement equipment;

[0027] S52. Freeze all network parameters of the visual encoder and text encoder, set the parameters of the material parameter prediction head network to a trainable state, set the initial learning rate of the material parameter prediction head network to 1e-5, use the AdamW optimizer, the batch size to 8, and the number of training epochs to 1000.

[0028] S53. Based on the cultural relic sample dataset, the weighted mean square error function is used as the training loss function, and the material parameter prediction head network is subjected to domain adaptive fine-tuning to obtain the domain adaptive fine-tuned material parameter prediction head network.

[0029] In one embodiment, the expression for the total training loss value of the training loss function is:

[0030]

[0031] in, This represents the total training loss value during the domain-adaptive fine-tuning process. This represents the weighting coefficient of the diffuse reflectance loss term. The weighting coefficients represent the normal distribution loss term. This represents the weighting coefficient of the roughness loss term. This represents the weighting coefficient of the specular loss term. This represents the mean square error loss between the predicted diffuse reflectance parameter and the measured true value of diffuse reflectance. This represents the mean square error loss between the predicted normal distribution parameters and the measured true values ​​of the normal distribution. This represents the mean square error loss between the predicted roughness parameters and the measured true roughness values. This represents the mean square error loss between the predicted specular coefficient parameters and the measured true specular coefficient values.

[0032] Secondly, this application also provides an intelligent reasoning system for cultural relic material parameters based on a multimodal large model, including:

[0033] The image preprocessing module is used to acquire high-resolution digital images and structured archival text data of the cultural relics to be tested, and to preprocess the high-resolution digital images to obtain preprocessed image data.

[0034] The text encoding module is used to perform word segmentation, stop word removal, and word embedding encoding on structured archive text data to obtain encoded text data;

[0035] The multimodal feature extraction module uses a multimodal pre-trained large model as a feature extractor. It extracts visual features from the preprocessed image data through a visual encoder to obtain visual feature vectors, and extracts text semantic features from the encoded text data through a text encoder to obtain text feature vectors.

[0036] The cross-modal alignment module is used to semantically align visual feature vectors and text feature vectors through a cross-attention mechanism to obtain cross-modal fused feature vectors;

[0037] The parameter prediction module is used to input the cross-modal fused feature vector into the material parameter prediction head network built on a multilayer perceptron to obtain the SVBRDF material parameters.

[0038] The parameter post-processing module is used to perform parameter parsing and physical constraint processing on the SVBRDF material parameters to obtain the SVBRDF material parameter inference results of the artifact to be tested.

[0039] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods described above.

[0040] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0041] The aforementioned intelligent reasoning method, system, device, and medium for cultural relic material parameters based on a multimodal large model, by preprocessing and encoding high-resolution digital images and structured historical archive text, can transform the visual appearance and historical semantic information of cultural relics into standardized multimodal data representations, laying the foundation for subsequent cross-modal fusion. By constructing a multimodal pre-trained large model based on a visual encoder and a text encoder for dual-channel feature extraction, it can capture the visual texture details and domain semantic connotations of cultural relics respectively. Based on the semantic alignment operation of the cross-attention mechanism, the visual feature vector and the text feature vector are deeply fused to generate cross-modal fused feature vectors that include pixel-level material details and physical semantic priors, which can effectively overcome the ambiguity problem when single-vision methods invert high-dimensional physical parameters from two-dimensional images. By using a material parameter prediction head network based on a multilayer perceptron to perform nonlinear mapping on the fused features, it can infer SVBRDF material parameters, including diffuse reflectance, normal distribution, roughness, and specular coefficient, end-to-end. Through parameter analysis and physical constraint processing, it ensures the optical self-consistency between parameters and outputs physically reasonable material parameter inference results, significantly improving the accuracy, efficiency, and security of cultural relic material parameter acquisition. This method can integrate the visual and semantic information of cultural relics and achieve efficient and accurate SVBRDF material parameter reasoning in a non-contact manner, supporting the digital protection and high-fidelity virtual display of large-scale cultural relics. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 A flowchart illustrating an intelligent reasoning method for cultural relic material parameters based on a multimodal large model, provided as an exemplary embodiment of this application;

[0044] Figure 2 A schematic diagram of a process for generating encoded text data is provided as an exemplary embodiment of this application;

[0045] Figure 3 This is a schematic diagram of the structure of an intelligent reasoning system for cultural relic material parameters based on a multimodal large model, provided as an exemplary embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0047] In one embodiment, such as Figure 1 As shown, an intelligent reasoning method for artifact material parameters based on a multimodal large model is provided. This embodiment illustrates the application of this method to an intelligent reasoning terminal. It is understood that this method can also be applied to an intelligent reasoning server, and further to a system including both an intelligent reasoning terminal and an intelligent reasoning server, and is implemented through the interaction between the intelligent reasoning terminal and the intelligent reasoning server. In this embodiment, the method includes the following steps:

[0048] S1. Acquire high-resolution digital images and structured archival text data of the cultural relic to be tested, and preprocess the high-resolution digital images to obtain preprocessed image data.

[0049] Specifically, the intelligent reasoning terminal can establish a secure communication connection with the museum's digital management terminal through a pre-set standardized data interface, and retrieve high-resolution digital images and structured archival text data corresponding to the cultural relic to be tested.

[0050] Furthermore, the intelligent inference terminal can sequentially perform non-local mean denoising, color correction based on a standard color chart, and normalization processing on high-resolution digital images to eliminate image differences caused by different shooting conditions and obtain pre-processed image data.

[0051] Optionally, the high-resolution digital images of the artifact to be tested can be front views, side views, and key local detail images of the artifact surface, acquired by professional artifact photography equipment under standard light source and diffused lighting conditions.

[0052] Optionally, structured archival text data can be used to characterize the basic attribute information and historical and archaeological information of cultural relics. Structured archival text data may include, but is not limited to, the name, dating, place of excavation, preservation status, production process and decorative features of cultural relics.

[0053] Optionally, the preprocessed image data can be used to characterize the visual information of the artifact surface after removing noise interference, color deviation, and size differences.

[0054] S2. Perform word segmentation, stop word removal, and word embedding encoding on the structured archive text data to obtain the encoded text data.

[0055] Specifically, the intelligent inference terminal can perform format cleaning on structured archival text data, removing special symbols, redundant spaces, line breaks, and formatting marks to obtain standardized plain text content. The intelligent inference terminal can then use Chinese word segmentation algorithms to segment the plain text content sentence by sentence, breaking down continuous text sequences into independent semantic lexical units.

[0056] Furthermore, the intelligent inference terminal can filter out function words, conjunctions, auxiliary words, and punctuation marks that have no actual semantic contribution by referring to a predefined stop word list specifically for the cultural relics field, thus obtaining a word sequence containing only effective semantic information. The intelligent inference terminal can map each effective word to a dense real-valued vector of fixed dimensions, and then perform global aggregation of all word vectors through average pooling to obtain a unified-dimensional text feature representation, i.e., the encoded text data.

[0057] Optionally, the predefined stop word list for the cultural relics field can be constructed based on a massive corpus of cultural relics and archaeological documents, which includes both general Chinese stop words and high-frequency general words without distinction in the cultural relics field.

[0058] Optionally, the encoded text data can be used to characterize the global semantic features of the structured archive text.

[0059] S3. A multimodal pre-trained large model is used as a feature extractor. Visual features are extracted from the preprocessed image data by a visual encoder to obtain visual feature vectors. Text semantic features are extracted from the encoded text data by a text encoder to obtain text feature vectors.

[0060] Specifically, the intelligent inference terminal can load a pre-trained multimodal pre-trained large model as a general feature extractor to perform parallel feature extraction on the pre-processed image data and the encoded text data respectively.

[0061] Furthermore, the visual encoder can perform block embedding and multi-layer self-attention computation on the preprocessed image data, extracting low-level visual features and high-level semantic features from the image layer by layer, and outputting a fixed-dimensional visual feature vector. The text encoder can perform multi-head self-attention computation on the encoded text data, extracting semantic dependencies and contextual information between words, and outputting a text feature vector with the same dimension as the visual feature vector.

[0062] Optionally, the multimodal pre-trained large model can be a cross-modal pre-trained model trained based on a contrastive learning framework. The multimodal pre-trained large model completes basic pre-training on a large-scale general image-text pairing dataset and is able to learn the association mapping relationship between general visual concepts and text semantics.

[0063] Optionally, low-level visual features may include, but are not limited to, the edges, textures, and colors of the artifact.

[0064] Optionally, advanced semantic features may include, but are not limited to, the material, decoration, and shape of the artifact.

[0065] Optionally, the visual feature vector can be used to characterize the multi-scale visual appearance features of the artifact surface. The visual feature vector may include, but is not limited to, pixel-level texture details and global spatial structure information.

[0066] Optionally, text feature vectors can be used to characterize the domain semantic connotations of archival texts.

[0067] S4. Semantically align the visual feature vector and the text feature vector through a cross-attention mechanism to obtain a cross-modal fusion feature vector.

[0068] Specifically, the intelligent inference terminal can convert visual feature vectors and text feature vectors into two-dimensional visual feature matrices and text feature matrices, respectively. The rows of the visual feature matrix and the text feature matrix correspond to different feature dimensions, and the columns correspond to different semantic units.

[0069] Furthermore, the intelligent inference terminal can calculate the cosine similarity between the visual feature matrix and the text feature matrix to obtain a cross-modal similarity matrix. The intelligent inference terminal can normalize the cross-modal similarity matrix using a preset temperature scaling factor to obtain a cross-modal attention weight matrix with values ​​ranging from 0 to 1. Based on the cross-modal attention weight matrix, the intelligent inference terminal can perform a weighted summation of the visual and text features to obtain a cross-modal fusion feature vector.

[0070] Optionally, cross-modal fusion feature vectors can be used to characterize comprehensive features that integrate pixel-level visual details and domain-level semantic priors, effectively eliminating the ambiguity of single visual features in the high-dimensional physical parameter inversion process.

[0071] S5. Input the cross-modal fusion feature vector into the material parameter prediction head network built on a multilayer perceptron to obtain the SVBRDF material parameters.

[0072] Specifically, the intelligent inference terminal can input cross-modal fused feature vectors into a material parameter prediction head network built on a multi-layer perceptron (MLP) to obtain SVBRDF material parameters through multi-layer nonlinear mapping.

[0073] Optionally, the material parameter prediction head network built on a multilayer perceptron can be a feedforward neural network composed of multiple fully connected layers stacked sequentially. Nonlinear activation functions are set between adjacent fully connected layers to fit the complex nonlinear mapping relationship between cross-modal fusion features and material parameters. Regularization mechanisms can also be introduced into the network to prevent overfitting.

[0074] Optionally, SVBRDF is an abbreviation for Spatially Varying Bidirectional Reflectance Distribution Function, which is used to quantitatively describe the reflection characteristics of incident light at different spatial locations on the surface of an object. It is a core parameter of physically based rendering technology.

[0075] Optionally, the SVBRDF material parameters can be a set of multi-dimensional parameters including diffuse reflectance, normal distribution, roughness, and specular coefficient. Diffuse reflectance can be used to characterize the diffuse reflection capability and spectral characteristics of the object surface to incident light, normal distribution can be used to characterize the micro-geometric undulation structure of the object surface, roughness can be used to characterize the macro-smoothness of the object surface, and specular coefficient can be used to characterize the specular reflection intensity and directional characteristics of the object surface.

[0076] S6. Perform parameter analysis and physical constraint processing on the SVBRDF material parameters to obtain the inference results of the SVBRDF material parameters of the artifact to be tested.

[0077] Specifically, the intelligent inference terminal can perform dimensionality analysis on the one-dimensional raw numerical vector output by the material parameter prediction head network based on a multilayer perceptron. According to a preset parameter dimensionality partitioning rule, it decomposes the unified one-dimensional raw numerical vector into four independent parameter components: diffuse reflectance, normal distribution, roughness, and specular coefficient. Then, it performs linear numerical range mapping on each parameter component, converting them into standard numerical ranges that conform to optical physics. Based on the fundamental physical laws of geometric optics and radiative transfer, the intelligent inference terminal can apply cross-physical constraints to each parameter component, ensuring optical self-consistency between parameters and eliminating physically unreasonable parameter combinations, thus obtaining the SVBRDF material parameter inference results for the artifact under test.

[0078] Optionally, the SVBRDF material parameter inference result of the artifact to be tested can be a standardized set of material parameters that has been physically verified.

[0079] Optionally, physical constraints may include, but are not limited to, constraints on the range of diffuse reflectance values, constraints on normal vector normalization, and constraints on the correlation between specular coefficient and roughness.

[0080] In the aforementioned intelligent reasoning method for cultural relic material parameters based on a multimodal large model, the intelligent reasoning terminal preprocesses and encodes high-resolution digital images and structured historical archive text, transforming the visual appearance and historical semantic information of cultural relics into standardized multimodal data representations, laying the foundation for subsequent cross-modal fusion. By constructing a multimodal pre-trained large model based on a visual encoder and a text encoder for dual-channel feature extraction, it can capture the visual texture details and domain semantic connotations of cultural relics respectively. Based on the semantic alignment operation of the cross-attention mechanism, the visual feature vector and the text feature vector are deeply fused, generating a cross-modal fused feature vector that includes pixel-level material details and physical semantic priors, effectively overcoming the ambiguity problem when single-vision methods invert high-dimensional physical parameters from two-dimensional images. By using a material parameter prediction head network based on a multilayer perceptron to perform nonlinear mapping on the fused features, it can infer SVBRDF material parameters, including diffuse reflectance, normal distribution, roughness, and specular coefficient, end-to-end. Through parameter analysis and physical constraint processing, the optical self-consistency between parameters is ensured, outputting physically reasonable material parameter reasoning results, significantly improving the accuracy, efficiency, and security of cultural relic material parameter acquisition. This method can integrate the visual and semantic information of cultural relics and achieve efficient and accurate SVBRDF material parameter reasoning in a non-contact manner, supporting the digital protection and high-fidelity virtual display of large-scale cultural relics.

[0081] In one embodiment, such as Figure 2 As shown, a flowchart for generating encoded text data is provided, S2, which may include:

[0082] S21. Clean the text content in the structured archive text data to obtain the cleaned text content.

[0083] For example, the intelligent inference terminal can parse the field structure of structured archival text data, identify the logical boundaries of different information fields, and perform standardized cleaning of the text content within each field. The intelligent inference terminal can sequentially remove format control characters, special symbols, redundant spaces, and line breaks from the text content in each field, correcting typos, garbled characters, and inconsistencies in expression caused by optical character recognition or manual input, and standardizing the writing format of proper nouns such as dates, place names, and craft names. Furthermore, the intelligent inference terminal can filter out administrative labels, archival numbers, and redundant explanatory text unrelated to the material properties of the cultural relics, retaining only the core semantic content related to the material, craftsmanship, and preservation status of the cultural relics, thus obtaining the cleaned text content.

[0084] S22. Perform Chinese word segmentation on the cleaned text content to obtain the segmentation results, and remove function words and punctuation marks from the segmentation results by comparing them with the preset stop word list to obtain a valid word sequence.

[0085] For example, the intelligent inference terminal can employ a Chinese word segmentation algorithm based on statistical learning and domain rule fusion to perform sentence-by-sentence semantic segmentation on the cleaned plain text content, breaking down continuous sequences of Chinese characters into independent semantic lexical units. During the word segmentation process, the intelligent inference terminal can load a pre-built dictionary of specialized terms in the cultural relics field to accurately identify material names, casting techniques, decorative patterns, and archaeological terms for bronzes, ceramics, jades, etc., and obtain the word segmentation results. The intelligent inference terminal can then iterate through the word segmentation results, filtering out words from the stop word list and residual punctuation marks to obtain a valid word sequence.

[0086] Optionally, the word segmentation results can be used to characterize the basic semantic unit division state of the text content, reflecting the lexical composition and semantic structure hierarchy of the text.

[0087] Optionally, the preset stop word list can be constructed based on the statistics of a massive corpus of cultural relics and archaeological documents, and may include, but is not limited to, general Chinese stop words and high-frequency general words in the field of cultural relics that lack differentiation.

[0088] Optionally, an effective lexical sequence can be used to characterize the set of core semantic information in the text that actually contributes to the reasoning of material parameters.

[0089] S23. Map each word in the effective vocabulary sequence to a fixed-dimensional word vector, and aggregate the word vectors through average pooling to obtain the encoded text data.

[0090] For example, the intelligent reasoning terminal can perform semantic mapping on each word in the effective vocabulary sequence based on a pre-trained word embedding model. By loading a word embedding model jointly pre-trained on a large-scale Chinese general corpus and a cultural relics and archaeology professional corpus, discrete text vocabulary symbols are converted into continuous high-dimensional real number vector representations, so that semantically similar professional words present similar spatial distribution characteristics in the vector space, and the semantic association and domain attribute information of the words are preserved, resulting in fixed-dimensional word vectors.

[0091] Furthermore, after completing the mapping from words to vectors, the intelligent inference terminal can perform average pooling on all word vectors corresponding to the effective word sequence, calculate the arithmetic mean of all word vectors in each dimension, and obtain a global text feature vector of a unified dimension, that is, the encoded text data.

[0092] Preferably, the expression for the global text feature vector can be:

[0093]

[0094] In the formula, This represents the encoded global text feature vector. This represents the total number of words contained in the valid word sequence. Indicates the first Fixed-dimensional word vectors corresponding to each effective word.

[0095] In this embodiment, the intelligent inference terminal can transform unstructured cultural relic archive text into a standardized global feature vector that includes the core semantics of the domain through cleaning, word segmentation, stop word removal, and word embedding average pooling, thereby providing a high-quality text representation foundation for multimodal feature alignment.

[0096] In one embodiment, S4 may include:

[0097] S41. Based on the visual feature vector, a visual feature matrix is ​​constructed, and based on the text feature vector, a text feature matrix is ​​constructed.

[0098] For example, the intelligent inference terminal can perform a dimension reshaping operation on a one-dimensional visual feature vector, converting the continuous visual feature vector into a two-dimensional visual feature matrix according to a preset semantic unit division rule.

[0099] Furthermore, the intelligent inference terminal can reshape the text feature vectors in the same dimensions to construct a text feature matrix.

[0100] Optionally, the visual feature matrix can be used to characterize the set of local visual features corresponding to different spatial regions in the image of the cultural relic. Each row of the visual feature matrix corresponds to the multi-dimensional visual attributes of a local semantic region in the image, and each column corresponds to a specific visual feature dimension.

[0101] Optionally, the text feature matrix can be used to characterize the set of local semantic features corresponding to different effective words in the archival text, with each row of the matrix corresponding to the semantic representation of a core word.

[0102] S42. Perform cross-modal similarity measurement on the visual feature matrix and the text feature matrix to obtain the cross-modal similarity matrix.

[0103] For example, the intelligent reasoning terminal can perform element-wise cosine similarity calculation on the visual feature matrix and the text feature matrix in a unified cross-modal semantic space to obtain a cross-modal similarity matrix.

[0104] Optionally, the cross-modal similarity matrix can be used to characterize the pairwise semantic matching relationship between all visual semantic units and text semantic units. The numerical value of the elements of the cross-modal similarity matrix can reflect the semantic similarity between the corresponding visual features and text features. The larger the value, the closer the semantic association between the two.

[0105] S43. Using a preset temperature scaling factor, normalize the cross-modal similarity matrix to obtain the cross-modal attention weight matrix.

[0106] For example, the intelligent inference terminal can use a preset temperature scaling factor to globally scale the cross-modal similarity matrix, and use a normalized exponential function (Softmax Function) to normalize the row dimensions of the scaled cross-modal similarity matrix to obtain the cross-modal attention weight matrix.

[0107] Optionally, the preset temperature scaling factor can be a hyperparameter used to adjust the sharpness of the attention distribution. By adjusting the temperature scaling factor, the degree of focus of the multimodal pre-trained large model on highly similar feature pairs can be controlled.

[0108] Optionally, the cross-modal attention weight matrix can be used to characterize the contribution weight of each text semantic feature to each visual local feature.

[0109] S44. Based on the cross-modal attention weight matrix, the visual feature vector and the text feature vector are weighted and fused to obtain the cross-modal fused feature vector.

[0110] For example, the intelligent inference terminal can use the contribution weights of the cross-modal attention weight matrix as weighting coefficients to perform a weighted summation of the text feature matrix, obtaining text-guided visual features with the same dimension as the visual feature vector. The intelligent inference terminal can then perform element-wise fusion of the text-guided visual features with the original visual feature vector to obtain a cross-modal fused feature vector.

[0111] Optionally, the cross-modal fusion feature vector may include, but is not limited to, detailed information and textual semantic priors.

[0112] In this embodiment, the intelligent inference terminal constructs a visual feature matrix and a text feature matrix, calculates cross-modal similarity and performs temperature scaling normalization, and uses attention weights to incorporate text semantic prior weights into visual features, generating a cross-modal fusion feature vector that can effectively eliminate single visual ambiguity.

[0113] In one embodiment, the expression for the cross-modal attention weight matrix can be:

[0114]

[0115] In the formula, This represents the cross-modal attention weight matrix. Represents the visual feature matrix. This represents the transpose of the visual feature matrix. Represents the text feature matrix. The feature dimensions represent the visual feature vector and the text feature vector.

[0116] For example, the intelligent inference terminal can transpose the visual feature matrix to obtain a transpose of the visual feature matrix. The intelligent inference terminal can then perform matrix multiplication between the transpose of the visual feature matrix and the text feature matrix to obtain a dot product matching matrix of visual and text features.

[0117] Furthermore, the intelligent inference terminal can divide the dot product matching matrix by the arithmetic square root of the feature dimensions of the visual feature vector and the text feature vector to obtain the cross-modal attention weight matrix.

[0118] In this embodiment, the intelligent inference terminal calculates the cross-modal attention weight matrix using the dot product scaling formula, and uses the square root of the feature dimension as a scaling factor to stabilize the gradient, thereby achieving an efficient and measurable association of similarity between visual and text features.

[0119] In one embodiment, the training method for the material parameter prediction head network may include:

[0120] S51. Obtain a dataset of no less than 50 cultural relics with labeled and measured true values ​​of SVBRDF parameters.

[0121] For example, the intelligent inference terminal can establish a communication connection with the standardized sample database of cultural relic protection institutions through a preset encrypted data interface to obtain a dataset of cultural relic samples jointly annotated by professional archaeologists and material testing personnel. The intelligent inference terminal can perform integrity verification and format normalization on the obtained cultural relic sample dataset, and remove invalid samples with blurred images, missing text information, or non-standard parameter annotations.

[0122] Optionally, the artifact sample dataset may include corresponding high-resolution images, archival text, and measured true values ​​of SVBRDF parameters obtained through physical measurement equipment.

[0123] S52. Freeze all network parameters of the visual encoder and text encoder, set the parameters of the material parameter prediction head network to trainable state, set the initial learning rate of the material parameter prediction head network to 1e-5, use the AdamW optimizer, the batch size to 8, and the number of training epochs to 1000.

[0124] For example, the intelligent inference terminal can load a pre-trained multimodal model and an initialized material parameter prediction head network, perform a parameter freeze operation, set all network parameters of the visual encoder and text encoder to a non-updatable state, and configure only the parameters of the material parameter prediction head network to a trainable state. The intelligent inference terminal can set a preset initial learning rate for the material parameter prediction head network, use the AdamW optimizer (Adaptive Moment Estimation with Weight Decay), and iteratively train according to a preset batch size and number of training epochs to gradually adjust the parameter distribution of the prediction head network.

[0125] S53. Based on the cultural relic sample dataset, the weighted mean square error function is used as the training loss function, and the material parameter prediction head network is subjected to domain adaptive fine-tuning to obtain the domain adaptive fine-tuned material parameter prediction head network.

[0126] For example, the intelligent inference terminal can input the cross-modal fusion features of the cultural relic sample dataset into the material parameter prediction head network during each round of iterative training to obtain the predicted SVBRDF material parameters.

[0127] Furthermore, the intelligent inference terminal can input the predicted SVBRDF material parameters and the corresponding measured SVBRDF parameter ground truth values ​​into a weighted mean square error function to calculate the training loss value for the current round. Based on the training loss value, the intelligent inference terminal can use the backpropagation algorithm to calculate the gradient of each trainable parameter of the material parameter prediction head network, and adjust the network parameters according to a preset update rule through the AdamW optimizer to iteratively reduce the training loss value. After multiple rounds of iterative training, a material parameter prediction head network adapted to the cultural relics field is obtained.

[0128] In this embodiment, the intelligent inference terminal completes the domain adaptive fine-tuning of the material parameter prediction head network by using a small number of labeled samples, freezing the encoder, and using weighted mean square error loss, so that the material parameter prediction head network can map cross-modal features to physically real material parameters.

[0129] In one embodiment, the expression for the total training loss value of the training loss function can be:

[0130]

[0131] in, This represents the total training loss value during the domain-adaptive fine-tuning process. This represents the weighting coefficient of the diffuse reflectance loss term. The weighting coefficients represent the normal distribution loss term. This represents the weighting coefficient of the roughness loss term. This represents the weighting coefficient of the specular loss term. This represents the mean square error loss between the predicted diffuse reflectance parameter and the measured true value of diffuse reflectance. This represents the mean square error loss between the predicted normal distribution parameters and the measured true values ​​of the normal distribution. This represents the mean square error loss between the predicted roughness parameters and the measured true roughness values. This represents the mean square error loss between the predicted specular coefficient parameters and the measured true specular coefficient values.

[0132] For example, in each round of iterative training, the intelligent inference terminal can extract the predicted values ​​of diffuse reflectance, normal distribution, roughness and specular coefficient output by the material parameter prediction head network, compare them dimension by dimension with the actual measured parameter values ​​of the cultural relic sample corresponding to the predicted values ​​of diffuse reflectance, normal distribution, roughness and specular coefficient, and calculate the mean square error loss term corresponding to each SVBRDF material parameter.

[0133] Furthermore, the intelligent inference terminal can perform a weighted summation of each mean square error loss term according to preset weight coefficients to obtain the total training loss value in the domain adaptive fine-tuning process.

[0134] Preferably, the expression for the mean square error loss between the predicted diffuse reflectance parameter and the measured true value of diffuse reflectance can be:

[0135]

[0136] In the formula, This represents the mean square error loss between the predicted diffuse reflectance parameter and the measured true value of diffuse reflectance. The first parameter representing the predicted diffuse reflectance is... Each channel component The first value representing the true value of the measured diffuse reflectance Each channel component This corresponds to the RGB three-channel configuration.

[0137] Preferably, the expression for the mean square error loss between the predicted normal distribution parameters and the measured true values ​​of the normal distribution can be:

[0138]

[0139] In the formula, This represents the mean square error loss between the predicted normal distribution parameters and the measured true values ​​of the normal distribution. The first parameter represents the predicted normal distribution parameter. One portion, The th value representing the true value of the measured normal distribution One portion, ,correspond Axial direction.

[0140] Preferably, the expression for the mean square error loss between the predicted roughness parameter and the measured true roughness value can be:

[0141]

[0142] In the formula, This represents the mean square error loss between the predicted roughness parameters and the measured true roughness values. Indicates the predicted roughness parameters. This represents the true value of the measured surface roughness.

[0143] Preferably, the expression for the mean square error loss between the predicted specular coefficient parameters and the measured true specular coefficient values ​​can be:

[0144]

[0145] In the formula, This represents the mean square error loss between the predicted specular parameters and the measured true specular parameters. This represents the predicted specular coefficient parameter. This represents the true value of the measured specular coefficient.

[0146] In this embodiment, the intelligent inference terminal constructs a weighted loss function by assigning differentiated weights to different SVBRDF material parameters, thereby optimizing the different SVBRDF material parameters and improving the overall physical accuracy of the prediction results.

[0147] The aforementioned intelligent reasoning method and system for cultural relic material parameters based on a multimodal large model transforms the visual appearance and historical semantic information of cultural relics into a unified multimodal data representation by standardizing and semantically encoding high-resolution digital images and structured archival text. It employs a multimodal pre-trained large model for dual-channel feature extraction, capturing both visual texture details and domain semantic connotations of the cultural relics. A cross-attention mechanism is used to measure the similarity of visual and textual feature matrices and normalize them with temperature scaling, generating a cross-modal fusion feature vector that integrates pixel-level details and physical semantic priors, effectively eliminating the ambiguity of retrieving high-dimensional physical parameters from a single visual perspective. Domain-adaptive fine-tuning of the material parameter prediction head network based on a multilayer perceptron is performed using a small number of labeled samples and a weighted mean square error loss function with differentiated weights, enabling the multilayer perceptron-based material parameter prediction head network to map cross-modal features to physically real material parameters. Finally, by performing dimensionality analysis and cross-physical constraint processing based on geometric optics on the predicted parameters, a physically self-consistent SVBRDF material parameter inference result is output. This technical solution replaces traditional physical measurement and single visual prediction modes with non-contact multimodal fusion reasoning. By using textual semantics to guide visual features across modalities and constrain them physically, it can achieve efficient and accurate reasoning of material parameters, thereby supporting the digital protection and high-fidelity virtual display of large-scale cultural relics.

[0148] In practical applications of intelligent reasoning for cultural relic material parameters based on multimodal large models, the following steps may be included:

[0149] Step 1: Construct a visual-semantic dual-channel input module. Acquire high-resolution digital images of cultural relics and structured archival text data. The high-resolution digital images have a resolution of no less than 4096×4096 pixels and adopt an RGB three-channel format.

[0150] Furthermore, the image data undergoes standardization preprocessing, including noise reduction, color correction, and size normalization.

[0151] Furthermore, the text data is segmented, stop words are removed, and word embedding encoding is performed.

[0152] Optionally, structured archival text data may include, but is not limited to, the name, date, place of excavation, preservation condition, and craftsmanship characteristics of the cultural relic.

[0153] Step 2: Extract multimodal features and perform cross-modal alignment. A large multimodal pre-trained model is used as the feature extractor, including a visual encoder and a text encoder. The visual encoder adopts the VisionTransformer (ViT-L / 14) architecture, receives the pre-processed artifact image, and outputs a visual feature vector with a dimension of 768.

[0154] Optionally, the text encoder uses a Transformer architecture, receives the encoded archive text, and outputs a text feature vector with a dimension of 768.

[0155] Furthermore, a cross-attention mechanism is used to achieve semantic alignment between visual and textual features, and a cross-modal similarity matrix is ​​calculated:

[0156]

[0157] in, Represents the visual feature matrix. This represents the transpose of the visual feature matrix. Represents the text feature matrix. The feature dimension is d=768. This is the cross-modal similarity matrix.

[0158] Furthermore, the cross-modal similarity matrix is ​​normalized using a temperature scaling factor τ=0.07 to obtain the cross-modal attention weights.

[0159] Step 3: Construct the material parameter prediction head network. A multilayer perceptron is designed as the material parameter prediction head, comprising three fully connected layers with hidden layer dimensions of 512, 256, and 128 respectively. ReLU activation is used between adjacent layers, and the last layer uses linear activation to output SVBRDF parameters. The SVBRDF parameters include: diffuse reflectance (a 3D vector corresponding to the RGB three channels), normal distribution (a 3D vector representing the surface normal direction), roughness (a 1D scalar), and specular (a 1D scalar), totaling an 8-dimensional output vector. The input to the material parameter prediction head network is a cross-modal aligned fused feature vector with a dimension of 768.

[0160] Step 4: Introduce a domain-adaptive fine-tuning strategy. Use a small number of labeled artifact material parameter samples to perform domain-adaptive fine-tuning on the multimodal pre-trained model. The fine-tuning sample size should be no less than 50 artifacts, and each artifact's fine-tuning sample should include high-resolution images, archival text, and measured ground truth values ​​of SVBRDF parameters. A strategy of freezing encoder parameters and training only the prediction head network is adopted. The initial learning rate is set to 1e-5, the AdamW optimizer is used, the batch size is 8, and the training epochs are 1000. The loss function is weighted mean squared error.

[0161]

[0162] in, , , , These represent the mean square error losses for diffuse reflectance, normal distribution, roughness, and specular coefficient, respectively. , , , The weighting coefficients for diffuse reflectance, normal distribution, roughness, and specular coefficient are 1.0, 1.5, 2.0, and 1.0, respectively.

[0163] Step 5: End-to-end output of material parameters. For the artifact to be tested, input its high-resolution image and structured archival text. After feature extraction, cross-modal alignment, and parameter prediction, the SVBRDF material parameters are directly output.

[0164] Optionally, SVBRDF material parameters can be directly used for high-fidelity digital rendering, material restoration, and virtual display of cultural relics.

[0165] Furthermore, the visual encoder ViT-L / 14 uses weights pre-trained on the ImageNet-21K dataset as initialization parameters and performs secondary pre-training on a large-scale cultural relic image dataset to enhance its ability to model specific textures and aging features of cultural relics.

[0166] Furthermore, the text encoder Transformer adopts a 12-layer, 12-head attention mechanism, and 768-dimensional hidden layer network structure, and is pre-trained on Chinese Wikipedia and cultural relics and archaeological literature corpora to improve its ability to understand professional terms.

[0167] Furthermore, the cross-attention alignment mechanism employs a bidirectional contrastive learning strategy to simultaneously optimize image-to-text and text-to-image retrieval tasks by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs to train the model.

[0168] Furthermore, the material parameter prediction head network introduces residual connections and layer normalization operations to improve training stability and prediction accuracy.

[0169] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0170] Based on the same inventive concept, this application also provides a system for implementing the above-mentioned intelligent reasoning method for cultural relic material parameters based on a multimodal large model. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the intelligent reasoning system for cultural relic material parameters based on a multimodal large model provided below can be found in the limitations of the intelligent reasoning method for cultural relic material parameters based on a multimodal large model described above, and will not be repeated here.

[0171] In one exemplary embodiment, such as Figure 3 As shown, a smart reasoning system 70 for cultural relic material parameters based on a multimodal large model is provided, including:

[0172] The image preprocessing module 71 can be used to acquire high-resolution digital images and structured archival text data of the cultural relic to be tested, and to preprocess the high-resolution digital images to obtain preprocessed image data.

[0173] The text encoding module 72 can be used to perform word segmentation, stop word removal, and word embedding encoding on structured archive text data to obtain encoded text data;

[0174] The multimodal feature extraction module 73 can be used to use a multimodal pre-trained large model as a feature extractor, extract visual features from the preprocessed image data through a visual encoder to obtain a visual feature vector, and extract text semantic features from the encoded text data through a text encoder to obtain a text feature vector.

[0175] The cross-modal alignment module 74 can be used to semantically align visual feature vectors and text feature vectors through a cross-attention mechanism to obtain cross-modal fused feature vectors;

[0176] The parameter prediction module 75 can be used to input cross-modal fused feature vectors into a material parameter prediction head network built on a multilayer perceptron to obtain SVBRDF material parameters.

[0177] The parameter post-processing module 76 can be used to perform parameter analysis and physical constraint processing on the SVBRDF material parameters to obtain the SVBRDF material parameter inference results of the artifact to be tested.

[0178] In one embodiment, the text encoding module includes:

[0179] The text cleaning unit can be used to clean the text content in structured archive text data to obtain cleaned text content.

[0180] The word segmentation and filtering unit can be used to perform Chinese word segmentation on the cleaned text content, obtain the word segmentation results, and remove function words and punctuation marks in the word segmentation results by comparing with the preset stop word list to obtain an effective word sequence;

[0181] Vector encoding units can be used to map each word in a valid vocabulary sequence to a fixed-dimensional word vector, and then aggregate the word vectors through average pooling to obtain the encoded text data.

[0182] In one embodiment, the cross-modal alignment module includes:

[0183] The feature matrix construction unit can be used to construct a visual feature matrix based on visual feature vectors and a text feature matrix based on text feature vectors.

[0184] The similarity measurement unit can be used to measure the cross-modal similarity between the visual feature matrix and the text feature matrix, and obtain the cross-modal similarity matrix.

[0185] The attention normalization unit can be used to normalize the cross-modal similarity matrix using a preset temperature scaling factor to obtain the cross-modal attention weight matrix.

[0186] The feature fusion unit can be used to perform weighted fusion of visual feature vectors and text feature vectors based on the cross-modal attention weight matrix to obtain a cross-modal fused feature vector; wherein, the cross-modal fused feature vector includes detailed information and text semantic prior.

[0187] In one embodiment, the attention normalization unit includes:

[0188] The expression for the cross-modal attention weight matrix is:

[0189]

[0190] In the formula, This represents the cross-modal attention weight matrix. Represents the visual feature matrix. This represents the transpose of the visual feature matrix. Represents the text feature matrix. The feature dimensions represent the visual feature vector and the text feature vector.

[0191] In one embodiment, the parameter prediction module further includes:

[0192] The sample acquisition unit can be used to acquire a sample dataset of no less than 50 cultural relics with labeled and measured SVBRDF parameter ground values; wherein, the cultural relic sample dataset includes corresponding high-resolution images, archival text, and measured SVBRDF parameter ground values ​​obtained through physical measurement equipment;

[0193] The parameter freezing unit can be used to freeze all network parameters of the visual encoder and text encoder, set the parameters of the material parameter prediction head network to a trainable state, set the initial learning rate of the material parameter prediction head network to 1e-5, use the AdamW optimizer, the batch size to 8, and the number of training epochs to 1000.

[0194] The weighted loss training unit can be used to train the material parameter prediction head network based on the cultural relic sample dataset, using the weighted mean square error function as the training loss function, and performing domain-adaptive fine-tuning on the material parameter prediction head network to obtain the domain-adaptive fine-tuned material parameter prediction head network.

[0195] In one embodiment, the weighted loss training unit includes:

[0196] The expression for the total training loss value of the training loss function is:

[0197]

[0198] in, This represents the total training loss value during the domain-adaptive fine-tuning process. This represents the weighting coefficient of the diffuse reflectance loss term. The weighting coefficients represent the normal distribution loss term. This represents the weighting coefficient of the roughness loss term. This represents the weighting coefficient of the specular loss term. This represents the mean square error loss between the predicted diffuse reflectance parameter and the measured true value of diffuse reflectance. This represents the mean square error loss between the predicted normal distribution parameters and the measured true values ​​of the normal distribution. This represents the mean square error loss between the predicted roughness parameters and the measured true roughness values. This represents the mean square error loss between the predicted specular coefficient parameters and the measured true specular coefficient values.

[0199] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the intelligent reasoning method for cultural relic material parameters based on a multimodal large model as described above.

[0200] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0201] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0202] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for intelligent reasoning of cultural relic material parameters based on a multimodal large model, characterized in that, The method includes: S1. Acquire high-resolution digital images and structured archival text data of the cultural relic to be tested, and preprocess the high-resolution digital images to obtain preprocessed image data; S2. The structured archive text data is segmented, stop words are removed, and word embedding is performed to obtain the encoded text data; S3. A multimodal pre-trained large model is used as a feature extractor. Visual features are extracted from the preprocessed image data by a visual encoder to obtain a visual feature vector. Text semantic features are extracted from the encoded text data by a text encoder to obtain a text feature vector. S4. Semantically align the visual feature vector and the text feature vector using a cross-attention mechanism to obtain a cross-modal fusion feature vector; S5. Input the cross-modal fusion feature vector into the material parameter prediction head network constructed based on multilayer perceptron to obtain SVBRDF material parameters; S6. Perform parameter analysis and physical constraint processing on the SVBRDF material parameters to obtain the SVBRDF material parameter inference results of the artifact to be tested.

2. The method according to claim 1, characterized in that, The S2 includes: S21. Clean the text content in the structured archive text data to obtain the cleaned text content; S22. Perform Chinese word segmentation on the cleaned text content to obtain the segmentation results, and remove function words and punctuation marks in the segmentation results by referring to the preset stop word list to obtain a valid word sequence; S23. Map each word in the effective vocabulary sequence to a word vector of fixed dimension, and aggregate the word vectors through average pooling to obtain the encoded text data.

3. The method according to claim 1, characterized in that, The S4 includes: S41. Based on the visual feature vector, a visual feature matrix is ​​constructed, and based on the text feature vector, a text feature matrix is ​​constructed. S42. Perform cross-modal similarity measurement on the visual feature matrix and the text feature matrix to obtain a cross-modal similarity matrix; S43. Normalize the cross-modal similarity matrix using a preset temperature scaling factor to obtain the cross-modal attention weight matrix; S44. Based on the cross-modal attention weight matrix, the visual feature vector and the text feature vector are weighted and fused to obtain the cross-modal fused feature vector; wherein, the cross-modal fused feature vector includes detailed information and text semantic prior.

4. The method according to claim 3, characterized in that, The expression for the cross-modal attention weight matrix is: In the formula, This represents the cross-modal attention weight matrix. This represents the visual feature matrix. This represents the transpose of the visual feature matrix. This represents the text feature matrix. The feature dimension represents the visual feature vector and the text feature vector.

5. The method according to claim 1, characterized in that, The training methods for the material parameter prediction head network include: S51. Obtain a dataset of no less than 50 cultural relics with labeled ground truth values ​​of measured SVBRDF parameters; wherein, the cultural relics sample dataset includes corresponding high-resolution images, archival texts, and ground truth values ​​of measured SVBRDF parameters obtained through physical measurement equipment; S52. Freeze all network parameters of the visual encoder and the text encoder, set the parameters of the material parameter prediction head network to a trainable state, set the initial learning rate of the material parameter prediction head network to 1e-5, use the AdamW optimizer, the batch size to 8, and the number of training rounds to 1000. S53. Based on the cultural relic sample dataset, the weighted mean square error function is used as the training loss function, and the material parameter prediction head network is subjected to domain adaptive fine-tuning to obtain the domain adaptive fine-tuned material parameter prediction head network.

6. The method according to claim 5, characterized in that, The expression for the total training loss value of the training loss function is: in, This represents the total training loss value during the domain adaptive fine-tuning process. This represents the weighting coefficient of the diffuse reflectance loss term. The weighting coefficients represent the normal distribution loss term. This represents the weighting coefficient of the roughness loss term. This represents the weighting coefficient of the specular loss term. This represents the mean square error loss between the predicted diffuse reflectance parameter and the measured true value of diffuse reflectance. This represents the mean square error loss between the predicted normal distribution parameters and the measured true values ​​of the normal distribution. This represents the mean square error loss between the predicted roughness parameters and the measured true roughness values. This represents the mean square error loss between the predicted specular coefficient parameters and the measured true specular coefficient values.

7. An intelligent reasoning system for cultural relic material parameters based on a multimodal large model, characterized in that, The system includes: The image preprocessing module is used to acquire high-resolution digital images and structured archival text data of the cultural relic to be tested, and to preprocess the high-resolution digital images to obtain preprocessed image data. The text encoding module is used to perform word segmentation, stop word removal, and word embedding encoding on the structured archive text data to obtain encoded text data; The multimodal feature extraction module is used to use a multimodal pre-trained large model as a feature extractor, to extract visual features from the preprocessed image data through a visual encoder to obtain a visual feature vector, and to extract text semantic features from the encoded text data through a text encoder to obtain a text feature vector. The cross-modal alignment module is used to semantically align the visual feature vector and the text feature vector through a cross-attention mechanism to obtain a cross-modal fused feature vector; The parameter prediction module is used to input the cross-modal fusion feature vector into the material parameter prediction head network constructed based on the multilayer perceptron to obtain the SVBRDF material parameters. The parameter post-processing module is used to perform parameter parsing and physical constraint processing on the SVBRDF material parameters to obtain the SVBRDF material parameter inference results of the artifact to be tested.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images