Multi-modal recognition algorithm based on images and characters

Abstract

The invention discloses a multi-modal recognition algorithm based on images and characters, which is suitable for state monitoring and protection of cultural relics. According to the algorithm, image information and character description of cultural relics are synchronously acquired, data are preprocessed by using denoising and OCR technologies, and standardized image and character feature vectors are generated. Then, performing multi-dimensional feature extraction on the image by adopting a color, shape and texture feature extraction algorithm, and performing weighted fusion on the image and character features through a multi-modal feature fusion algorithm to generate a comprehensive feature vector; in combination with historical state data, the current state of the cultural relic is evaluated based on a deep learning model, a state evaluation value and a recovery index are generated, acquisition parameters are automatically adjusted through a feedback mechanism, and a data acquisition process is optimized. According to the method, real-time dynamic monitoring of the state of the cultural relics, intelligent restoration suggestion generation and optimization of the cultural relic protection effect can be realized, and the method has relatively high adaptability and wide application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and natural language processing, and specifically to a multimodal recognition algorithm based on images and texts. Background Art

[0002] In traditional technologies for the protection and identification of cultural relics, condition monitoring and protection mainly rely on manual operations. The condition of cultural relics is recorded, inspected and evaluated manually, usually by directly observing surface changes of cultural relics, such as color fading, cracks, shape deformation, etc. However, manual inspection is not only inefficient, but also limited by the experience and subjective judgment of the observer, making it difficult to ensure the accuracy and continuity of cultural relic condition monitoring. In complex cultural relic protection scenarios, especially in the process of rescue excavation of cultural relics, manual inspection often has omissions or delays, and cannot capture subtle changes in cultural relics in a timely and accurate manner, which may lead to further damage to the cultural relics.

[0003] With the development of science and technology, intelligent cultural relics protection systems have been gradually introduced. Most of the current technical solutions rely on single-modal information such as image processing or text description to monitor and identify cultural relics. However, single-modal information is usually difficult to fully reflect the actual status of cultural relics. Taking image processing as an example, although the color, shape and texture characteristics of cultural relics can be captured, it is difficult to provide a comprehensive status assessment based solely on images when facing complex scenes or surface details of cultural relics. The text description method relies more on manual annotation, lacks dynamic monitoring capabilities, and cannot promptly reflect the changes that occur in cultural relics during the protection process.

[0004] In addition, during intelligent operation, single-modal data is difficult to capture the multi-dimensional characteristics of cultural relics under environmental changes, especially when cultural relics are exposed to the external environment for a long time, color fading, fine cracks and shape deformation are difficult to be comprehensively evaluated through a single data source. Therefore, how to integrate multi-modal data (including images and text) to improve the level of intelligent protection of cultural relics has become a problem that needs to be solved urgently by current technology. Summary of the invention

[0005] The purpose of the present invention is to provide a multimodal recognition algorithm based on images and text, which has the advantages of high-precision cultural relics status assessment, dynamic monitoring and automatic protection suggestion generation, and solves the problems that a single modality cannot fully reflect the status of cultural relics, manual monitoring is inefficient, and cultural relics protection is difficult to be intelligent.

[0006] To achieve the above object, the present invention provides the following technical solution: a multimodal recognition algorithm based on images and texts, comprising the following steps:

[0007] S1. Data collection and preprocessing: The images of cultural relics and the corresponding text label information are collected synchronously by high-definition cameras, and denoising, enhancement and OCR recognition are performed to generate standardized data as input for subsequent feature extraction;

[0008] S2, extract multi-dimensional features: Based on the standardized data generated by S1, the color, shape and texture features are extracted through a multimodal feature extraction algorithm to generate a multimodal feature vector;

[0009] S3, fusion feature vector: Based on the multimodal feature vector generated by S2, it is weightedly fused with the text feature through the multimodal feature fusion algorithm to generate a comprehensive feature vector;

[0010] S4, assess the status of cultural relics: Based on the comprehensive feature vector and historical status data generated in S3, the cultural relic status assessment algorithm is used for deep learning reasoning to generate cultural relic status assessment values ​​and restoration indicators;

[0011] S5. Output and feedback of evaluation results: Integrate the cultural relics status evaluation results generated by S4 into an evaluation report, and dynamically adjust the collection parameters through the feedback mechanism to optimize the cultural relics protection effect and data collection process.

[0012] Preferably, the step S1 includes the following sub-steps:

[0013] S1.1 Synchronous acquisition of image and text information: Synchronous acquisition of two-dimensional image information and corresponding text label information of cultural relics through high-definition cameras; the image information is in RGB format, and the text label information is a high-resolution image; S1.2 Data denoising and enhancement: Based on the original data generated by S1.1, the image and text data are denoised, and median filtering is used to remove image noise; contrast enhancement and gamma correction technology are used to improve image clarity and visibility; S1.3 OCR text recognition: Based on the text data processed by S1.2, character recognition is performed through OCR technology, and the text information in the image is converted into editable text data; S1.4 Data standardization: The image and text data obtained in S1.3 are standardized, including size unification, color space conversion and format standardization; the generated standardized data will be used as the input for multimodal feature extraction in S2.

[0014] Preferably, S2 comprises the following sub-steps:

[0015] S2.1 Color feature extraction: Based on the standardized image data generated by S1, perform color histogram analysis, calculate the color distribution characteristics of each channel, and generate a color feature vector C(I); it is expressed as: Among them, p i Represents the pixel value of each color channel in the image, I iRepresents the frequency distribution of the color histogram; S2.2 Shape feature extraction: For the standardized image data generated by S1, the edge detection algorithm is used to identify the edge of the object, and the shape analysis is performed in combination with the Hough transform to generate the shape feature vector S(I); it is expressed as: in is the result of the image gradient operation, which is used to identify edge features and perform shape analysis; S2.3 Texture feature extraction: Based on the standardized image data generated by S1, a Gabor filter is used for texture analysis to generate a texture feature vector T(I), which is expressed as: T(I) = I*G θ,λ,ψ , where * represents the convolution operation, G θ,λ,ψ is the parameter set of the Gabor filter, including direction θ, wavelength λ and displacement ψ; S2.4 multimodal feature vector generation: the color feature vector, shape feature vector and texture feature vector extracted by S2.1 to S2.3 are combined to finally generate a multimodal feature vector F feature (I), which is expressed as: F feature (I)=[C(I),S(I),T(I)].

[0016] Preferably, S3 comprises the following sub-steps:

[0017] S3.1 Loading multimodal feature vector: Based on the multimodal feature vector F generated in S2 feature (I) as input; S3.2 Load text feature vector: Load the text feature vector T generated by the text description information; S3.3 Set feature weights: Set different weight values ​​W for the multimodal feature vector and the text feature vector according to the specific needs of cultural relics identification and the importance of different features C ,W S ,W T ; Color feature weight W C Reflects the influence of color in the assessment of cultural relics status; shape feature weight W S Used to determine the shape integrity of the artifacts; texture feature weight W T Used to evaluate the surface state and texture changes of cultural relics; S3.4 Calculate the comprehensive feature vector: Through the multimodal feature fusion algorithm,

[0018] The multimodal feature vector F in S3.1 feature (I) and the text feature vector T in S3.2 are weighted fused to generate a comprehensive feature vector F fused , which is expressed as:

[0019] F fused =W C ·C(F feature )+W S ·S(F feature )+WT ·T(F feature )

[0020] S3.5 Output comprehensive feature vector: Return the comprehensive feature vector F after weighted fusion fused , used in the artifact condition assessment step of S4.

[0021] Preferably, S4 comprises the following sub-steps:

[0022] S4.1 Input comprehensive feature vector and historical status data: The comprehensive feature vector F generated by S3 is fused , as the input of the current state of the cultural relic, and at the same time load the historical cultural relic state data H for comparative analysis; the historical state data contains the characteristic records of the cultural relic in different periods, which is used to judge the change of the current state; S4.2 State change detection: Calculate the current comprehensive feature vector F fused The difference between the historical state data H evaluates the state change; it is expressed as: D = F fused -H, where D is used to measure the changes in the color, shape and texture characteristics of the cultural relics, reflecting whether the cultural relics are damaged, faded or worn on the surface; S4.3 State Assessment Model Reasoning: The comprehensive feature vector F of S4.1 is converted into fused The historical status data H is input into the deep learning model M to perform status assessment reasoning; the deep learning model M is based on a multi-layer perceptron, comprehensively analyzes multimodal features, and generates a cultural relic status assessment value E, which is expressed as: E = M (F fused ,H), the evaluation value E is the quantitative result of the current status of the cultural relics; S4.4 calculates the restoration index: generates the restoration index R according to the status evaluation value E to evaluate the restoration degree and protection effect of the cultural relics; it is expressed as: R = RecoveryIndicator(E,D), the restoration index R comprehensively combines the evaluation value E and the status difference D to show the overall restoration status, recovery rate and potential risks of the cultural relics; S4.5 outputs the evaluation results: outputs the evaluation value E and the restoration index R as the evaluation results to generate a cultural relic status evaluation report.

[0023] Preferably, S5 comprises the following sub-steps:

[0024] S5.1 Integrate the evaluation results: Integrate the cultural relic status evaluation value E and the restoration index R generated by S4 to form a comprehensive evaluation result; S5.2 Generate an evaluation report: Generate a cultural relic status evaluation report based on the comprehensive evaluation results of S5.1; S5.3 Trigger the feedback mechanism: According to the evaluation results in S5.1 and S5.2, automatically trigger the feedback mechanism to dynamically adjust the acquisition parameters; S5.4 Optimize acquisition parameters: According to the feedback results in S5.3, optimize and adjust the parameters of the acquisition equipment, including adjusting the acquisition angle, light source intensity, camera focal length and shooting mode of the high-definition camera; S5.5 Update the data acquisition process: Update the optimized acquisition parameters to the data acquisition module and redefine the data acquisition process.

[0025] Preferably, the step S4.3 further comprises the following sub-steps:

[0026] S4.3.1 Data preprocessing: After the comprehensive feature vector F fused Before the historical state data H is input into the deep learning model M, the input data is normalized; each feature component is standardized to be in the range of 0 to 1; S4.3.2 Model structure selection: According to the characteristics of multimodal features, a multilayer perceptron is selected as the model structure to process the linear combination of multimodal features; when extracting complex associations between features, a convolutional neural network is used as the model structure; S4.3.3 Model training: The model M is trained using the labeled cultural relics dataset, and the model parameters are adjusted using the back propagation algorithm; the training process aims to minimize the state evaluation error, and the cross entropy loss function is used as the optimization criterion; S4.3.4 Reasoning process: The comprehensive feature vector F preprocessed in S4.3.1 is used as the training object. fused The historical status data H is input into the trained model M for inference to generate a status evaluation value E; the inference result includes the current status score of the cultural relic and a comparative analysis with the historical status; S4.3.5 Interpretation of evaluation results: Visually interpret the status evaluation value E generated by the inference, and use feature importance analysis and interpretability methods to identify key influencing factors.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. Multimodal data fusion: The present invention effectively integrates the color, shape, and texture features of the image with the text description information of the cultural relics by setting a multimodal feature extraction and fusion algorithm, thereby achieving the effect of improving the accuracy of cultural relics status recognition;

[0029] 2. Automated monitoring and feedback mechanism: The present invention is equipped with a cultural relic status assessment algorithm, which can dynamically monitor the status changes of cultural relics in real time and automatically adjust data collection parameters through a feedback mechanism to ensure that cultural relic protection measures are more accurate and efficient;

[0030] 3. Intelligent restoration suggestion generation: The present invention can generate cultural relic restoration suggestions based on historical data by combining deep learning model reasoning, thereby improving the intelligent level of cultural relic protection;

[0031] 4. Wide applicability: The present invention can be applied to multiple scenarios such as rescue excavation of cultural relics, long-term protection and display of cultural relics, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The figure is a general flow chart of the multimodal recognition algorithm based on images and texts of the present invention;

[0033] Figure 2 The specific steps of multimodal feature extraction of the present invention;

[0034] Figure 3 This is a flow chart of the multimodal feature fusion algorithm of the present invention;

[0035] Figure 4 This is a flow chart of the algorithm for assessing the state of cultural relics of the present invention;

[0036] Figure 5 A flow chart for outputting and feeding back evaluation results of the present invention;

[0037] Figure 6 A flowchart of the sub-steps of the deep learning model reasoning of the cultural relic status of the present invention;

[0038] Figure 7 Flowchart of the extended steps for deep learning model reasoning and evaluation of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figures 1 to 7 The present invention provides a technical solution: a multimodal recognition algorithm based on images and texts. The multimodal recognition algorithm based on images and texts of the present invention first collects cultural relic images and corresponding text label information synchronously through a high-definition camera to complete the preliminary data collection and preprocessing operations. This process includes multiple specific steps to ensure that the collected data is sufficiently accurate and lays the foundation for subsequent feature extraction and fusion.

[0041] S1.1 Synchronous acquisition of images and text information: During the acquisition of cultural relics, a high-definition camera is used to simultaneously acquire the image information of the cultural relics and the text label information attached to them. The image information of the cultural relics is in a two-dimensional RGB format, which can fully display the color, shape and detail characteristics of the cultural relics; the text label information is acquired by taking high-resolution images to ensure that the description of the cultural relics is clear and visible. The purpose of this synchronous acquisition is to ensure that the image and text information of the cultural relics can correspond to each other, so that the subsequent feature extraction and fusion have higher accuracy and consistency.

[0042] This step requires the camera to have high resolution and accuracy so that the acquired cultural relic images can show enough details, such as the texture, cracks and color changes on the surface of the cultural relic. In addition, the clarity of the text label information has an important impact on the subsequent OCR recognition effect, so it is necessary to use appropriate light sources and camera angles to ensure that the text image is not distorted or blurred.

[0043] S1.2 Data denoising and enhancement: The collected image information may be disturbed by the external environment, including noise and undesirable lighting effects, so it is necessary to perform denoising and enhancement processing on the image and text data. The denoising of image data adopts the method of median filtering to eliminate noise, thereby reducing the invalid information in the data. Median filtering has a good effect of maintaining edge details, and can avoid losing key information of the cultural relic image when removing noise. For image enhancement, the present invention adopts contrast enhancement and gamma correction technology to improve the overall clarity and visual effect of the cultural relic image. Contrast enhancement mainly improves the visibility of details by adjusting the brightness range of the image, and gamma correction is used to adjust the light and dark areas of the image to make the details of the cultural relic more prominent. Through these processes, the color, shape and texture features of the cultural relic can be presented more clearly, which is convenient for subsequent feature extraction.

[0044] S1.3 OCR text recognition: For text label information, after denoising and enhancement processing, the text is recognized through OCR (optical character recognition) technology, and the text information in the image is converted into editable text data. OCR technology can recognize characters in images and digitize them for subsequent analysis and processing. In this step, the algorithm first detects the character outlines of the enhanced text image, and then compares the recognized characters with the standard character set through the character matching algorithm, and finally converts the text information in the image into editable text form. The accuracy of OCR processing depends on the clarity of the text image and the ability to resolve the characters, so the early denoising and enhancement processing is the key to ensuring the accuracy of OCR. Through this process, the description information of the cultural relics can be converted from static images to structured text data, which is convenient for comprehensive analysis with image features.

[0045] S1.4 Data standardization: After the image and text data are identified, the present invention performs standardization on the obtained data to ensure the consistency and compatibility of the data. The standardization of image data includes size unification, color space conversion and format standardization. Size unification can make the image data of cultural relics maintain the same resolution during feature extraction, avoiding the interference of different image sizes on the feature extraction algorithm; color space conversion converts the color information of different images into a unified RGB color space, which is convenient for subsequent color feature extraction; format standardization is used to convert the cultural relic images into the same format (such as JPEG or PNG) to ensure the consistency of data processing. Text data also needs to be standardized, including unified character encoding, format conversion and removal of redundant characters, to ensure that the text description information can be seamlessly combined with the image data as the input of multimodal feature extraction. In the present invention, the S2 step aims to extract color, shape and texture features of the standardized image data generated by S1 through a multimodal feature extraction algorithm, and finally generate a multimodal feature vector. This process includes analyzing and processing different attributes of cultural relic images to obtain a more comprehensive feature expression, thereby improving the accuracy and reliability of cultural relic identification. The parameters in each feature extraction step rely on real data collected from the cultural relic images, ensuring their accuracy and traceability.

[0046] S2.1 Color feature extraction: Based on the standardized image data generated by S1, the main purpose of color feature extraction is to obtain information about the color distribution in the cultural relic image by calculating the color histogram. Specifically, the color distribution characteristics of each image color channel (usually RGB channels) will be calculated separately. The calculation formula is: Among them, p i Represents the pixel value of each color channel in the image, I i Represents the frequency distribution of the color histogram; this method can effectively reflect the color changes of cultural relics images, especially for color fading or pigment degradation. These data come from RGB format images taken by high-definition cameras, and the data of each channel is analyzed by image processing software.

[0047] In practical applications, this histogram method can be used to track the color changes of cultural relics in different time periods, thereby detecting whether the cultural relics have faded, polluted or other color-related state changes. Compared with the existing manual comparison method, color feature extraction can achieve more accurate and automated color information capture, ensuring the timeliness and accuracy of cultural relic status monitoring.

[0048] S2.2 Shape feature extraction: In the shape feature extraction part, the present invention uses an edge detection algorithm (such as Canny edge detection) to identify the edge features of the cultural relics. Through edge detection, the contour information of the cultural relics and the background can be effectively separated, which is convenient for further shape analysis. The mathematical expression of the algorithm is:

[0049] S(I)=▽I, where ▽I is the result of the gradient operation of the image, which is used to identify the edge features in the image. These gradient values ​​are obtained by calculating the changes in the pixel values ​​of the image. The pixel data in the cultural relic image comes from the image taken by a high-definition camera, and the gradient information is calculated based on these pixel values ​​to ensure that the edges and shape features in the image can be clearly extracted. In order to enhance the accuracy of the shape features, the present invention also combines the Hough transform technology for further analysis of the shape. The Hough transform can identify geometric shapes (such as straight lines, circles, etc.) in the image, thereby ensuring that the shape features of the cultural relics can be completely and accurately extracted. In this way, shape features such as cracks and damaged edges of cultural relics can be accurately captured, providing reliable data for subsequent status monitoring.

[0050] S2.3 Texture feature extraction: Texture features play a vital role in the evaluation of the surface state of cultural relics. To this end, the present invention uses Gabor filter for texture analysis. Gabor filter has excellent feature extraction capabilities in both frequency domain and spatial domain, and is particularly suitable for detecting subtle changes on the surface of cultural relics. The mathematical expression of Gabor filter is: T(I) = I*G θ,λ,ψ , where * represents the convolution operation, G θ,λ,ψ is the parameter set of the Gabor filter, including direction θ, wavelength λ and phase shift ψ. These parameters are set according to the surface texture of the cultural relic to ensure that the texture characteristics of the cultural relic can be captured, including surface wear, cracks and minor damage. These parameters can be set through experimental analysis of a large amount of cultural relic image data or through existing cultural relic surface data sets to ensure the best texture detection effect. Through this texture analysis method, the system can accurately identify possible damage or weathering marks on the surface of cultural relics. Especially in cultural relics that have been preserved for a long time or exposed to the external environment, the application of Gabor filters can significantly improve the sensitivity and accuracy of detection.

[0051] S2.4 Multimodal feature vector generation: After completing the extraction of color, shape and texture features, the system fuses the extracted multidimensional features to generate the final multimodal feature vector F feature (I), which is expressed as: F feature(I) = [C(I), S(I), T(I)], this comprehensive feature vector combines the color, shape and texture features of the image, providing a comprehensive feature expression for subsequent cultural relic status assessment and analysis. The color, shape and texture data are all derived from the cultural relic images taken by high-definition cameras, ensuring the accuracy and traceability of the data. Through this multimodal feature fusion method, compared to single modal features, the present invention can more comprehensively capture the detailed changes of cultural relics, whether it is color fading, shape damage or surface texture damage, it can be fully reflected through the extraction and fusion of multidimensional features, providing strong support for the accurate identification and status monitoring of cultural relics.

[0052] In the present invention, the main task of step S3 is to transform the multimodal feature vector F generated in S2 feature (I) Fuse with the text feature vector T to generate a comprehensive feature vector F fused The fusion process sets different feature weights and combines them according to the importance of each feature in the assessment of the state of cultural relics. The final generated comprehensive feature vector will provide input for the subsequent cultural relic state assessment algorithm.

[0053] S3.1 Loading multimodal feature vector: First, the system generates a multimodal feature vector F based on the S2 step. feature (I) as the input of the fusion algorithm. The feature vector is obtained by extracting and processing the color, shape and texture features of the cultural relics, ensuring that the image information can fully and accurately represent the current state of the cultural relics. The data of these feature vectors comes from the image data taken by the high-definition camera and is generated by the feature extraction algorithm. The color feature is generated by the image pixels in RGB format, the shape feature is based on the edge detection algorithm, and the texture feature is extracted by the Gabor filter, so the parameter source of each feature is clear and traceable.

[0054] S3.2 Loading text feature vector: During the fusion process, in addition to image features, the system also needs to load text feature vector T, which is extracted from the cultural relic label information through OCR technology. By recognizing and processing text descriptions, descriptive text information related to the cultural relics is generated, including the name, historical background, material and other detailed descriptions of the cultural relics. These text feature data are structured into vector representations after OCR recognition, which is convenient for fusion with image features. The source of the text features is also clear and traceable. The text information is obtained through the text label image captured by the high-definition camera and processed by OCR to ensure that the generated text feature vector can accurately reflect the description information of the cultural relics.

[0055] S3.3 Setting feature weights: In order to ensure the rationality of feature fusion, the present invention introduces the concept of feature weights. By setting different weights W for different features (color, shape, texture and text), C ,W S ,W T ; The system can adjust the importance of features according to the specific identification requirements of cultural relics. For example, the color feature weight W C It is used to reflect the importance of color in the assessment of the state of cultural relics. In particular, this weight can be set to a higher value when it is necessary to detect color fading. The shape feature weight W S It is used to evaluate the shape integrity of cultural relics; the texture feature weight W T Used to evaluate changes in surface details and texture; text feature weights are used as a reference for descriptive information. These weights can be set based on experimental data or historical artifacts evaluation experience to ensure flexibility and accuracy in weight settings in different scenarios.

[0056] S3.4 Calculate the comprehensive feature vector: After loading the multimodal feature vector and text feature vector and setting the feature weights, the system uses the feature fusion algorithm to weight the features and generate a comprehensive feature vector F fused , the calculation formula of this process is:

[0057] F fused =W C ·C(F feature )+W S ·S(F feature )+W T ·T(F feature )

[0058] This formula shows that after each feature vector is weighted by the corresponding weight and then fused, the generated comprehensive feature vector can more comprehensively characterize the current state of the cultural relic. Through this fusion algorithm, the organic combination of image features and text features ensures the comprehensiveness and accuracy of the system in cultural relic identification and status monitoring. The setting of weights provides flexibility for cultural relic evaluation in different scenarios, ensuring that the importance of features can be adjusted according to actual needs.

[0059] S3.5 outputs the comprehensive feature vector: the comprehensive feature vector F after weighted fusion fused It is used as output for the subsequent cultural relic status assessment in S4. The feature vector contains both the image information and text description information of the cultural relic, so it can more accurately reflect the overall status of the cultural relic.

[0060] In step S4, the present invention generates a comprehensive feature vector F based on step S3. fusedThe historical status data H is used to perform deep learning reasoning using the cultural relic status assessment algorithm to generate the assessment value and restoration index of the cultural relic status. This step aims to accurately assess the current status of the cultural relic through multimodal feature analysis and comparison with historical data, and generate recommendations on the restoration and protection of the cultural relic.

[0061] S4.1 Input comprehensive feature vector and historical state data: First, the present invention converts the comprehensive feature vector F generated in step S3 into fused As the input of the current state of the cultural relic, the historical state data H is loaded at the same time. The historical state data includes the state records of the cultural relic at different time points, which usually include the color, shape and texture characteristics of the cultural relic in the past. By comparing the current comprehensive feature vector with the historical data, the system can determine whether the cultural relic has changed significantly over time. Data source: comprehensive feature vector F fused The multimodal features from step S3 are fused, and the historical status data H comes from the previous digital records of cultural relics, which usually include monitoring data during the long-term preservation of cultural relics or status data during rescue excavations.

[0062] S4.2 State change detection: After inputting the comprehensive feature vector and historical state data, the present invention detects the state change of the cultural relic by calculating the difference D between the current state of the cultural relic and the historical state. The specific calculation formula is: D = F fused -H, where the difference value D reflects the changes in the color, shape and texture characteristics of the cultural relic, helping to assess whether the cultural relic is damaged, faded or worn. Difference calculation is an important step in assessing changes in the state of cultural relics. By comparing with historical state data, the system can detect subtle state deviations.

[0063] S4.3 State Assessment Model Reasoning: In order to further quantify the state of the cultural relics, the present invention calculates the comprehensive feature vector F fused The historical status data H is input into the deep learning model M for status assessment reasoning. The deep learning model M is trained based on a large amount of historical data of cultural relics, and can comprehensively analyze the multimodal characteristics of cultural relics to generate a quantitative assessment value E of the current status of cultural relics. The calculation formula is: E = M (F fused ,H), the evaluation value E is the quantitative result of the current state of the cultural relic, reflecting the comprehensive state of the cultural relic in terms of color, shape and texture. The evaluation model infers the input data through a multi-layer neural network, and uses the historical state data H to match and analyze the current state of the cultural relic, thereby generating a more accurate evaluation result. Data source: The deep learning model M is trained based on a historical data set, including a large number of records of the state changes of cultural relics. This data set can come from historical monitoring data from different museums or archaeological institutions to ensure that the evaluation model has a high degree of generalization ability.

[0064] S4.4 Calculate the restoration index: After obtaining the status evaluation value E, the system will further generate the restoration index R, which is used to quantify the restoration speed and protection effect of the cultural relics. The specific calculation formula is:

[0065] R = RecoveryIndicator (E, D), the recovery index R is calculated based on the evaluation value E and the state difference D, reflecting the degree of recovery and protection effect of the cultural relics. If the state of the cultural relics deviates greatly from the historical data, the recovery index R will reflect that the cultural relics may need further protection measures or restoration suggestions.

[0066] S4.5 Output evaluation results: Finally, the system generates a cultural relic status evaluation report with the status evaluation value E and restoration index R as output results. The report will record the current status of the cultural relic, status change analysis and restoration suggestions in detail, providing a strong basis for decision-making for cultural relic protection personnel. By outputting the evaluation results, the system can provide dynamic and real-time monitoring data during the cultural relic protection process, and automatically adjust the collection parameters to optimize the protection measures.

[0067] Among them, in the present invention, step S4.3 mainly uses the deep learning model M to analyze the comprehensive feature vector F of the cultural relics. fused The final cultural relic status assessment value E is generated by processing the historical status data H. This process relies on deep learning algorithms to automatically analyze the status changes of cultural relics and can continuously improve the prediction accuracy through model training over time.

[0068] S4.3.1 Data preprocessing: After the comprehensive feature vector F fused Before the historical state data H is input into the deep learning model, the input data needs to be normalized. Each feature will be standardized to the range of 0 to 1 to ensure that the input data can be effectively processed in the model and avoid calculation bias caused by inconsistent feature value ranges. Data standardization: The values ​​of color, shape, and texture features are standardized to a uniform range to avoid the negative impact of differences in the order of magnitude of different feature values ​​on model training. Parameter source: The multimodal features of the cultural relics come from the S3 step, and the historical state data comes from the previously recorded cultural relic state information.

[0069] S4.3.2 Model structure selection: According to the complexity of multimodal features, the present invention selects multi-layer perceptron (MLP) as the basic model structure. MLP is a classic neural network structure, which is suitable for processing linear and nonlinear data combination tasks. When processing the comprehensive features of cultural relics, MLP can effectively extract deep patterns and laws through the connection of multi-layer networks.

[0070] Model description: MLP processes linear combinations of multimodal features through the connection of multiple layers of neurons. When extracting complex relationships between features, each layer of the network is connected through weights to form a nonlinear decision boundary.

[0071] Algorithm: The backpropagation algorithm of the neural network is used to adjust the model weights and continuously improve the accuracy of the model by minimizing the error. Backpropagation calculates the gradient of each prediction error and adjusts the weight of each neuron, thereby optimizing the prediction ability of the entire model.

[0072] S4.3.3 Model training: Model training is done by using a large number of labeled artifact datasets to adjust the model parameters. During the training process, the back propagation algorithm and gradient descent method are used to minimize the model error. After each training iteration, the model adjusts the weights according to the prediction error of the artifact status to gradually optimize the accuracy of the model.

[0073] Loss function: The cross entropy loss function is used during model training to measure the difference between the predicted value and the actual value. The smaller the loss, the better the prediction effect of the model.

[0074] Optimization algorithm: The purpose of the back propagation algorithm is to adjust the weights in the network by calculating the gradient of the error, and to propagate the error backwards to optimize the entire network. By continuously adjusting the parameters of the model, the system can gradually reduce the error in the assessment of the state of the cultural relics, and ultimately improve the accuracy of the assessment.

[0075] S4.3.4 Inference process: After the training is completed, the system will convert the preprocessed comprehensive feature vector F fused The historical status data H is input into the trained deep learning model M for inference. The model will generate a cultural relic status assessment value E based on the historical data and current features. The inference process uses the weights and structure of a multi-layer neural network to calculate the score of the cultural relic status, ensuring an accurate assessment of the status of the cultural relic;

[0076] Reasoning results: The state evaluation value E is generated through model reasoning. The evaluation results include the current state score and a comparative analysis with the historical state.

[0077] S4.3.5 Interpretation of evaluation results: Finally, the system provides a visual interpretation of the state evaluation value E generated by reasoning. Through feature analysis and interpretability methods, the system can identify the specific reasons for the change in the state of the cultural relics and provide assessments and suggestions on damage and restoration. The specific explanation includes information such as color fading and crack changes, providing visual reference data for cultural relic protection.

[0078] In the present invention, step S5 mainly integrates the cultural relic status assessment results generated in S4 into the final assessment report, and dynamically adjusts the collection parameters through the feedback mechanism to further optimize the cultural relic protection effect and data collection process. This step ensures the closed-loop nature of the entire assessment process, and through dynamic adjustment, the monitoring and protection of the cultural relic status can be continuously optimized.

[0079] S5.1 Integration of evaluation results: First, the cultural relic status evaluation value E and restoration index R generated by S4 are integrated to form a comprehensive evaluation result. The evaluation value E represents the current comprehensive status of the cultural relic, while the restoration index R is used to evaluate the restoration progress and protection effect of the cultural relic. By combining these two indicators, the system can comprehensively evaluate the overall status of the cultural relic and its changes during the protection process. Data source: The evaluation value E and restoration index R come from the deep learning model inference results of step S4, and the historical status data H is used as a comparison benchmark to calculate the restoration index R.

[0080] S5.2 Generate an assessment report: After integrating the assessment results, the system generates a complete cultural relic status assessment report. The report will detail the current status of the cultural relic, including changes in multimodal features such as color, shape, and texture. In addition, the report also includes suggestions for the restoration of the cultural relic and analysis of changes in the status of the cultural relic to help conservation personnel make decisions. Data source: The cultural relic status assessment report is based on the calculation results of S4, combined with historical status records, to describe the changes in the cultural relic and provide relevant suggestions.

[0081] S5.3 Feedback mechanism triggering: In order to optimize the accuracy of data collection and the protection effect, the present invention designs a feedback mechanism. According to the evaluation results in S5.1 and the cultural relics status assessment report in S5.2, the system will automatically trigger the feedback mechanism to dynamically adjust the acquisition parameters. If the evaluation results of the cultural relics show obvious deviations or large changes in status, the system will make real-time adjustments to the parameters in the acquisition process (such as the focal length of the camera, shooting angle, etc.) based on the feedback. Data source: The adjustment of the feedback mechanism is based on the evaluation value E and the restoration index R, and combined with historical monitoring data, the relevant parameters of the acquisition equipment are adjusted in real time.

[0082] S5.4 Collection parameter optimization: After the present invention is triggered by the feedback mechanism, the parameters of the collection equipment will be further optimized and adjusted. The specific adjustment contents include:

[0083] The acquisition angle of the high-definition camera: by adjusting the camera angle, a better image of the cultural relic can be obtained; the intensity of the light source: according to the lighting conditions, the intensity of the light source is adjusted to ensure that the details of the cultural relic are clearly presented in the image; the focal length and shooting mode of the camera: the focal length is adjusted according to the size and details of the cultural relic, and the appropriate shooting mode is switched to ensure that cultural relics of different sizes and shapes can be clearly captured. Through these optimization measures, the system can ensure the high quality of cultural relic data during each acquisition process, while reducing errors caused by improper environment or equipment settings. Data source: The acquisition parameters such as the camera angle, light intensity, and focal length are adjusted according to the actual shooting needs of the cultural relic, and the system dynamically optimizes these parameters based on real-time feedback data.

[0084] S5.5 Update data collection process: After the collection parameters are optimized, the system will update the optimized collection parameters to the data collection module and redefine the data collection process. The updated process ensures that the system can make adaptive adjustments according to the changes in the status of cultural relics, thereby maximizing the protection effect and the accuracy of data collection. The system will continue to apply the optimized collection process in the subsequent monitoring process to ensure that the data collected each time is of high quality and consistency. At the same time, as the status of cultural relics continues to change during the protection process, the system will continue to monitor and further optimize according to the new evaluation results.

[0085] Example 1

[0086] In the current field of cultural relics protection and identification, single-modal image processing or text recognition methods are mainly relied on. However, single-modal technology is often difficult to accurately assess the state changes of cultural relics in complex environments, especially when faced with multiple changes such as color fading, shape damage and surface wear of cultural relics, traditional methods are difficult to fully capture these features. Therefore, the technical purpose of the present invention is to achieve accurate state assessment and restoration analysis of cultural relics through a multimodal fusion algorithm.

[0087] The innovation of the present invention is to introduce a multimodal feature extraction and fusion algorithm, which improves the accuracy and comprehensiveness of cultural relic status monitoring by combining multimodal information of images and text. The technology can be applied to a variety of scenarios such as cultural relic protection in museums, cultural relic status assessment at archaeological sites, and rescue excavation of cultural relics. Under different environments and lighting conditions, the algorithm can flexibly adapt to achieve dynamic monitoring and real-time assessment of cultural relics.

[0088] In the following scenarios, the present invention shows excellent application effects:

[0089] Museum cultural relics protection: In low-light environments, the multimodal fusion algorithm improves the clarity and information integrity of cultural relics image features.

[0090] Archaeological excavation site: Through multimodal recognition algorithms, the status of cultural relics can be monitored in real time in harsh environments to ensure that the cultural relics are not damaged during excavation and transportation.

[0091] Cultural Relic Restoration Assessment: Combined with historical data, the system can accurately identify changes in the status of cultural relics and generate restoration suggestions based on restoration indicators, providing data support for cultural relic restoration.

[0092] Comparative experiment and detailed table display

[0093] In order to verify the performance differences of the present invention under different circumstances, we conducted several comparative experiments, including: color accuracy, shape recognition rate, texture recognition rate, recognition speed, state assessment accuracy, recovery suggestion accuracy, algorithm robustness and data processing efficiency. As shown below:

[0094]

[0095]

[0096] Experimental analysis, color accuracy: The color recognition accuracy of the present invention has reached 95%, which is significantly better than the existing single-modality method. This is because the multimodal feature fusion algorithm can maintain a high color stability under different lighting conditions. Shape recognition rate: Through the multimodal feature extraction algorithm, the shape recognition rate of the system has reached 93%, which is higher than the existing image and text recognition technology. Especially when facing damaged cultural relics, the multimodal algorithm can more accurately identify its shape features. Texture recognition rate: In texture analysis, the application of Gabor filter increases the texture recognition rate of the present invention to 91%, and it performs particularly well in identifying surface wear and cracks of cultural relics. Recognition speed: Due to the use of parallel processing mechanism and optimization of deep learning algorithm, the recognition speed of the present invention is more than 20% faster than the existing technology, making real-time application at archaeological excavation sites possible. Status assessment accuracy: Through the combination of multimodal fusion and deep learning model, the accuracy of cultural relic status assessment is increased to 92%, which is at least 12% higher than that of single-modality technology. Restoration suggestion accuracy: The accuracy of restoration suggestions depends on the comprehensive analysis of multimodal features. The present invention has higher reliability and consistency when generating cultural relic restoration suggestions. Algorithm robustness: In the case of noisy data and incomplete data, the algorithm of the present invention shows high robustness and is suitable for application in various complex environments. Data processing efficiency: Thanks to the standardization of multimodal data and the optimization of deep learning models, the data processing efficiency of the present invention reaches 97%, which is much higher than the existing single-modality method.

[0097] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal recognition algorithm based on images and text, characterized in that: The steps include: S1. Data collection and preprocessing: The images of cultural relics and the corresponding text label information are collected synchronously by high-definition cameras, and denoising, enhancement and OCR recognition are performed to generate standardized data as input for subsequent feature extraction; S2, extract multi-dimensional features: Based on the standardized data generated by S1, the color, shape and texture features are extracted through a multimodal feature extraction algorithm to generate a multimodal feature vector; S3, fusion feature vector: Based on the multimodal feature vector generated by S2, it is weightedly fused with the text feature through the multimodal feature fusion algorithm to generate a comprehensive feature vector; S4, assess the status of cultural relics: Based on the comprehensive feature vector and historical status data generated in S3, the cultural relic status assessment algorithm is used for deep learning reasoning to generate cultural relic status assessment values ​​and restoration indicators; S5. Output and feedback of evaluation results: Integrate the cultural relics status evaluation results generated by S4 into an evaluation report, and dynamically adjust the collection parameters through the feedback mechanism to optimize the cultural relics protection effect and data collection process.

2. The multimodal recognition algorithm based on image and text according to claim 1, characterized in that: The S1 step includes the following sub-steps: S1.1 Synchronous acquisition of images and text information: Synchronous acquisition of two-dimensional image information and corresponding text label information of cultural relics through a high-definition camera; the image information is in RGB format, and the text label information is a high-resolution image; S1.2 Data denoising and enhancement: Based on the original data generated by S1.1, denoise the image and text data and use median filtering to remove image noise; Improve image clarity and visibility through contrast enhancement and gamma correction technology; S1.3 OCR text recognition: Based on the text data processed by S1.2, character recognition is performed through OCR technology to convert the text information in the image into editable text data; S1.4 Data standardization: The image and text data obtained in S1.3 are standardized, including size unification, color space conversion and format standardization; the generated standardized data will be used as input for multimodal feature extraction in S2.

3. The multimodal recognition algorithm based on image and text according to claim 2 is characterized in that: The S2 comprises the following sub-steps: S2.1 Color feature extraction: Based on the standardized image data generated by S1, perform color histogram analysis, calculate the color distribution characteristics of each channel, and generate a color feature vector C(I); it is expressed as: Among them, p i Represents the pixel value of each color channel in the image, I i Represents the frequency distribution of color histogram; S2.2 Shape feature extraction: For the standardized image data generated by S1, an edge detection algorithm is used to identify the edge of the object, and the shape analysis is performed in combination with the Hough transform to generate a shape feature vector S(I); it is expressed as: in It is the result of the gradient operation of the image, which is used to identify edge features and perform shape analysis; S2.3 Texture feature extraction: Based on the standardized image data generated in S1, a Gabor filter is used for texture analysis to generate a texture feature vector T(I), which is expressed as: T(I) = I*G θ,λ,ψ , where * represents the convolution operation, G θ,λ,ψ is the parameter set of the Gabor filter, including direction θ, wavelength λ and displacement ψ; S2.4 Multimodal feature vector generation: The color feature vector, shape feature vector and texture feature vector extracted from S2.1 to S2.3 are combined to finally generate a multimodal feature vector F feature (I), which is expressed as: F feature (I)=[C(I),S(I),T(I)].

4. The multimodal recognition algorithm based on images and text according to claim 3, characterized in that: The S3 comprises the following sub-steps: S3.1 Loading multimodal feature vector: Based on the multimodal feature vector F generated in S2 feature (I) as input; S3.2 Loading text feature vector: Loading the text feature vector T generated by the text description information; S3.3 Setting feature weights: According to the specific needs of cultural relic identification and the importance of different features, different weight values ​​W are set for multimodal feature vectors and text feature vectors. C ,W S ,W T ; Color feature weight W C Reflects the influence of color in the assessment of cultural relics status; shape feature weight W S Used to determine the shape integrity of artifacts; Texture feature weight W T Used to assess the surface condition and texture changes of cultural relics; S3.4 Calculate the comprehensive feature vector: Through the multimodal feature fusion algorithm, the multimodal feature vector F in S3.1 is converted into feature (I) and the text feature vector T in S3.2 are weighted fused to generate a comprehensive feature vector F fused , which is expressed as: F fused =W C ·C(F feature )+W S ·S(F feature )+W T ·T(F feature ) S3.5 Output comprehensive feature vector: Return the comprehensive feature vector F after weighted fusion fused , used in the artifact condition assessment step of S4.

5. The multimodal recognition algorithm based on images and text according to claim 4, characterized in that: The S4 comprises the following sub-steps: S4.1 Input comprehensive feature vector and historical status data: The comprehensive feature vector F generated by S3 is fused , as the input of the current state of the cultural relic, and at the same time load the historical state data H for comparative analysis; the historical state data contains the characteristic records of the cultural relic in different periods, which is used to judge the changes in the current state; S4.2 State change detection: Calculate the current comprehensive feature vector F fused The difference between the historical state data H evaluates the state change; it is expressed as: D = F fused -H, where D is used to measure changes in the color, shape, and texture characteristics of the cultural relics, reflecting whether the cultural relics are damaged, faded, or have surface wear; S4.3 State Assessment Model Reasoning: The comprehensive feature vector F of S4.1 is fused The historical status data H are input into the deep learning model M for status assessment reasoning; the deep learning model M is based on a multi-layer perceptron, which comprehensively analyzes multimodal features and generates a cultural relic status assessment value E, which is expressed as: E=M(F fused ,H),The evaluation value E is the quantitative result of the current state of the cultural relics; S4.4 Calculate the restoration index: Generate the restoration index R based on the status assessment value E to evaluate the restoration degree and protection effect of the cultural relics; it is expressed as: R = RecoveryIndicator (E, D), the restoration index R combines the assessment value E and the status difference D to show the overall restoration status, restoration rate and potential risks of the cultural relics; S4.5 Output the evaluation results: Output the evaluation value E and restoration index R as the evaluation results and generate a cultural relic status evaluation report.

6. The multimodal recognition algorithm based on images and text according to claim 5, characterized in that: The S5 comprises the following sub-steps: S5.1 Integration of assessment results: Integrate the cultural relic status assessment value E and restoration index R generated in S4 to form a comprehensive assessment result; S5.2 Generate assessment report: Generate a cultural relic status assessment report based on the comprehensive assessment results of S5.1; S5.3 Feedback mechanism triggering: Based on the evaluation results in S5.1 and S5.2, the feedback mechanism is automatically triggered to dynamically adjust the acquisition parameters; S5.4 Optimization of acquisition parameters: According to the feedback results in S5.3, optimize and adjust the parameters of the acquisition equipment, including adjusting the acquisition angle, light source intensity, camera focal length and shooting mode of the high-definition camera; S5.5 Update data collection process: Update the optimized collection parameters to the data collection module and redefine the data collection process.

7. The multimodal recognition algorithm based on images and text according to claim 4, characterized in that: The step S4.3 further comprises the following sub-steps: S4.3.1 Data preprocessing: After the comprehensive feature vector F fused Before the historical state data H is input into the deep learning model M, the input data is normalized; each feature component is standardized to be in the range of 0 to 1; S4.3.2 Model structure selection: Based on the characteristics of multimodal features, a multilayer perceptron is selected as the model structure to process the linear combination of multimodal features; when extracting complex associations between features, a convolutional neural network is used as the model structure; S4.3.3 Model training: The model M is trained using the labeled cultural relics dataset, and the model parameters are adjusted using the back propagation algorithm; the training process aims to minimize the state evaluation error, and the cross entropy loss function is used as the optimization criterion; S4.3.4 Inference process: The comprehensive feature vector F preprocessed in S4.3.1 is fused The historical status data H is input into the trained model M for inference to generate the status evaluation value E; the inference result includes the current status score of the cultural relic and the comparative analysis with the historical status; S4.3.5 Interpretation of evaluation results: Visually interpret the state evaluation value E generated by reasoning, and use feature importance analysis and interpretability methods to identify key influencing factors.

Images