A historical material cross-validation reasoning method, device, equipment and storage medium

Abstract

The application discloses a kind of historical materials cross-validation reasoning method, device, equipment and storage medium, method includes: image data is identified to generate style factor;Style factor is converted into style vector and style vector is embedded into style self-adapting visual model;Image data is input into style self-adapting visual model, and semantic vector is obtained;From image data, a plurality of entity information is extracted to splice and encode, and semantic representation vector is obtained;Semantic vector and semantic representation vector are aligned training, form graph-text-history three modal co-embedding space;Image entity and title and colophon text entity in graph-text-history three modal co-embedding space are carried out multi-layer semantic propagation, and graph structure is obtained;Entity extracted in graph structure is classified according to semantics, and combined as several groups of triples;Triplet and the triplet recorded in historical database are compared in vector space distance, and verification result is obtained.The application significantly improves the verification capability of image content in history research.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for cross-verification reasoning based on historical data. Background Technology

[0002] In recent years, significant progress has been made in the digital organization and structured storage of historical knowledge. Structured historical databases, such as the China Biographical Database (CBDB), have become important basic data for scholars to conduct research on the relationships between historical figures and the paths of official careers, providing key support for quantitative analysis and in-depth exploration of historical research.

[0003] With the rapid development of image computing and multimodal artificial intelligence technologies, image data, especially traditional Chinese painting data, is gradually becoming an important supplementary source for verifying historical materials due to its rich historical information. As a unique cultural medium, traditional Chinese painting often contains multiple information such as time, scene, figures, clothing, geographical background, and inscriptions (such as colophons, signatures, and seals), possessing a natural multimodal and composite historical expression characteristic, capable of carrying historical context and event details from both visual and textual dimensions.

[0004] Currently, some studies have attempted to use OCR technology to automatically recognize inscriptions in traditional Chinese paintings, or to use object detection technology to identify visual content such as figures, buildings, and geographical elements in images, achieving some progress in information extraction from traditional Chinese paintings. However, existing technologies still mainly remain at the level of single-modal content recognition, and a systematic method has not yet been formed to achieve systematic association, cross-validation, and logical reasoning between the structured information implicit in traditional Chinese paintings and records in historical databases. At the same time, it has also failed to achieve historical context reconstruction based on visual interaction, and cannot provide researchers with intuitive multimodal information comparison and reasoning enhancement tools, resulting in the potential of traditional Chinese painting data in assisting historical research not being fully realized. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus, device, and storage medium for cross-verification and reasoning based on historical materials, addressing the problem that existing technologies cannot systematically associate, verify, and reason with the structured information implicit in traditional Chinese paintings and records in historical databases.

[0006] In a first aspect, embodiments of the present invention provide a method for cross-verification and reasoning based on historical materials, including:

[0007] Acquire image data, identify the dynasty and style type of the image data, and generate style factors;

[0008] The style factors are transformed into style vectors and the style vectors are embedded into the multi-head attention mechanism of the style adaptive vision model;

[0009] The image data is input into a style-adaptive visual model with embedded style vectors to obtain the semantic vector of each visual block;

[0010] A visual language model is used to generate postscript text from the image data, and a large language model is used to extract multiple entity information from the postscript text. The multiple entity information is then concatenated and encoded to obtain a semantic representation vector.

[0011] The semantic vector and the semantic representation vector are aligned and trained by jointly using a weighted alignment function and a triple loss function to form a three-modal co-embedding space of image, text, and history; wherein, the three modalities of image, text, and history include: image, text, and historical materials;

[0012] The image entities and epigraph text entities in the three-modal co-embedded space of image, text, and history are constructed into a joint semantic graph, and the joint semantic graph is subjected to multi-layer semantic propagation through a cross-modal graph model to obtain the graph structure;

[0013] The entities extracted from the graph structure are classified semantically and combined into several groups of triples;

[0014] The triples are embedded into the spatiotemporal semantic embedding space using a triple path embedding function, and the vector space distance between the triples is compared with that of the triples recorded in the historical database to obtain the verification results.

[0015] Secondly, embodiments of the present invention provide a historical data cross-verification reasoning device, comprising:

[0016] An acquisition unit is used to acquire image data, identify the dynasty and style type of the image data, and generate style factors;

[0017] An embedding unit is used to convert the style factor into a style vector and embed the style vector into the multi-head attention mechanism of the style adaptive visual model;

[0018] The input unit is used to input the image data into a style-adaptive visual model that embeds style vectors to obtain the semantic vector of each visual block.

[0019] The splicing unit is used to generate postscript text from the image data using a visual language model, extract multiple entity information from the postscript text using a large language model, splice and encode the multiple entity information to obtain a semantic representation vector;

[0020] The alignment unit is used to perform alignment training on the semantic vector and the semantic representation vector by jointly using a weighted alignment function and a triple loss function to form a three-modal co-embedding space of image, text, and history; wherein, the three modalities of image, text, and history include: image, text, and historical materials;

[0021] The construction unit is used to construct a joint semantic graph from image entities and epigraph text entities in the three-modal co-embedded space of image, text, and history, and to perform multi-layer semantic propagation on the joint semantic graph through a cross-modal graph model to obtain a graph structure;

[0022] Combining units are used to semantically classify entities extracted from a graph structure and combine them into several sets of triples.

[0023] The comparison unit is used to embed the triples into the spatiotemporal semantic embedding space through the triple path embedding function, and to compare the triples with the triples recorded in the historical database by vector space distance to obtain the verification result.

[0024] Thirdly, embodiments of the present invention provide a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the historical data cross-verification reasoning method described in the first aspect.

[0025] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the historical data cross-validation reasoning method described in the first aspect.

[0026] This invention discloses a method, apparatus, device, and storage medium for cross-validation reasoning based on historical materials. The method includes: acquiring image data and identifying the dynasty and style type of the image data, generating style factors; converting the style factors into style vectors and embedding the style vectors into a multi-head attention mechanism of a style-adaptive visual model; inputting the image data into the style-adaptive visual model with embedded style vectors to obtain a semantic vector for each visual block; generating inscription text from the image data using a visual language model, and extracting multiple entity information from the inscription text using a large language model, concatenating and encoding the multiple entity information to obtain a semantic representation vector; and then adding... A weighted alignment function and a triple loss function are jointly used to align and train the semantic vector and the semantic representation vector, forming a three-modal co-embedding space of image, text, and history. The three modalities of image, text, and history include: images, text, and historical materials. Image entities and inscription text entities within the three-modal co-embedding space are constructed into a joint semantic graph, and a multi-layer semantic propagation is performed on the joint semantic graph through a cross-modal graph model to obtain a graph structure. Entities extracted from the graph structure are semantically categorized and combined into several sets of triples. The triples are embedded into the spatiotemporal semantic embedding space using a triple path embedding function, and the vector space distance of the triples is compared with that of triples recorded in the historical database to obtain verification results. This invention effectively integrates linguistic elements such as images and inscriptions in traditional Chinese paintings through a style-adaptive visual model, a visual language model, and a large language model, establishing a fine-grained semantic alignment mechanism between images and text. This method does not rely on large-scale labeled data to uniformly model visual elements and the semantic structure of ancient texts, providing semantic support for information such as figures and scenes in traditional Chinese paintings. This invention introduces a three-modal alignment framework of image, text, and history into traditional Chinese painting data. By constructing a graph structure using a cross-modal graph model, visual entities and textual entities are uniformly mapped and contextual reasoning is performed. This enables entity association judgment and conflict detection in a multimodal space. This method is particularly suitable for handling complex situations common in traditional Chinese paintings, such as unclear historical background, ambiguous figures, and inconsistencies between image and text. It significantly enhances the verification capability of image content in historical research and helps establish a more explanatory image-based historical narrative path. This invention also provides a cross-verification and reasoning device for historical materials, a computer-readable storage medium, and a computer device, all possessing the aforementioned beneficial effects, which will not be elaborated further here. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the cross-verification and reasoning method using historical materials.

[0029] Figure 2 This is a diagram demonstrating the initial effect of the cross-verification and reasoning method based on historical materials.

[0030] Figure 3 This is a second effect demonstration diagram of the cross-verification reasoning method based on historical materials;

[0031] Figure 4 This is a schematic block diagram of a historical data cross-verification reasoning device. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] It should be understood that, when used in this specification and the appended claims, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more of its features, integrals, steps, operations, elements, components and / or collections thereof.

[0034] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0036] Please see Figures 1-3 This embodiment provides a method for cross-validation and reasoning based on historical materials, including:

[0037] S101: Acquire image data, identify the dynasty and style type of the image data, and generate style factors;

[0038] Specifically, the process involves acquiring digital images of traditional Chinese paintings to be processed. These images are sourced from digital databases of historical and cultural heritage, scanned copies of traditional Chinese paintings from museum collections, or high-resolution image files of traditional Chinese paintings used for academic research. The image formats supported include common high-resolution formats such as JPEG, TIFF, and RAW, ensuring that details such as inscriptions, seals, and visual elements in the images are clearly discernible.

[0039] Next, the acquired digitized Chinese painting images are identified in terms of dynasty and style type: if the image has clear metadata, including file naming conventions, storage path markings, or metadata annotations, the dynasty to which the painting belongs, such as Northern Song, Southern Song, Yuan, Ming, etc., and the style type, such as meticulous brushwork, freehand brushwork, line drawing, blue-green landscape, etc., are directly extracted based on the metadata; if the image lacks clear metadata, the color distribution characteristics, line drawing methods, brushstroke texture details, and frame mounting styles of the image are analyzed through auxiliary algorithms, and a comprehensive inference is made by combining the historical painting style evolution rules and expert experience database to complete the labeling of the dynasty and style type.

[0040] Based on the style types identified above, a corresponding style factor is generated through a pre-defined style extraction network. The style extraction network adopts a lightweight convolutional neural network architecture. The input is a Chinese painting image labeled with its style type. By extracting and quantifying the style features of the image, it outputs a style factor that matches the style of the Chinese painting. This style factor will be used as the input parameter of the subsequent style-adaptive visual Transformer model to dynamically modulate the model's attention mechanism to adapt to the visual feature recognition requirements of the corresponding style of Chinese painting.

[0041] In some embodiments, auxiliary algorithms analyze the color distribution characteristics, line drawing methods, brushstroke texture details, and border mounting styles of images. Combined with historical patterns of painting style evolution and expert experience databases, a comprehensive inference is made to complete the labeling of the dynasty and style type, including:

[0042] When analyzing color distribution features, an improved HSV color space segmentation algorithm is used to perform pixel-level color analysis on traditional Chinese painting images. First, K-means clustering is used to determine the dominant color combinations, extracting the pixel proportion, hue angle range, and average saturation value of each dominant color. For example, it identifies whether the proportion of azurite (hue angle 210°-240°) and malachite (hue angle 120°-150°) exceeds 60% in blue-green landscape paintings. Second, spectral reflectance simulation technology is used to distinguish pigment types. The high saturation and low transmittance characteristics unique to mineral pigments (such as azurite and cinnabar) are calculated and compared with the low saturation and high transmittance characteristics of plant pigments (such as indigo and gamboge) to generate a pigment type distribution matrix. Furthermore, for works dominated by ink, the distribution entropy of the ink grayscale levels (0-255) is extracted to determine whether it is the five-level ink gradation of "burnt, thick, heavy, light, and clear" common in Yuan dynasty literati paintings, or the uniform ink characteristics of Song dynasty court paintings.

[0043] When extracting the line drawing method, the Canny edge detection and skeleton extraction algorithms are used to construct the line feature vector. First, the average thickness (pixel width) and standard deviation of the lines are calculated. The standard deviation of the line thickness in meticulous brushwork is usually less than 3 pixels, while that in freehand brushwork can reach 8-15 pixels. Then, the radius of curvature at the turning points of the lines is analyzed. The radius of curvature of the clothing lines in Song Dynasty meticulous brushwork figure paintings is mostly 5-10 pixels, showing the regular "iron wire drawing" feature, while the radius of curvature of the lines in Ming Dynasty freehand brushwork figure paintings fluctuates greatly, often showing the natural curvature of "orchid leaf drawing". Finally, the proportion of continuous line length is counted. In line drawing works, the proportion of lines with a continuous length of more than 50 pixels is usually more than 70%, while in works combining meticulous brushwork and freehand brushwork, this proportion is mostly between 30% and 50%.

[0044] When analyzing the details of brushstroke texture, the Gray-Level Co-occurrence Matrix (GLCM) is used to extract the spatial distribution features of the brushstrokes, and the energy value, entropy value, and contrast of the texture are calculated. The entropy value of the brushstroke texture in freehand painting is usually greater than 0.8, reflecting the randomness of ink diffusion, while the entropy value of the texture in meticulous painting is mostly less than 0.4, showing a regular arrangement of brushstrokes. Through deep feature mapping of convolutional neural networks, the superposition layers of brushstrokes are identified. The "hemp fiber texture" brushstrokes in Yuan Dynasty landscape paintings are mostly superimposed with 3-5 layers, while the "axe-cut texture" in Qing Dynasty paintings are mostly thick brushstrokes with 1-2 layers. At the same time, the blurring of the brushstroke edges is calculated. The blurring of the brushstroke edges (the proportion of areas with pixel gradients less than 10) in freehand painting exceeds 40%, while that in meticulous painting is less than 15%.

[0045] When recognizing the mounting style, the mounting area is extracted using an image edge segmentation algorithm, and the width ratio of the mounting border (the ratio of border width to image width) is calculated. The border ratio of Tang Dynasty scroll paintings is mostly 1:8-1:10, while that of Qing Dynasty hanging scroll paintings is 1:5-1:6. A texture classifier is used to identify the mounting material, distinguishing between brocade (characterized by regular textures of interwoven warp and weft threads) and Xuan paper (with irregular fiber textures). The decorative patterns are identified through a pattern detection module. Simple geometric patterns (such as meander patterns) are common in Song Dynasty mounting, while floral and bird patterns are more common in Ming Dynasty mounting, and dragon and ruyi patterns are the main features in Qing Dynasty mounting. The complexity parameter of the pattern (the number of pattern elements per square centimeter) is extracted as an auxiliary feature.

[0046] Next, the extracted color, line, brushstroke, and border feature parameters are integrated into a 128-dimensional feature vector and input into a pre-trained style inference model. This model has a built-in time-series database of the evolution of historical painting styles, which includes the feature evolution trajectory of the mainstream styles of each dynasty: for example, the Northern Song Dynasty (960-1127) was dominated by meticulous brushwork and heavy colors, and blue-green landscapes, with high color saturation and regular lines; the Southern Song Dynasty (1127-1279) shifted to simplified landscapes, with concise brushstrokes and an increased proportion of ink; the Yuan Dynasty (1271-1368) was dominated by literati painting, emphasizing ink tones and calligraphic brushstrokes; the Ming Dynasty (1368-1644) had diverse styles, with the meticulous Zhejiang School and the freehand Wu School coexisting; the Qing Dynasty (1636-1912) court paintings emphasized meticulous brushwork, while the Yangzhou Eight Eccentrics were characterized by freehand flower-and-bird paintings.

[0047] Meanwhile, the style inference model utilizes an expert experience database containing feature threshold ranges from over 5000 labeled samples: for example, the azurite / malachite ratio threshold for Northern Song Dynasty blue-green landscape paintings is ≥55%, and the standard deviation of line thickness is ≤2.5 pixels; for Yuan Dynasty ink-wash landscape paintings, the ink color entropy value is ≥0.7, and the number of brushstroke layers is ≥3. Next, by calculating the cosine similarity between the feature vector of the image to be analyzed and the style templates of each dynasty, combined with historical time constraints (e.g., if a certain type of brushstroke feature only appears in the late Ming Dynasty, then Song Dynasty annotations are excluded), three high-confidence candidate results are selected.

[0048] Finally, the candidate results are verified by the multi-feature cross-validation module: if the color and line features point to the Northern Song Dynasty, but the brushstroke features do not conform to the Northern Song Dynasty pattern, the confidence of the candidate is reduced; otherwise, if all features match the style template of a certain dynasty by more than 85%, it is determined as the final annotation result, and the dynasty (such as "Northern Song Dynasty") and style type (such as "meticulous blue-green landscape") are output.

[0049] S102: Convert the style factor into a style vector and embed the style vector into the multi-head attention mechanism of the style adaptive visual model;

[0050] After generating the style factors for traditional Chinese painting, the style factors are transformed into style vectors through the following steps and embedded into the multi-head attention mechanism of the Style-AdaptiveVisionTransformer (SA-ViT) model. The specific process is as follows:

[0051] Style factors are essentially a set of quantified parameters containing the core features of traditional Chinese painting styles, encompassing dimensions such as color adaptation coefficients, line sensitivity weights, brushstroke texture feature values, and style type identifiers. To adapt to the model's attention mechanism, they need to be transformed into fixed-dimensional style vectors through a dedicated feature mapping network.

[0052] This feature mapping network employs a three-layer lightweight fully connected architecture. The input layer receives 16-dimensional quantized parameters corresponding to style factors. After a linear transformation by the first fully connected layer (16 input dimensions, 32 output dimensions), a non-linear feature map is introduced through the ReLU activation function to enhance vector expressiveness. The second fully connected layer (32 input dimensions, 64 output dimensions) further improves feature abstraction and simultaneously employs layer normalization to stabilize data distribution and avoid gradient vanishing. The output layer (64 input dimensions, 64 output dimensions) compresses vector values ​​to the 0-1 range through the Sigmoid activation function, ultimately generating a 64-dimensional style vector. This dimension is consistent with the key vector (K) and query vector (Q) dimensions of each attention head in the multi-head attention mechanism of the SA-ViT model, providing dimensionality compatibility support for subsequent embedding operations.

[0053] During the transformation process, the feature adaptability of different style types is optimized: for the meticulous style, the feature mapping network adjusts the weights of the fully connected layers to strengthen the proportion of line regularity-related parameters in the style vector; for the freehand style, the weights of the vector components corresponding to brushstroke ambiguity and color shading features are increased to ensure that the style vector can accurately represent the core style attributes of traditional Chinese painting.

[0054] In this embodiment, the calculation formula for the multi-head attention mechanism is:

[0055] ;

[0056] Among them, Q, K, and V are three vector sets obtained by linear mapping of visual image patches in the image; The style adjustment factor is generated by the style extraction network of the input image to dynamically adapt to the visual characteristics of different Chinese painting styles (such as meticulous brushwork, freehand brushwork, line drawing, etc.). This represents the dot product operation between the Q vector and the K vector, used to measure the semantic relevance of each patch to other patches; This represents the dimension of the vector and serves to normalize the numerical values. Standard scaling term. This is used to avoid the model training from becoming unstable due to excessively large values.

[0057] Furthermore, the Query vector represents the patch features that need to be focused on, the Key vector is used to calculate the relevance between the Key and the Query, and the Value vector provides the image feature values ​​corresponding to that relevance. In the Transformer structure, the calculation of attention depends on the degree of matching between the Query and the Key, which determines which features in the Value should be focused on.

[0058] By adjusting this style factor The result of multiplying the weighted coefficients by the dot product The model can adaptively adjust the sensitivity of attention distribution to different styles of images. For example, when processing freehand paintings, the model can enhance its perception of blurred lines and freehand imagery; while in meticulous paintings, it can focus more on precise boundaries and details.

[0059] Next, the Softmax function is used to normalize all attention scores into a probability distribution, ensuring that the weight of each visual patch is reasonably allocated within the overall semantic context. Finally, the normalized attention weights are multiplied by the Value matrix to obtain the semantically enhanced representation of each patch in the image, thereby improving the model's understanding of complex traditional Chinese painting contexts. The model then outputs a set of image patches with semantic labels (such as figures in official robes, palace architecture, and mountain backgrounds) along with their spatial location information, providing fine-grained visual input for subsequent image-text alignment.

[0060] S103: Input the image data into a style-adaptive visual model with embedded style vectors to obtain the semantic vector of each visual block;

[0061] Image data is segmented into multiple visual patches, each patch being transformed into query (Q), key (K), and value (V) vectors via a linear mapping. Style vectors are generated by a style extraction network and embedded as external modulation factors in the multi-head attention computation.

[0062] The model processes multiple visual patches in parallel using a multi-head attention mechanism, with each head incorporating a style vector to ensure adaptability to different style features (such as the fine detail of meticulous brushwork or the blurriness of freehand brushwork). The output is a semantic vector for each visual patch, which encodes the patch's semantic information (such as people, buildings, backgrounds, etc.) and its spatial coordinates in the image, providing fine-grained input for subsequent image-text alignment.

[0063] In some embodiments, a dataset of Chinese painting images containing four typical degradation types—mold, fading, scratches, and partial defects—is collected, covering 3,000 works from the Northern Song to the Qing dynasties in different styles (meticulous brushwork, freehand brushwork, blue-green landscape, etc.). Each image is labeled with the degradation type, coordinates of the degradation area, and a corresponding non-degradation reference image (derived from a high-resolution digital negative from a museum or a standard version restored by experts). The dataset is expanded: by simulating different degrees of degradation (such as mild fading, moderate mold, and severe scratches), a degradation intensity grading sample library is constructed to ensure that the model covers various non-ideal scenarios from slight to severe.

[0064] Next, an improved Generative Adversarial Network (GAN) architecture is adopted, consisting of a generator and a discriminator. The generator uses the U-Net++ structure (an upgraded version of U-Net, with core improvements in dense skip connections and deep supervision mechanisms), adding a multi-scale feature fusion module between the encoder and decoder to enhance the semantic connection between the degraded region and the background. The discriminator uses the PatchGAN structure (a discriminator architecture in GANs, characterized by independent discrimination of local image regions rather than global judgment of the entire image), focusing on the authenticity of local restoration. The model loss function is optimized for the characteristics of traditional Chinese painting: in addition to the traditional adversarial loss, perceptual loss (based on a pre-trained network for extracting features of traditional Chinese painting style) and structural similarity loss are added to ensure that the restored image retains both the original style and restores detailed structures (such as line continuity and color consistency).

[0065] If a diffusion model is chosen, a conditional diffusion architecture is employed, using the degradation type and degradation region as conditional inputs to achieve restoration through a progressive denoising process. Prior knowledge of traditional Chinese painting brushstroke textures is incorporated during training to constrain texture generation during the diffusion process, preventing brushstroke features in the restored area from being inconsistent with the original style.

[0066] Then, a lightweight convolutional neural network is used to analyze the pixel distribution, edge continuity, and color deviation of the image, automatically identify the degradation type (such as mold spots appearing as irregular dark patches, and scratches appearing as linear gray-scale abrupt changes), and locate the boundary coordinates of the degradation area.

[0067] Then, based on the detection results, the corresponding repair branch is called: for mold spots, a combination of texture filling and color calibration is used to remove the mold spots and restore the image content below; for faded images, color mapping curves are used for adjustment, and the original hue and saturation are restored based on the color distribution of non-degraded images of the same style; for scratches, edge smoothing and texture completion are used, and the scratch areas are filled using the line features and brushstroke rules of neighboring pixels; for local defects, semantic completion is used, and the content of the defect area that conforms to the semantics is generated based on the overall composition logic of traditional Chinese painting (such as the distance and layers of mountains and rivers, and the integrity of the limbs of figures).

[0068] After restoration, the quality is evaluated using peak signal-to-noise ratio (PSNR) and structural similarity index. If the indicators do not meet the preset thresholds (PSNR ≥ 35dB, structural similarity index ≥ 0.9), the restoration parameters are iteratively optimized until the quality requirements are met. Finally, the restored and enhanced high-definition Chinese painting image is output, which is adapted to the input size and feature extraction requirements of the SA-ViT model.

[0069] S104: Generate epigraph text from the image data using a visual language model, extract multiple entity information from the epigraph text using a large language model, and concatenate and encode the multiple entity information to obtain a semantic representation vector;

[0070] Specifically, a visual language model suitable for classical Chinese text recognition is selected. This model is a pre-trained visual language model based on the Donut or TroOCR extended architecture, which directly generates the inscription text through end-to-end image-text modeling. First, the inscription area of ​​the input digitized Chinese painting image is preprocessed: an image segmentation algorithm is used to automatically locate the area where the inscription is located, eliminating interfering elements such as the main scenery and seals in the painting, and retaining the complete inscription text area; the located inscription area is adjusted to the input size adapted to the model, grayscale processing is used to enhance the contrast between the text and the background, and adaptive threshold binarization is used to eliminate noise interference caused by yellowing of paper and ink bleeding, ensuring that the strokes of the text are clear and distinguishable.

[0071] Next, the preprocessed inscription area image is input into the visual language model. The model extracts visual features of the inscription text through the encoder, including information such as character stroke outlines, font structure, and line spacing. The decoder generates continuous, structured inscription text based on the extracted visual features and a pre-trained ancient Chinese corpus. The model training phase incorporates calligraphy font samples from various dynasties and ancient Chinese syntax rules, enabling it to robustly handle issues such as handwriting distortion, irregular typesetting, variant characters, and blurred handwriting, directly outputting fluent and complete inscription text without the need for additional text detection and recognition stages. For example, for a traditional Chinese painting image containing multiple inscriptions of varying font sizes, the model can automatically distinguish paragraph boundaries and generate corresponding text content according to the spatial order of the inscriptions in the image.

[0072] The generated inscription text is then input into a large language model optimized for classical Chinese. This model has built-in functions for ancient Chinese semantic parsing and named entity recognition. The entity information is extracted through the following steps: First, the model performs sentence segmentation and syntactic analysis on the inscription text, identifies grammatical components such as subject-verb-object structure, time adverbs, and place adverbs in the text, and divides them into semantic units. Then, the named entity recognition model is called. Based on pre-trained ancient Chinese entity annotation samples, the core entity information in the text is identified and extracted to construct a "semantic structure graph of the inscription". This graph is used to support subsequent multimodal alignment and semantic verification tasks. The core entity information includes personal names, place names, dynasty names, time expressions, event names, and seal-related identifiers.

[0073] During the extraction process, the model incorporates historical knowledge graphs for entity calibration. For example, it maps font sizes and aliases appearing in the epigraphs to corresponding standard personal names, transforms ancient place names into unified geographical identifiers, and standardizes non-standardized time expressions (such as a specific year, month, and day) into a standardized time format. The final output includes multiple structured entity information sets, each containing an entity type label and entity content, ensuring the accuracy and standardization of the extracted entity information and meeting the requirements for subsequent encoding and semantic alignment.

[0074] The extracted entity information was then systematically concatenated, following the priority of entity types and the semantic logic of the text: entities were arranged in the order of person name, place name, time, event, and other entities. Entities of the same type were linked sequentially according to their order of appearance in the epigraph text, ensuring that the concatenated entity sequence preserved the semantic relationships of the original text. For example, the extracted person name, time, and place entities were concatenated in the logical order of "person-time-place" to form a coherent entity sequence.

[0075] Next, the concatenated entity sequence is input into the encoding layer of the pre-trained classical Chinese language model. This encoding layer employs a Transformer architecture, using a multi-head attention mechanism to capture semantic relationships between entities and transform the textual features of each entity into a high-dimensional vector representation. During the encoding process, the model integrates entity attribute information from the historical knowledge graph. For example, when encoding personal names, it incorporates related features such as the person's dynasty and identity to enhance the semantic expressiveness of the vectors. Subsequently, pooling operations are used to aggregate the vector representations of all entities to obtain a fixed-dimensional global semantic representation vector.

[0076] To adapt to subsequent multimodal alignment tasks, the aggregated vectors are mapped to a unified dimension through a linear projection layer. After layer normalization to stabilize the vector distribution, the semantic representation vector of the epigraph text is finally obtained. This vector contains the core information of all entities in the epigraph and the semantic relationships between entities, enabling subsequent semantic alignment and matching with the semantic vectors of image visual blocks in the same dimensional space.

[0077] In some embodiments, a visual language model is used to generate inscription text from image data, and a large language model is used to extract multiple entity information from the inscription text. The multiple entity information is then concatenated and encoded to obtain a semantic representation vector, followed by:

[0078] Lexical analysis, syntactic parsing, and nested named entity recognition technologies are used to extract personal names, place names, dynasties, time expressions, and event structures from the postscript text;

[0079] Based on personal names, place names, dynasties, time expressions, and event structures, semantic entities are represented by nodes, and syntactic and semantic dependencies are represented by edges, resulting in a semantic structure graph of the epigraph.

[0080] Specifically, a lexical analysis tool optimized for classical Chinese is used to preprocess the inscription text. First, the morphemes are split according to the semantic logic of ancient Chinese, distinguishing single-word words, two-word words, and multi-word phrases. For example, phrases containing dynasty identifiers are split into dynasty roots and auxiliary morphemes. Subsequently, the morphemes after splitting are tagged with词性, and the tagging categories include nouns, verbs, time words, place name specific words, etc. The proper nouns related to people's names, place names, and dynasties are marked as the key, providing a basis for subsequent entity recognition.

[0081] Then, for syntactic parsing and modeling, a dependency syntactic analysis model based on the Transformer architecture is introduced to model the syntactic structure of the inscription text. The model constructs a syntactic dependency tree by analyzing the dependency relationships between words, such as subject-predicate, verb-object, modifier-head, adverbial-head, complement-head, etc. For example, in the sentence "In the third year of Zhenghe, it was bestowed upon Cai Jing", the verb-object relationship between "bestowed" and "Cai Jing" and the adverbial-head relationship between "In the third year of Zhenghe" and "bestowed" are identified to clarify the syntactic association path between entities. For complex sentence structures, the model distinguishes the syntactic dependencies of the main clause and the subordinate clause through hierarchical parsing to ensure the accuracy of relationship recognition under complex sentence patterns.

[0082] Next, for nested named entity recognition, a nested named entity recognition model combining bidirectional LSTM and CRF is adopted to accurately extract the common entity nesting phenomena in the inscription text. The model first identifies the outer-layer entities, such as the nesting relationship between "Northern Song Dynasty" (dynasty entity) and "Xuanhe period" (time entity) in "During the Xuanhe period of the Northern Song Dynasty"; then it deeply analyzes the inner-layer entities, such as the nesting structure between "Xue'an Puguang" (person name entity) and "Grand Scholar of Zhaowen Hall" (identity-related entity) in "Xue'an Puguang, Grand Scholar of Zhaowen Hall". Through multiple rounds of iterative recognition, the four core entities of people's names, place names, dynasties, and time expressions in the text are comprehensively extracted, while capturing the hierarchical associations of entities.

[0083] Subsequently, based on the results of lexical analysis and syntactic parsing, the event structure of the inscription text is extracted. The event structure takes the core verb as the trigger word, such as "bestowed", "inscribed", "composed", etc., and is constructed in combination with its associated subject, object, time adverbial, and place adverbial, including five core elements: event subject, event action, event object, event occurrence time, and event occurrence place. For example, the event structure is extracted from "On the eighth day of the fourth month of the leap year in the third year of Zhenghe, it was bestowed upon this minister Cai Jing": the subject is Emperor Huizong of Song, the action is bestowed, the object is Cai Jing, the time is the eighth day of the fourth month of the leap year in the third year of Zhenghe, and the place is the Inner Palace, forming a complete event structure unit.

[0084] Next, nodes are defined and initialized. The extracted entities (names, places, dynasties, and time expressions) and event structures are used as nodes in the semantic structure graph. Each node contains attributes such as node ID, entity type, entity content, and event element set. For example, the attributes of the name node include ID, type (name), content (Cai Jing), and associated dynasty (Northern Song Dynasty); the attributes of the event structure node include ID, type (event), core action (bestowed), subject (Emperor Huizong of Song), and object (Cai Jing), thus completing the initialization of all nodes.

[0085] The definition and construction of edges are based on syntactic dependencies and semantic associations to define the edges between nodes. The types of edges include two categories: syntactic dependency edges and semantic dependency edges.

[0086] Syntactic dependency edges are constructed based on the results of dependency parsing. For example, a verb-object relation edge is constructed between the name node (Cai Jing) and the verb node (Ci), and an adverbial-head relation edge is constructed between the time node (the eighth day of the fourth intercalary month of the third year of Zhenghe) and the verb node (Ci).

[0087] Semantic dependency edges are constructed based on the semantic association between entities and events. For example, an affiliation edge is constructed between the dynasty node (Northern Song Dynasty) and the personal name node (Cai Jing), a location association edge is constructed between the event node (the painting bestowal event) and the location node (the Forbidden City), and a temporal association edge is constructed between the time node and the event node.

[0088] Subsequently, a semantic structure graph for the epigraphs was constructed using a graph database storage format. Nodes were used as core entity units, and edges as connecting links, forming a visual network structure. During construction, a cyclic verification mechanism was used to eliminate redundant edges and erroneous associations. For example, when a syntactic dependency edge and a semantic dependency edge had a logical conflict, the rationality was verified using a historical knowledge graph, retaining edges that conformed to semantic logic. Simultaneously, node clustering optimization was performed, merging different representations of the same entity (such as font size and real name) into a single node, associating all its attributes and edges to ensure the simplicity and accuracy of the structure graph.

[0089] The final generated semantic structure diagram of the postscript clearly presents the type, attributes, and syntactic and semantic relationships of each entity in the postscript text, providing structured semantic support for subsequent cross-modal alignment with visual entities in images and for association verification with historical databases.

[0090] S105: The semantic vector and the semantic representation vector are aligned and trained by a weighted alignment function and a triple loss function to form a three-modal co-embedding space of image, text and history; wherein, the three modalities of image, text and history include: image, text and historical materials;

[0091] In this embodiment, a "three-modal alignment" mechanism of image, text, and history is introduced as the core technology. Based on the CLIP-type multimodal machine learning model's image-text comparison learning framework, historical data constraint signals are added to propose a constraint-based multimodal alignment strategy for traditional Chinese painting. Specific methods include:

[0092] Image patches and text phrases are encoded separately, and cross-modal semantic alignment is achieved through a multi-head attention mechanism. This not only models the entire image and the entire text, but also performs fine-grained mapping of sub-regions and words. During the training phase, structured information about people in historical databases (such as CBDB) (including attributes such as official rank, era, and geography) is introduced as an auxiliary supervision signal, making it a key alignment target whether person A in the image and person B in the inscription can be mapped to the same entity C in the database.

[0093] Specifically, by jointly aligning and training the semantic vector and semantic representation vector using a weighted alignment function and a triple loss function, a three-modal co-embedding space of image, text, and history is formed, including:

[0094] After encoding the image patch sequence and the epigraph phrase representation separately, the semantic vector and semantic representation vector are aligned and trained using the following formula:

[0095] ;

[0096] in, Represents the dot product of vectors; This indicates the structured similarity of whether a text / image pair co-maps to a certain historical entity; This represents the attention weights in adaptive learning; Represents a semantic vector; Represents a semantic representation vector.

[0097] In some embodiments, the method of jointly aligning and training semantic vectors and semantic representation vectors using a weighted alignment function and a triple loss function to form a graph-text-history trimodal co-embedding space further includes:

[0098] Calculate the triplet loss between the semantic vector and the semantic representation vector;

[0099] Calculate the alignment loss between the semantic vector and the semantic representation vector;

[0100] Calculate the history consistency supervision loss between semantic vectors and semantic representation vectors;

[0101] The training parameters were tuned using triplet loss, alignment loss, and history consistency supervision loss.

[0102] Specifically, during training, the following triple loss function is jointly introduced to optimize cross-modal alignment performance:

[0103] ;

[0104] in, The triplet loss is used to ensure good discriminative power between image-text matching pairs. Specifically, this loss encourages the model to bring the correct image-text matching pairs (image patch and its corresponding text phrase) closer together, while pushing the distance between unmatched pairs further apart, thereby constructing a clearer semantic boundary between image and text in the embedding space. This loss function controls the fine-grained alignment accuracy between local visual patches and text phrases in an image. In traditional Chinese painting, a single inscription may involve multiple images (such as landscapes, palaces, and figures), and the image may also contain multiple corresponding semantic regions. This loss function guides the model to match these detailed elements one by one, ensuring that the visual and text are aligned at the local semantic level, thus improving the accuracy of image-text understanding. The historical consistency supervision loss serves to supervise whether the image-text pair can be correctly matched with entities (triplets, such as person-time-place) in the structured historical database. For example, when an image identifies a person wearing Northern Song Dynasty official robes, and the accompanying text mentions "Wang Shen" and "the third year of Zhenghe," this loss term will help the model learn that this image-text content should match the corresponding record in the database for "Northern Song Dynasty - Wang Shen - Zhenghe period," thereby establishing cross-modal and cross-source information consistency.

[0105] also, and Two adjustable weighting coefficients are used to control the contribution of alignment loss and historical supervision loss to the total loss function. By dynamically setting these two parameters, the three objectives can be balanced and optimized according to data characteristics or different training stages. This triple loss joint mechanism supports the model in automatically mining the deep correlation between semantic imagery and inscription text in images, and further constrains and verifies the model output through structured historical information, effectively improving the accuracy and historical consistency of cross-modal semantic understanding.

[0106] In some embodiments, a degradation invariance constraint term is added to the original loss function for SA-ViT model optimization. This constraint term is implemented through contrastive learning, requiring the model to extract visual features from a complete version and a slightly degraded version of the same Chinese painting as closely as possible. Specifically, the feature representations of the normal image and the slightly degraded image are aligned in the feature space, enabling the model to learn to ignore common degradation interference.

[0107] Specifically, a degradation invariance constraint is defined, with the core objective of guiding the model to learn visual feature representations that are insensitive to slight degradation. The constraint is constructed based on the feature distance between the "original clear version" and the "slightly degraded version" of the same Chinese painting image, as shown in the following formula:

[0108] ;

[0109] in, For the clear version features of the i-th visual patch, Let f(⋅) be the feature of the i-th visual patch after slight degradation (such as mild fading or minor scratches), f(⋅) be the feature extraction function of the SA-ViT model, |||2 be the L2 norm, and N be the total number of visual patches in the image. This constraint term minimizes the feature distance between sharp and slightly degraded patches, making the model insensitive to feature perturbations caused by slight degradation.

[0110] Next, the degradation invariance constraint term is integrated into the original loss function system of SA-ViT, and the reconstructed total loss function is:

[0111] ;

[0112] Among them, L triplet For triplet loss, L align For local alignment loss, L hist The historical consistency supervision loss is λ3, which is the weight coefficient of the degradation invariance constraint term (experiments have shown that setting it to 0.3 yields the best results, ensuring degradation invariance without affecting the original feature extraction accuracy).

[0113] During the training of SA-ViT, restored and enhanced images and slightly degraded image samples are input simultaneously. The gradient is backpropagated through the reconstructed total loss function. On the one hand, the restored and enhanced images ensure that the model learns complete and accurate style features and semantic information; on the other hand, the degradation invariance constraint guides the model to ignore noise interference caused by slight degradation and focus on core semantic features (such as human contours, building structures, and brushstroke patterns). A dynamic learning rate strategy is adopted during training. In the early stage, the optimization of the original loss term is the main focus, and in the later stage, the weight of λ3 is gradually increased to enhance the degradation invariance generalization ability.

[0114] After inputting a Chinese painting image with physical degradation, the image restoration and enhancement preprocessing module first performs degradation type detection, targeted restoration, and quality calibration, outputting a clear and stylistically accurate Chinese painting image. This image is then input into the SA-ViT model with degradation invariance constraints. The model uses a style factor modulated attention mechanism, combined with degradation-insensitive feature learning, to accurately extract semantic patches (such as figures in official robes, palace buildings, and mountain backgrounds) from the image, outputting visual feature vectors with semantic labels and spatial locations.

[0115] S106: Construct a joint semantic graph by embedding image entities and epigraph text entities in the three-modal co-embedded space of image, text, and history, and perform multi-layer semantic propagation on the joint semantic graph through a cross-modal graph model to obtain a graph structure;

[0116] Specifically, this embodiment uses a Cross-modal Graph Transformer (CMGT) for joint entity modeling and alignment. The model represents image entities (such as clothing and buildings) and text entities (such as names, times, and locations) together as nodes in a joint graph structure, and learns context-sensitive semantic relationships between entities by calculating multi-hop semantic paths between nodes through multiple layers of Transformers.

[0117] The following section provides a detailed explanation of how to construct a joint semantic graph from image entities and epigraph text entities within the trimodal embedding space of image, text, and history:

[0118] From the integrated co-embedding space of image, text, and history, two core entities are extracted: one is image entities, including visual block semantic entities output by the style-adaptive visual model, covering clothing, architectural styles, landscapes, seal styles, etc., with each image entity accompanied by a corresponding semantic vector and spatial location information; the other is inscription text entities, including personal names, place names, dynasties, time expressions, event structures, etc., extracted and calibrated by a large language model, with each text entity associated with node attributes and semantic vectors in the inscription semantic structure graph. Simultaneously, reference entities from the historical database are introduced, including structured entities such as historical figures, geographical divisions, and dynastic sequences potentially related to the images and inscription content. Their semantic vectors are generated from structured fields in the database, ensuring consistency with the dimensions of the three-modal co-embedding space.

[0119] Next, the extracted image entities, inscription text entities, and historical reference entities are all used as nodes in the joint semantic graph. Each node is assigned a unique identifier ID, containing information such as node type (image / text / historical reference), core content, semantic vector, and associated attributes. For example, image entity nodes are labeled with specific categories such as "person's clothing - official robes" and "building - palace," text entity nodes are labeled with attributes such as "person's name - Cai Jing" and "time - Zhenghe 3rd year," and historical reference entity nodes are labeled with structured information such as "historical figure - Wang Ximeng" and "dynasty - Northern Song Dynasty," thus completing the initial configuration of all nodes.

[0120] Then, based on the cosine similarity of the semantic vectors of each entity in the trimodal co-embedding space, and combining the spatiotemporal association rules and logical association rules between entities, edges between nodes are constructed and their weights are calculated. These edges include semantic association edges, spatiotemporal association edges, and logical association edges.

[0121] Semantic association edges: connect entity nodes whose semantic vector cosine similarity is higher than a preset threshold (0.75). For example, between the “Cai Jing” node in the epigraph text and the “official robe figure” node in the image, and the “Cai Jing” reference node in the historical database, semantic association edges are constructed due to their high semantic correlation, and the weight is the similarity calculation result.

[0122] Spatiotemporal related edges: connect entity nodes that conform to historical time sequence and geographical range constraints, such as the time node "Zhenghe 3rd year" and the dynasty node "Northern Song" and the location node "Bianjing". Spatiotemporal related edges are constructed based on historical spatiotemporal logic, and the weights are quantified according to spatiotemporal matching (the weight of a complete match is 1.0, and the weight of a partial match decreases proportionally).

[0123] Logical association edges: connect entity nodes that have causal, subordinate, or event-related relationships, such as the event node "bestowing a painting" with the subject node "Emperor Huizong of Song", the object node "Cai Jing", and the time node "the third year of Zhenghe". Logical association edges are constructed based on the semantic structure of the postscript and historical logic, and the weights are assigned according to the degree of association (core association weight 0.9-1.0, indirect association weight 0.5-0.8).

[0124] Subsequently, a graph structure storage format is adopted to organize the nodes and edges into an initial joint semantic graph according to the above definition, ensuring that the association between nodes accurately maps the semantic associations, spatiotemporal constraints and logical dependencies in the trimodal co-embedding space, providing a structured foundation for subsequent semantic propagation.

[0125] Next, the Cross-Modal Graph Transformer (CMGT) is used as the core processing model to perform hierarchical semantic propagation on the constructed joint semantic graph, realizing entity relationship reasoning and cross-modal verification. The specific process is as follows:

[0126] The cross-modal graph model is configured with 8 Transformer encoding units, each layer having 12 multi-head attention heads. The input consists of the node feature matrix and edge type matrix of the joint semantic graph. During the initialization phase, the cross-modal initial vector after node concatenation is used as the input feature, and the edge type matrix is ​​encoded and labeled according to three types of relationships: logical, temporal, and spatial, providing a basis for calculating the relationship weights in the semantic propagation of each layer.

[0127] The three core operations of each Transformer coding unit: Each coding unit performs the following three operations sequentially to achieve the progressive propagation of semantic information:

[0128] Different weight calculation factors are introduced for different edge types. For logical relationship edges, the weight is calculated based on the cosine similarity of semantic vectors between entities and the association strength in the historical knowledge graph; for temporal relationship edges, the weight is quantified by combining the continuity of historical time sequence and matching metrics; for spatial relationship edges, the weight is determined based on the relevance of geographical divisions and the adaptability to historical scenarios. This approach allows for precise differentiation of the importance of different types of relationships, improving the targeting of semantic propagation.

[0129] Next, taking the current node as the core, the feature information of directly related nodes (1-hop neighbors) is first aggregated, and then gradually expanded to indirectly related nodes (2-hop neighbors - 3-hop neighbors). During the fusion process, the feature contribution of neighboring nodes is allocated according to the edge weight. The higher the weight of the neighboring node, the greater the proportion of its features in the fusion, ensuring that the core related information is retained while taking into account the auxiliary information in the indirect associations.

[0130] The fused neighborhood features are then fused with the initial features of the current node through residual connections. After layer normalization, the fused features are input into a feedforward neural network to perform a nonlinear transformation. The feedforward neural network uses the GELU activation function to enhance the nonlinear expressive power of the features, and finally outputs the updated node feature vector, completing a single round of semantic propagation.

[0131] Through iterative propagation across 8 layers of coding units, the model achieves semantic processing in different dimensions through hierarchical division of labor at the bottom, middle, and top levels:

[0132] The lower-level network (layers 1-2) focuses on capturing local structure and mining direct correlation features between nodes. For example, it identifies the correspondence between the objects held by officials and their official titles, and the association between a certain action in an inscription and the directly involved subject, providing support for basic semantic relationship modeling.

[0133] Mid-level network (layers 3-5): Focuses on spatiotemporal context fusion, associating entities with cross-type entities such as their dynasty, activity location, and time node. For example, establishing spatiotemporal binding between officials in their official robes and the Northern Song Dynasty and Bianjing (Kaifeng), integrating the temporal relationship between inscription time expressions and historical events, and constructing a multi-dimensional semantic association network;

[0134] High-level networks (layers 6-8): Enable complex semantic inference by combining image visual features, inscription text descriptions, and structured information from historical databases to complete entity authentication and relationship confirmation. For example, by comparing the style features of official robes and the names described in the inscriptions with records of officials from the Song Renzong era in historical databases, the model can determine whether the person in the official robes in the image is the official described in the inscription. Simultaneously, it automatically detects potential contradictions. For instance, if the era corresponding to the style features of the official robes deviates from the time recorded in the inscription, the model will reduce the association weight of the relevant nodes and mark it as a spatiotemporal conflict.

[0135] After multi-layer semantic propagation, each node in the joint semantic graph absorbs cross-modal features and related information from the global context. The edge weights between nodes accurately reflect the strength of relationships and historical consistency between entities. The final output graph structure includes optimized node feature matrices, edge weight matrices, and relationship type annotations, clearly presenting the logical, temporal, and spatial relationships between image entities and epigraph text entities, as well as their consistency with historical facts. This provides highly reliable structured data support for subsequent extraction of "person-time-place" triples and cross-validation with historical databases.

[0136] S107: Classify the entities extracted from the graph structure according to their semantics and combine them into several sets of triples;

[0137] Specifically, by analyzing the type attributes of each node and its connectivity within the graph structure, all entity nodes are divided into three main categories: people, time, and location. People entities include not only visually recognizable figures but also names extracted from the inscription text. Time entities encompass historical dates, dynasty names, and stylistic features inferred from visual characteristics extracted from the inscriptions. Location entities include geographical names appearing in the text and scene nodes with regional characteristics from images.

[0138] After completing entity classification, the system initiates a triplet combination mechanism. This mechanism searches for association paths that conform to the person-time-place pattern based on the semantic edge relationships already established in the graph structure. For each candidate person node, the system explores the time and place nodes connected to it along the relationship edges, forming an initial candidate set of triples.

[0139] Next, the rationality of each candidate triple is evaluated by calculating the semantic consistency of the three entity nodes in the embedding space. For example, it verifies whether a Song Dynasty figure node forms a semantically consistent triple with a Song Dynasty time node and a typical Song Dynasty geographical node. At the same time, it identifies and eliminates triple combinations with obvious spatiotemporal contradictions, such as the erroneous association between a Tang Dynasty figure and a Song Dynasty time node.

[0140] For complex situations, a multi-path fusion strategy is adopted. When a character node is associated with multiple time or location nodes through different paths, the weight and confidence of each path are comprehensively considered, and the most reliable triple combination is selected.

[0141] After constructing the triples, a unified vector representation is generated for each triple. This vector is calculated from the feature vectors of the three constituent entities using a specific fusion function, preserving the independent semantics of each entity while encoding the strength of the association between them. These triple vectors are organized into a structured set of triples, serving as direct input for subsequent comparison and verification with historical databases.

[0142] The entire process of constructing the triples fully considers the characteristics of traditional Chinese painting data, ensuring both the formal standardization of the triples and the semantic rationality through multi-dimensional verification. This structured information organization lays a solid foundation for subsequent cross-modal verification and visualization, enabling the historical information implicit in traditional Chinese paintings to be represented and reasoned about in a machine-understandable and processable form.

[0143] S108: The triples are embedded into the spatiotemporal semantic embedding space using the triple path embedding function, and the vector space distance of the triples is compared with that of the triples recorded in the historical database to obtain the verification result.

[0144] After obtaining several sets of triples, these triples are then projected into a spatiotemporal semantic embedding space using a triple path embedding function, and compared with existing record triples in a historical database (such as CBDB) using vector space distance. If the distance between matching triples is small, they can be considered to be highly semantically consistent, thus preliminarily verifying whether the information reflected in the graph and the caption is authentic and verifiable. This mechanism also supports flexible matching, and can display analysis labels such as "historical data unverified" or "image and text may conflict" when some information is missing.

[0145] The formula for calculating the ternary path embedding function is as follows:

[0146] ;

[0147] Where P represents a person; T represents time; and L represents location; It is an embedding function for human entities, used to encode the multimodal features of a person in an image or epigraph text into a unified vector representation; It is a temporal embedding function that maps temporal representations to a shared temporal semantic space; This is a location embedding function that captures the correlation between location-related visual imagery and textual descriptions in traditional Chinese paintings. The three sub-functions can be implemented using a general embedding modeling approach. Specifically, the textual features of each entity can be encoded first (e.g., using language models to embed names, time expressions, and place names), and then combined with structured information (such as standardized historical time, geographic database location information, and personal identity attributes) for feature fusion. Finally, a multilayer perceptron (MLP) method is used to map the embeddings of each entity class to a unified semantic space dimension to support joint modeling and vectorized representation of the three.

[0148] Next, the output vectors of the three functions need to be mapped to the same shared semantic space, for example, all being 512-dimensional vectors, which can be represented by a unified projection layer or a pre-trained shared Transformer encoder. Finally, the vectors of the three functions are added together to form the function. This enables the fusion representation of the three elements—person, time, and place—in a unified spatiotemporal semantic space.

[0149] This design enables the model to perceive the semantic coupling relationship between "who, when, and where," providing a more solid embedding foundation for image-text history consistency verification, spatiotemporal mismatch identification, and image semantic restoration.

[0150] Next, the triples will be compared with the triples recorded in the historical database using vector space distance to obtain the verification results, which specifically include:

[0151] Calculate the cosine distance between the triplet and the triplets recorded in the historical database;

[0152] Determine whether the cosine distance is less than a predetermined threshold;

[0153] If the cosine distance is less than a predetermined threshold, the triple is determined to be a successful match with the triple recorded in the historical database.

[0154] If the cosine distance is greater than or equal to a predetermined threshold, the triplet is determined to be a mismatch with the triplet recorded in the historical database.

[0155] More specifically, if the triples form a valid combination, their embedding distance should be equal to the embedding distance of a real person's record in the historical database. Embedded To determine proximity, this embodiment uses the following formula:

[0156] ;

[0157] in, It is the vector representation of triples after processing by a unified embedding function; Represents a candidate entity in the historical database (For example, the record of a real historical figure) Embedded representation in the same semantic space, which can be obtained based on structured fields (such as dynasty, place of birth, length of service, etc.) through encoding networks or retrieval matching; then, through computation and The cosine distance between the images and historical records is used to measure the similarity between the images and historical records in the semantic space. The semantic matching threshold represents the maximum acceptable embedding error range. If the cosine distance between two embeddings is less than the semantic matching threshold (i.e., the predetermined threshold), a match is considered valid. If the cosine distance between two embeddings is greater than or equal to the semantic matching threshold, a match is considered invalid. This criterion is used to evaluate the temporal and spatial semantic consistency between text, images, and historical facts in a diagram. This mechanism also supports flexible matching scenarios with incomplete alignment and can automatically alert users to potential risks such as "conflict between text and image" or "unverified historical data."

[0158] After initial matching, the results are compared with biographical information in a structured historical database to extract attributes such as tenure, activity locations, and social status, constructing a ternary spatiotemporal path graph of "person-time-place." The model further determines whether there is semantic consistency between the visual features of the figures in the image and the inscription text, such as whether the clothing of the figures matches the era of the inscription, and whether the image background corresponds to the historical geographical environment. This mechanism supports flexible matching for non-perfectly aligned scenes and can automatically provide warnings of potential conflicts between the image and text, thus effectively improving the historical accuracy and academic interpretability in the processing of traditional Chinese paintings.

[0159] Finally, in the visualization cross-validation module, this embodiment maps the multimodal analysis results to a set of structured interfaces, including: highlighted inscriptions, seals, background buildings, and landscape images in the main image view, along with their matched historical database nodes; inscription content displayed in sentences, with semantic elements such as names of people, places, and time words identified using color markings; and entity lookup tables labeled with "...". Figure 1 Labels such as "Inconsistent in text and image", "Consistent in text and image, but inconsistent with historical facts", and "Consistent in text and image, but confirmed by historical materials" are used.

[0160] Taking "A Panorama of Rivers and Mountains" as an application example, in this embodiment, combined with Figure 2 and Figure 3 The entity annotation and cross-validation reasoning process shown further illustrates the three-modal analysis and visualization method of graph, text, and history adopted in this invention.

[0161] First, such as Figure 2As shown in the diagram, key visual entities were extracted and labeled from the painting. After preprocessing the overall image structure, a style-adaptive visual model was used to identify semantically valuable visual regions in the image, including the Qianlong inscription area on the right (labeled Part 1) and the Pu Guang and Cai Jing inscription areas on the left (labeled Part 2), which were labeled with orange boxes. Simultaneously, the system identified several clear seal entities in the image, such as "Xuantong Appreciation," "Wuyi Zhai Fine Appreciation Seal," and "Shiqu Baoji," distinguishing them with dark red boxes. Furthermore, based on the painting's content, aesthetically and semantically representative areas of the landscape imagery (such as mountains and waterways) were identified and visually labeled with blue and green boxes, respectively, constituting a structured input of multiple entities in the image modality.

[0162] Next, enter Figure 3 The entity data extraction and analysis stage involves identifying the location of the inscriptions, extracting the text content, and directly marking the corresponding semantic entities on each inscription's original text with rounded rectangles of different colors. Specifically, this includes: person entities (blue background), time entities (purple background), location entities (orange background), inferable statements (green background), and potentially conflicting content (pink background). For example, in the Qianlong inscriptions "Wang and Zhao" and "Who was the one who adjusted the cauldron at that time?", the system could not directly obtain the exact identity information of the individuals from this paragraph. However, through cross-inscription comparison, it could be found that "Wang Jinqing" and "Zhao Qianli" mentioned in Pu Guang's inscription could be used to infer the specific referents of "Wang and Zhao". Furthermore, through the referential relationship between "Jing" and "Er" in the context, and the explicit appearance of "Cai Jing" in the inscription, it could be inferred that the person implied in "Who was the one who adjusted the cauldron at that time?" refers to Cai Jing (who was the prime minister of Emperor Huizong of Song). This inference process is visually represented by green arrows in the diagram, showing the cross-referencing of entities between multiple inscriptions.

[0163] The processing of image seal information is also simultaneously incorporated into the entity data information in the intermediate part. For example, after seals such as "Shiqu Baoji", "Wuyi Zhai Jingjianxi", and "Xuantong Jianshang" are accurately located in the image, the text information is extracted through the seal recognition model, and compared with the seal year and usage scope recorded in the database. Finally, the dynasty to which the seal belongs and the background of appreciation are mapped, providing a time-space reference for the content of the inscription.

[0164] Finally, as Figure 3As shown in the rightmost section, the system verifies the semantics of the figures and times in the inscriptions by using structured historical data fields in the database (including figures' names, nicknames, dynasties, birth and death years, social relationships, etc.). Taking "Wang Ximeng was born in 1096" as an example, the system analyzes the two phrases in Cai Jing's inscription, "bestowed in the third year of Zhenghe (i.e., 1113)" and "Ximeng was eighteen years old," and uses time calculation logic (18 years before 1113) to determine that it is 1096. Further combining the inscription data in the image with the database record (Wang Ximeng: born in 1096), the system concludes that the results are consistent. The relevant verification paths are marked in the diagram by brown rounded rectangles connected by dashed lines, reflecting the consistent reasoning process of image, text, and history.

[0165] This embodiment breaks through the limitations of traditional visual methods, achieving a semantic-level deep fusion of text and images in traditional Chinese paintings. Traditional Chinese paintings contain a large amount of freehand expression, abstract composition, and non-standardized fonts, making it difficult for traditional computer vision methods to extract their deep semantic information. This embodiment uses a multimodal approach as its core, employing a style-adaptive visual Transformer and an end-to-end image-text modeling model to effectively integrate linguistic elements such as images and inscriptions in traditional Chinese paintings, establishing a fine-grained semantic alignment mechanism between text and images. This method does not rely on large-scale labeled data to uniformly model visual elements and the semantic structure of classical Chinese texts, providing semantic support for information such as figures and scenes in traditional Chinese paintings.

[0166] Secondly, this embodiment achieves cross-modal consistency verification of image and text information with historical context, enhancing the interpretability of traditional Chinese painting information. This embodiment introduces a "image-text-history" trimodal alignment framework into traditional Chinese painting data for the first time. By constructing a joint semantic graph using a cross-modal graph Transformer, visual entities and text entities are uniformly mapped and contextual reasoning is performed, enabling entity association judgment and conflict detection in a multimodal space. This method is particularly suitable for handling complex situations common in traditional Chinese paintings, such as unclear historical background, ambiguous figures, and inconsistencies between images and text. It significantly improves the verification capability of image content in historical and cultural research, helping to establish a more interpretive image-based historical narrative path.

[0167] Finally, this embodiment provides a new multimodal technical approach for digital humanities research, promoting the intelligent evolution of understanding traditional Chinese painting. Compared to traditional research methods primarily based on image processing, the multimodal cross-validation method constructed around traditional Chinese painting in this embodiment demonstrates high versatility and scalability. This method can be applied to various cultural data, such as images of ancient figures, images of documents, and images of exhibited cultural relics, enabling the analysis and judgment of historical consistency between images and text. This achievement not only enriches the technical methodological system of image history and digital textual research, but also provides cutting-edge technical support for cultural heritage protection, knowledge dissemination, and intelligent exhibitions, possessing significant academic value and social impact.

[0168] Please see Figure 4This embodiment provides a historical data cross-verification reasoning device 200, including:

[0169] The acquisition unit 201 is used to acquire image data, identify the dynasty and style type of the image data, and generate style factors;

[0170] Embedding unit 202 is used to convert the style factor into a style vector and embed the style vector into the multi-head attention mechanism of the style adaptive visual model;

[0171] The input unit 203 is used to input the image data into a style-adaptive visual model with embedded style vectors to obtain the semantic vector of each visual block;

[0172] The splicing unit 204 is used to generate postscript text from the image data using a visual language model, extract multiple entity information from the postscript text using a large language model, splice and encode the multiple entity information to obtain a semantic representation vector;

[0173] Alignment unit 205 is used to perform alignment training on the semantic vector and the semantic representation vector through a weighted alignment function and a triple loss function to form a three-modal co-embedding space of image, text and history; wherein, the three modalities of image, text and history include: image, text and historical materials;

[0174] Construction unit 206 is used to construct a joint semantic graph by embedding image entities and epigraph text entities in the three-modal co-embedded space of image, text and history, and to perform multi-layer semantic propagation on the joint semantic graph through a cross-modal graph model to obtain a graph structure;

[0175] Combination unit 207 is used to semantically classify entities extracted from the graph structure and combine them into several sets of triples;

[0176] The comparison unit 208 is used to embed the triples into the spatiotemporal semantic embedding space through the triple path embedding function, and to compare the triples with the triples recorded in the historical database by vector space distance to obtain the verification result.

[0177] Furthermore, the calculation formula for the multi-head attention mechanism is as follows:

[0178] ;

[0179] Among them, Q, K, and V are three vector sets obtained by linear mapping of visual image patches in the image; The style adjustment factor is generated by the style extraction network from the input image; This represents the dot product operation between the Q vector and the K vector; Indicates the dimension of the vector.

[0180] Furthermore, the alignment unit 205 includes:

[0181] An alignment training subunit is used to perform alignment training on the semantic vector and the semantic representation vector using the following formula:

[0182] ;

[0183] in, Represents the dot product of vectors; This indicates the structured similarity of whether a text / image pair co-maps to a certain historical entity; This represents the attention weights in adaptive learning; Represents a semantic vector; Represents a semantic representation vector.

[0184] Furthermore, the alignment unit 205 also includes:

[0185] The triplet loss calculation subunit is used to calculate the triplet loss between the semantic vector and the semantic representation vector.

[0186] An alignment loss calculation subunit is used to calculate the alignment loss between the semantic vector and the semantic representation vector;

[0187] A supervision loss calculation subunit is used to calculate the history consistency supervision loss between the semantic vector and the semantic representation vector;

[0188] The tuning subunit is used to tune the training parameters using the triplet loss, the alignment loss, and the history consistency supervision loss.

[0189] Furthermore, the calculation formula for the ternary path embedding function is as follows:

[0190] ;

[0191] Where P represents a person; T represents time; and L represents location; It is the embedding function for the character entity; It is a time embedding function; It is a location embedding function.

[0192] Furthermore, the comparison unit 208 includes:

[0193] The distance calculation subunit is used to calculate the cosine distance between the triplet and the triplets recorded in the historical database;

[0194] The distance determination subunit is used to determine whether the cosine distance is less than a predetermined threshold.

[0195] The successful determination subunit is used to determine that if the cosine distance is less than a predetermined threshold, the triplet is successfully matched with the triplet recorded in the historical database.

[0196] The failure determination subunit is used to determine that the triplet fails to match the triplet recorded in the historical database if the cosine distance is greater than or equal to a predetermined threshold.

[0197] Furthermore, the splicing unit 204 includes:

[0198] The text extraction subunit is used to extract personal names, place names, dynasties, time expressions, and event structures from the postscript text using lexical analysis, syntactic parsing, and nested named entity recognition technologies.

[0199] The subunit is used to represent semantic entities through nodes and syntactic and semantic dependencies through edges, based on the personal names, place names, dynasties, time expressions and event structures, to obtain the semantic structure graph of the epigraph.

[0200] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0201] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can implement the methods provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0202] The present invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the methods provided in the above embodiments. Of course, the computer device may also include various network interfaces, power supplies, and other components.

[0203] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

[0204] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusivity.

[0205] The term "comprises" implies that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for cross-verification and reasoning based on historical materials, characterized in that, include: Acquire image data, identify the dynasty and style type of the image data, and generate style factors; The style factors are transformed into style vectors and the style vectors are embedded into the multi-head attention mechanism of the style adaptive vision model; The image data is input into a style-adaptive visual model with embedded style vectors to obtain the semantic vector of each visual block; A visual language model is used to generate postscript text from the image data, and a large language model is used to extract multiple entity information from the postscript text. The multiple entity information is then concatenated and encoded to obtain a semantic representation vector. The semantic vector and the semantic representation vector are aligned and trained by jointly using a weighted alignment function and a triple loss function to form a three-modal co-embedding space of image, text, and history; wherein, the three modalities of image, text, and history include: image, text, and historical materials; The image entities and epigraph text entities in the three-modal co-embedded space of image, text, and history are constructed into a joint semantic graph, and the joint semantic graph is subjected to multi-layer semantic propagation through a cross-modal graph model to obtain the graph structure; The entities extracted from the graph structure are classified semantically and combined into several groups of triples; The triples are embedded into the spatiotemporal semantic embedding space using a triple path embedding function, and the vector space distance between the triples is compared with that of the triples recorded in the historical database to obtain the verification results. The step of aligning and training the semantic vector and the semantic representation vector using a weighted alignment function and a triple loss function to form a three-modal co-embedding space for graph, text, and history includes: The semantic vector and the semantic representation vector are aligned and trained using the following formula: in, Represents the dot product of vectors; This indicates the structured similarity of whether a text / image pair co-maps to a certain historical entity; This represents the attention weights in adaptive learning; Represents a semantic vector; Represents a semantic representation vector; The step of jointly aligning and training the semantic vector and the semantic representation vector using a weighted alignment function and a triple loss function to form a three-modal co-embedding space of graph, text, and history further includes: Calculate the triplet loss between the semantic vector and the semantic representation vector; Calculate the alignment loss between the semantic vector and the semantic representation vector; Calculate the history consistency supervision loss between the semantic vector and the semantic representation vector; The training parameters are tuned using the triplet loss, the alignment loss, and the history consistency supervision loss.

2. The historical material cross-verification reasoning method according to claim 1, characterized in that, The calculation formula for the multi-head attention mechanism is as follows: Among them, Q, K, and V are three vector sets obtained by linear mapping of visual image patches in the image; The style vector is generated by the style extraction network from the input image; This represents the dot product operation between the Q vector and the K vector; Indicates the dimension of the vector.

3. The historical material cross-verification reasoning method according to claim 1, characterized in that, The formula for calculating the ternary path embedding function is as follows: Where P represents a person; T represents time; and L represents location; It is the embedding function for the character entity; It is a time embedding function; It is a location embedding function.

4. The historical material cross-verification reasoning method according to claim 1, characterized in that, The triplet is compared with the triplets recorded in the historical database using vector space distance, and the verification results include: Calculate the cosine distance between the given triplet and the triplets recorded in the historical database; Determine whether the cosine distance is less than a predetermined threshold; If the cosine distance is less than a predetermined threshold, then the triplet is determined to be a successful match with the triplet recorded in the historical database. If the cosine distance is greater than or equal to a predetermined threshold, then the triplet is determined to be a mismatch with the triplet recorded in the historical database.

5. The historical source cross-verification reasoning method according to claim 1, characterized in that, The process of generating inscription text from the image data using a visual language model, extracting multiple entity information from the inscription text using a large language model, concatenating and encoding the multiple entity information to obtain a semantic representation vector, and then includes: Lexical analysis, syntactic parsing, and nested named entity recognition technologies were used to extract personal names, place names, dynasties, time expressions, and event structures from the postscript text; Based on the aforementioned personal names, place names, dynasties, time expressions, and event structures, semantic entities are represented by nodes, and syntactic and semantic dependencies are represented by edges, resulting in a semantic structure graph of the epigraph.

6. A historical material cross-verification reasoning device, characterized in that, include: An acquisition unit is used to acquire image data, identify the dynasty and style type of the image data, and generate style factors; An embedding unit is used to convert the style factor into a style vector and embed the style vector into the multi-head attention mechanism of the style adaptive vision model; The input unit is used to input the image data into a style-adaptive visual model that embeds style vectors to obtain the semantic vector of each visual block. The splicing unit is used to generate postscript text from the image data using a visual language model, extract multiple entity information from the postscript text using a large language model, splice and encode the multiple entity information to obtain a semantic representation vector; The alignment unit is used to perform alignment training on the semantic vector and the semantic representation vector by jointly using a weighted alignment function and a triple loss function to form a three-modal co-embedding space of image, text, and history; wherein, the three modalities of image, text, and history include: image, text, and historical materials; The construction unit is used to construct a joint semantic graph from image entities and epigraph text entities in the three-modal co-embedded space of image, text, and history, and to perform multi-layer semantic propagation on the joint semantic graph through a cross-modal graph model to obtain a graph structure; Combining units are used to semantically classify entities extracted from a graph structure and combine them into several sets of triples. The comparison unit is used to embed the triples into the spatiotemporal semantic embedding space through the triple path embedding function, and to compare the triples with the triples recorded in the historical database by vector space distance to obtain the verification result; The alignment unit includes: An alignment training subunit is used to perform alignment training on the semantic vector and the semantic representation vector using the following formula: in, Represents the dot product of vectors; This indicates the structured similarity of whether a text / image pair co-maps to a certain historical entity; This represents the attention weights in adaptive learning; Represents a semantic vector; Represents a semantic representation vector; The alignment unit further includes: The triplet loss calculation subunit is used to calculate the triplet loss between the semantic vector and the semantic representation vector. An alignment loss calculation subunit is used to calculate the alignment loss between the semantic vector and the semantic representation vector; A supervision loss calculation subunit is used to calculate the history consistency supervision loss between the semantic vector and the semantic representation vector; The tuning subunit is used to tune the training parameters using the triplet loss, the alignment loss, and the history consistency supervision loss.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the historical data cross-validation reasoning method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the historical data cross-validation reasoning method as described in any one of claims 1 to 5.

Images