A historical textual research analysis method, device and equipment

Abstract

This application relates to the field of digital humanities technology and discloses a method, apparatus, and device for historical verification analysis. The method includes: acquiring target data to be verified and determining modal data matching the target data, the modal data including first modal data and second modal data; querying target reference data matching the modal data to form a set of reference data matching the modal data; performing confidence screening on the reference data set to determine a set of candidate data matching the modal data, the candidate data set including a first candidate set and a second candidate set; performing cross-modal semantic consistency matching on the first and second candidate sets; constructing a text-image association graph based on the matching results; correcting the edge weights of the text-image association graph; and determining the historical verification information of the target data based on the corrected text-image association graph. The technical solution provided by this application can improve the reliability and accuracy of historical verification.

Description

Technical Field

[0001] This application relates to the field of digital humanities technology, and in particular to a method, apparatus and equipment for historical research and analysis. Background Technology

[0002] In the context of historical and cultural research, ancient paintings and ancient documents have long been separated into different research systems. Existing studies on the relationship between images and texts are mostly focused on modern image-text data pairs, which are difficult to apply to cross-modal data such as ancient paintings and ancient books, which have characteristics such as large time spans and loose semantic connections.

[0003] In practical applications, the data that researchers need to verify often exhibits local features of a single modality, such as only obtaining local image details of a certain ancient painting. Traditional cross-modal retrieval relies on bimodal alignment. Under the condition of modality missing, it is difficult to effectively trigger simultaneous retrieval of related ancient paintings and ancient texts, and it is easy to generate a lot of noisy results, making the verification results incomplete and inaccurate.

[0004] Therefore, improving the reliability and accuracy of historical research has become a key focus of current research in the field of historical research. Summary of the Invention

[0005] This application provides a historical research and analysis method, apparatus, and equipment that can improve the reliability and accuracy of historical research.

[0006] The first aspect of this application provides a historical verification analysis method, the method comprising: acquiring target data to be verified and determining modal data matching the target data, the modal data including first modal data and second modal data; querying target reference data matching the modal data in a pre-constructed reference database to form a reference data set matching the modal data, the reference data set including a first reference set and a second reference set; performing confidence screening on the reference data set to determine a candidate data set matching the modal data, the candidate data set including a first candidate set and a second candidate set; performing cross-modal semantic consistency matching on the first candidate set and the second candidate set, constructing a graph-text association graph based on the matching results, performing edge weight correction on the graph-text association graph, and determining the historical verification information of the target data based on the corrected graph-text association graph.

[0007] In one implementation, determining modal data matching the target data includes: determining the modal type of the target data; based on the modal type, determining whether the target data needs data augmentation; when it is determined that the target data needs data augmentation, identifying the target data as first modal data, determining the augmentation data type of the target data, and augmenting the target data based on the augmentation data type to obtain second modal data; when it is determined that the target data does not need data augmentation, splitting the target data into the first modal data and the second modal data.

[0008] In one embodiment, the reference database includes a data index table; querying target reference data that matches the modal data in the pre-built reference database to form a set of reference data that matches the modal data includes: for any modal data in the first modal data and the second modal data, encoding the modal data into a query vector, determining multiple reference data vectors that match the query vector according to the data index table; determining the identification information carried by each reference data vector, determining the corresponding target reference data in the reference database according to the identification information; storing the target reference data corresponding to the first modal data in a first reference set, and storing the target reference data corresponding to the second modal data in a second reference set.

[0009] In one embodiment, the reference database is constructed as follows: acquiring a historical database, wherein the historical database includes a historical image database and a historical text database; preprocessing historical data in any of the historical databases, and generating a historical data vector corresponding to the preprocessed historical data; storing the historical data as reference data in the corresponding reference database, and storing the historical data vector as a reference data vector in the data index table of the reference database, wherein the reference database includes an image database and a text database.

[0010] In one implementation, confidence screening of the reference data set to determine candidate data sets matching the modality data includes: for any reference set in the reference data set, determining the similarity between any two target reference data in the reference set; constructing a node weight graph of the reference set, wherein the nodes of the node weight graph are target reference data, and the edge weights of the node weight graph are the similarity between connected nodes; performing confidence scoring on any of the nodes, and based on the confidence scoring results, performing confidence propagation on the node weight graph; and selecting candidate reference data that meet a preset confidence threshold from the node weight graph after confidence propagation, and storing the candidate reference data in a candidate data set corresponding to the reference set.

[0011] In one implementation, performing cross-modal semantic consistency matching on the first candidate set and the second candidate set, and constructing a graph-text association graph based on the matching results includes: pairing candidate reference data of the first candidate set and the second candidate set to form multiple graph-text data pairs, wherein the graph-text data pairs include image data and text data; determining the semantic consistency of each graph-text data pair; and constructing a graph-text association graph based on each graph-text data pair and its corresponding semantic consistency; wherein the nodes in the graph-text association graph include the image data and the text data, and the edge weight is the semantic consistency of the corresponding graph-text data pair.

[0012] In one implementation, edge weight correction of the text-image association graph includes: identifying anomalous node pairs in the text-image association graph; updating the edge weights of the anomalous node pairs to obtain optimized edge weights for the anomalous node pairs; and obtaining supplementary text annotations for the anomalous node pairs; identifying other node pairs of the anomalous node pairs; and correcting the edge weights of the other node pairs based on the optimized edge weights and supplementary text annotations for the anomalous node pairs to obtain an updated text-image association graph.

[0013] In one implementation, edge weight correction of other node pairs in the graph-text association graph based on the optimized edge weights and supplementary text annotations of the anomalous node pairs includes: acquiring first image data and first text data corresponding to the anomalous node pairs, and acquiring second image data and second text data corresponding to the other node pairs; determining the visual similarity between the first image data and the second image data, determining the first semantic similarity between the first text data and the second text data, and determining the second semantic similarity between the supplementary text annotations and the second text data; and updating the edge weights of the other node pairs based on the visual similarity, the first semantic similarity, and the second semantic similarity to complete the edge weight correction of the other node pairs.

[0014] A second aspect of this application provides a historical verification analysis apparatus, comprising: a data preparation unit for acquiring target data to be verified and determining modal data matching the target data, the modal data including first modal data and second modal data; a data matching unit for querying target reference data matching the modal data in a pre-constructed reference database to form a reference data set matching the modal data, the reference data set including a first reference set and a second reference set; a data filtering unit for performing confidence filtering on the reference data set to determine a candidate data set matching the modal data, the candidate data set including a first candidate set and a second candidate set; and an information acquisition unit for performing cross-modal semantic consistency matching on the first candidate set and the second candidate set, constructing a text-image association graph based on the matching results, performing edge weight correction on the text-image association graph, and determining the historical verification information of the target data based on the corrected text-image association graph.

[0015] A third aspect of this application provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the historical research and analysis method described in the first aspect above.

[0016] The technical solution provided in one or more embodiments of this application can quickly and accurately determine the text-image association graph representing historical research information through multimodal data querying, matching, and correction. Specifically, multimodal data of the target data is acquired, and when the target data is a single modality, another modality is generated through a large model for semantic matching as a supplement. This improves the accuracy and comprehensiveness of historical research through multimodal linked queries. Furthermore, corresponding modality query matching and confidence filtering are performed in a pre-built reference database to determine the candidate data set for each modality. A text-image association graph is then constructed based on the candidate data set for each modality, and the edge weights of the text-image association graph are corrected. This allows for more accurate and intuitive acquisition and presentation of historical research information through the text-image association graph. In addition, during the above confidence filtering and edge weight correction process, a dynamic confidence correction and edge weight correction mechanism based on expert interaction can be used to further enhance the reliability of the research results. This technical solution expands the retrieval scope, thereby improving the comprehensiveness and cultural adaptability of modality retrieval, and thus enhancing the convenience, accuracy, and comprehensiveness of historical research. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A schematic diagram illustrating the steps of a historical verification and analysis method provided for the implementation of this application; Figure 2 A schematic diagram illustrating the steps of determining a reference data set according to an embodiment of this application; Figure 3 A schematic diagram illustrating the steps of determining a candidate data set according to an embodiment of this application; Figure 4(a) is a schematic diagram of the first stage of a historical research and analysis method provided in an embodiment of this application; Figure 4(b) is a schematic diagram of the second stage of a historical verification and analysis method provided in an embodiment of this application; Figure 4(c) is a schematic diagram of the third stage of a historical verification and analysis method provided in an embodiment of this application; Figure 4(d) is a schematic diagram of the fourth stage of a historical verification and analysis method provided in an embodiment of this application; Figure 5 A schematic diagram of a historical verification and analysis device provided for one embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Furthermore, the use of terms such as "first," "second," etc., in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of embodiments in this application, unless otherwise stated, "multiple" means two or more. Additionally, the use of "based on" or "according to" implies openness and inclusiveness, because processes, steps, calculations, or other actions "based on" or "according to" one or more of the stated conditions or values ​​may in practice be based on additional conditions or beyond the stated values.

[0021] Ancient paintings and texts are important material and spiritual carriers of ancient civilization, possessing irreplaceable value in traditional art, philology, and art history research. However, for a long time, images and texts have belonged to different research systems, and the implicit connections between them have often relied on researchers' personal knowledge and experience for manual comparison and inference. This method is not only inefficient and has limited coverage, but also makes it difficult to systematically discover deep cross-modal semantic connections from massive amounts of unstructured historical data.

[0022] With the development of artificial intelligence and cross-modal learning technologies, deep learning-based image-text matching and retrieval methods have emerged. However, due to the characteristics of cross-modal data such as ancient paintings and ancient books—such as their large time span, loose semantic connections, and significant stylistic differences—traditional cross-modal methods often struggle to achieve accurate alignment in historical research scenarios. Specifically, in actual historical research, the clues researchers possess are often unimodal, such as only a partial fragment of an ancient painting or only a passage of text from an ancient book. Moreover, this unimodal clue may trace back to information from another modality, such as related ancient texts associated with the ancient painting at different points in time. This makes it difficult for traditional cross-modal matching to initiate effective retrieval, easily generating a large amount of noisy results, resulting in incomplete and inaccurate research findings.

[0023] Furthermore, historical research, as a reasoning process highly dependent on domain knowledge, typically requires expert verification and correction of its initial judgments. Existing cross-modal matching methods, once the retrieval and matching results are generated, remain fixed. When applied to historical research, errors in assessment at a particular point in time can cause the entire assessment to deviate from the correct direction. The lack of correction for local results makes the analysis process rigid, prone to errors, and results in low reliability. Therefore, this application provides one or more embodiments of a historical research analysis method, apparatus, and device to solve the above problems, improve the efficiency and depth of historical clue discovery, verification, and interpretation, and enhance the reliability and accuracy of historical research.

[0024] Please see Figure 1One embodiment of this application provides a historical research and analysis method, which may include the following steps: S1: Obtain the target data to be verified and determine the modal data that matches the target data, wherein the modal data includes first modal data and second modal data.

[0025] The target data mentioned above can be either ancient text data or ancient painting data to be verified, or either ancient text data or ancient painting data. The corresponding modal types are text modality and / or image modality, and the corresponding matching modal data are text data and / or image data. It should be noted that even if the target data belongs to only one modal type, the modal data to be determined still includes both text data and image data. Using multimodal data to represent the target data compensates for the shortcomings of single-modal queries, thereby expanding the search scope and improving the comprehensiveness and cultural adaptability of modal retrieval.

[0026] For example, when the target data is a single modality, it is directly identified as the first modality data. Data of another modality, semantically matching the target data, is generated through large model output, feature matching, etc., and this second modality data is used as the second modality data, i.e., the complement modality data of the first modality data. For example, when the target data includes multiple modalities, it is split according to modality type to form first modality data and second modality data. It should be noted that the resulting modality data does not specify a particular modality type using "first" or "second," but only indicates that the modality types are different.

[0027] S3: In a pre-built reference database, query target reference data that matches the modal data to form a reference data set that matches the modal data. The reference data set includes a first reference set and a second reference set.

[0028] The aforementioned reference database includes a pre-built image reference database and a text reference database. Multiple target reference data matching the first modality data are queried in the corresponding type databases to form a first reference set. Similarly, multiple target reference data matching the second modality data are queried in the corresponding type databases to form a second reference set. The target reference data can be understood as several data points in the reference database that have a high similarity to the target data. The aforementioned reference database can be constructed based on a large amount of pre-processed known ancient text and painting data. Optionally, specific arrangement rules or specific data indexes can be set when constructing the reference database to accelerate the query speed of modality data, save matching time, and improve the convenience and accuracy of historical research.

[0029] S5: Perform confidence screening on the reference data set to determine a candidate data set that matches the modality data, wherein the candidate data set includes a first candidate set and a second candidate set.

[0030] In this embodiment, the confidence level screening operation described above can be simply understood as scoring the confidence level of each target reference data in the reference data set, and screening the target reference data according to a preset confidence threshold to retain reference data with high correlation to the modal data, thereby forming a candidate data set. Optionally, in the confidence level screening process, to avoid bias in the confidence level scoring, knowledge graph technology can be introduced, treating each target reference data as a node, and propagating the nodes according to their confidence level to correct or add confidence levels to each target reference data. Specifically, after confidence level screening, the first reference set forms a first candidate set, and after confidence level screening, the second reference set forms a second candidate set.

[0031] It should be noted that the obtained candidate data sets do not specify a particular modality type using "first" or "second," but only indicate that the modality types of the two are different and the modality types are the same as those referred to by the corresponding reference data sets. It should also be noted that the aforementioned first reference set may include the aforementioned first modality data, and the aforementioned second reference set may include the aforementioned second modality data.

[0032] S7: Perform cross-modal semantic consistency matching on the first candidate set and the second candidate set, construct a graph-text association graph based on the matching results, perform edge weight correction on the graph-text association graph, and determine the historical verification information of the target data based on the corrected graph-text association graph.

[0033] The aforementioned cross-modal semantic consistency matching operation can be simply understood as encoding and mapping the reference data in the first and second candidate sets one by one to the same semantic space, and calculating and matching the similarity between the encoded vectors. Further, reference data with high semantic consistency can be used as nodes to draw a network graph structure to form a text-image association graph. The first and second candidate sets are distinguished according to modal type; one candidate set contains all image data, and the other contains all text data. For any image data, cross-modal semantic consistency matching is performed between the image data and each piece of text data. Image data and text data with high semantic consistency are connected as nodes in the text-image association graph, and their semantic consistency is used as the edge weight to form the edge. This process completes cross-modal semantic consistency matching for all image data and text data, forming the aforementioned text-image association graph. Optionally, the aforementioned text-image association graph also includes first-modal data nodes and second-modal data nodes.

[0034] The aforementioned edge weight correction operation can be understood as correcting abnormal edge connections in the text-image association graph. For example, a human-computer interaction interface can be designed to present the obtained text-image association graph to domain experts, who can then manually correct the graph. This correction includes identifying abnormal node pairs (composed of image and text data nodes) and changing the edge weight values ​​of these pairs, or adding simple annotations to a text data node. Optionally, based on expert feedback, other adjacent node pairs of the abnormal node pair can be corrected, or other node pairs in the text-image association graph can be corrected based on feedback regarding the abnormal node pair. For example, node propagation can also be performed on other node pairs based on a certain edge weight to adjust the semantic consistency between nodes, thereby adjusting the node connections in the graph.

[0035] In this embodiment, the corrected text-image association graph can be considered to represent historical verification information related to the target data through its nodes and their connections. For example, other text-image nodes strongly associated with the first or second modality data node and their semantic consistency are directly read from the text-image association graph and presented as historical verification information in the form of a list or linked list. For example, the evolution of the target data over historical time can also be visualized through the temporal attributes of each node (e.g., representing dynastic information) to infer the rise and fall of a theme or image. Optionally, the text-image association graph can be directly presented in a human-computer interaction view to most intuitively display the historical verification information. Optionally, users can be supported in filtering node connections in the text-image association network according to a semantic consistency threshold and selectively analyzing the corrected association results to write a historical verification report.

[0036] Based on the above ideas, the technical solution provided in this embodiment of the application can quickly and accurately determine the text-image association graph representing historical research information through multimodal data querying, matching, and correction. Specifically, multimodal data of the target data is acquired, and when the target data is a single modality, another modality is generated through a large model for semantic matching as a supplement. This improves the accuracy and comprehensiveness of historical research through multimodal linkage querying. Furthermore, corresponding modality query matching and confidence filtering are performed in a pre-constructed reference database to determine the candidate data set for each modality. A text-image association graph is then constructed based on the candidate data set for each modality, and the edge weights of the text-image association graph are corrected. This allows for a more accurate and intuitive acquisition and presentation of historical research information through the text-image association graph. In addition, during the above confidence filtering and edge weight correction process, a dynamic confidence correction and edge weight correction mechanism based on expert interaction can be used to further enhance the reliability of the research results. This technical solution expands the retrieval scope, thereby improving the comprehensiveness and cultural adaptability of modality retrieval, and thus enhancing the convenience, accuracy, and comprehensiveness of historical research.

[0037] In one implementation, when determining modal data that matches the target data, data augmentation or data splitting is performed based on the modality type of the target data to determine first modality data and second modality data. Specifically, the modality type of the target data is first determined. This modality type includes image modality and text modality, corresponding to image data and text data, respectively. Determining the modality type of the target data means judging whether the target data belongs to image data, text data, or both.

[0038] Furthermore, based on the determined modality type, it is determined whether the target data needs data augmentation. If the modality type is characterized as unimodal (image data or text data), it is determined that the target data needs augmentation; if the modality type is characterized as multimodal (image data and text data), it is determined that the target data does not need augmentation. When it is determined that the target data does not need augmentation, the target data is split into first modality data and second modality data. For example, the target data includes both image data and text data, which are respectively designated as the first modality data and the second modality data.

[0039] Specifically, when it is determined that the target data needs data augmentation, the target data is identified as first modality data, and the augmentation data type is determined. Based on the augmentation data type, the target data is augmented to obtain second modality data, where the augmentation data type is another modality type. For example, if the modality type of the target data is image data, then this target data is used as the first modality data, and text modality augmentation is performed to obtain the second modality data. The data augmentation operation can involve using a multimodal model, combined with preset modality generation prompts, to generate multiple candidate modality descriptions as the second modality data. The modality generation prompts include text generation prompts and image generation prompts.

[0040] For example, the BLIP3-o model is used to supplement the target data to generate one or more second modal datasets. Using the first and second modal datasets as the basis for queries, multiple searches are performed in the data index table. The retrieved target reference data is deduplicated to obtain a first reference set and a second reference set. This improves the coverage of relevant candidates without increasing the user's input burden and provides a higher-quality candidate pool for subsequent expert-annotated refinement. Optionally, a results presentation view is designed and provided, including an image presentation view and a text presentation view, to present the preliminary search results, i.e., the reference dataset. The image presentation view displays the retrieved ancient painting images and their metadata (such as dynasty, author, and location) in a thumbnail raster or timeline format. The text presentation view displays the retrieved ancient book text fragments and their metadata (such as document name and date) in a list or card format.

[0041] In one embodiment, when the first modal data is image data, the BLIP3-o model is used to extract visual features. Based on these visual features and a preset text generation prompt template, a second modal data, i.e., text data, is generated. The aforementioned preset text generation prompt template is used to guide the generation of descriptive text that conforms to the context of ancient Chinese paintings and encompasses the content and artistic conception of the painting.

[0042] The reference style for the aforementioned preset text generation prompt template is as follows: "[Image Input]; {Task Description} [Select a meaningful part from the image, which can be a specific local detail or the entire scene, and describe it in the style of traditional Chinese writing. Do not imagine or infer any elements that do not exist in the image. Please use the following writing format]: {Format Constraints} [Concise statements, using parallelism and abstract concepts, discussing moral principles and behavioral norms]. {Case Supplement} [Those who wished to manifest their virtue to the world first governed their states; those who wished to govern their states first regulated their families; those who wished to regulate their families first cultivated their personal lives]. Begin describing after analyzing the image." Specifically, the BLIP3-o model is called multiple times to generate autoregressive text based on the fused features to obtain second-modal data. During this process, the generation quality and diversity are controlled by adjusting the following key hyperparameters: 'Temperature', 'Max_length', and 'Number'. 'Temperature' adjusts the diversity of generated text (default 0.7), 'Max_length' sets the maximum length of generated text (default 100 characters), and 'Number' determines the number of different styles of text descriptions generated (default 4).

[0043] In another embodiment, when the first modal data is text data, the first modal data is combined with an image generation prompt template through the BLIP3-o model. The image generation prompt template is used to specify the era style and art form of the target image, and the second modal data, i.e., image data, is generated through the BLIP3-o model.

[0044] The image-generated prompt template is as follows: "{User Input} [A Tang Dynasty lady in elegant attire sits in a garden, gently playing a guqin]. {Style Fill} [Traditional Chinese mural style, painted with mineral pigments on walls or large fixed building surfaces. The colors are rich and stable, the lines are strong and flowing, and the shapes are dignified and solemn. It emphasizes decoration and narrative, with a full composition and clear layers. The overall style is solemn and grand, possessing rich and solemn classical aesthetic characteristics]." The BLIP3-o model iteratively denoises random noise and synthesizes images that conform to the text description to obtain the second modality data. During this process, the generation process is controlled by adjusting the following key hyperparameters: 'Seed', 'Guidance_scale', and 'Number'. 'Seed' represents a fixed random seed to ensure reproducibility and is set to 0 by default; 'Guidance_scale' controls the semantic alignment strength between text and images and is set to 7.5 by default; and 'Number' determines the number of images with different styles generated and is set to 4 by default.

[0045] The technical solution provided in this embodiment further refines the method for determining modal data, ensuring that matching multimodal data can be determined regardless of the modality of the target data. Specifically, when the target data is multimodal and includes both image and text data, it is directly split. When the target data is unimodal and contains only image or text data, the target data is determined as the first modal data, and a multimodal model is used in conjunction with preset modality generation prompts to expand the data and generate the second modal data. This improves the coverage of relevant candidates without increasing the user's input burden, thereby expanding the search scope, improving the comprehensiveness and cultural adaptability of modal retrieval, and ultimately enhancing the convenience, comprehensiveness, and accuracy of historical verification.

[0046] In one implementation, please refer to Figure 2 In the process of performing modal data matching queries, efficient multimodal queries can be achieved by referencing the data index tables set in the database. Querying target reference data that matches the modal data in the pre-built reference database to form a reference data set that matches the modal data includes the following steps: S31: For any modal data in the first modal data and the second modal data, encode the modal data into a query vector, and determine multiple reference data vectors that match the query vector according to the data index table; S33: Determine the identification information carried by each reference data vector, and determine the corresponding target reference data in the reference database based on the identification information; S35: Store the target reference data corresponding to the first modal data into the first reference set, and store the target reference data corresponding to the second modal data into the second reference set.

[0047] In this embodiment, the aforementioned reference database includes a data index table representing structured index components. This data index table stores reference data vectors and their corresponding identification information for rapid similarity retrieval. The reference data vectors are vector representations of the original reference data (ancient texts / paintings) in a pre-built historical database, obtained using the same encoding format. For example, the data index table is constructed based on the ANN algorithm and includes multiple data indexes (ANN indexes) for quickly filtering reference data vectors, supporting high-dimensional vector approximate nearest neighbor search. In historical research scenarios, the database size typically reaches millions or even tens of millions of units. Using a data index table for approximate indexing avoids comparing all vectors in the reference database, thereby reducing search complexity and improving query matching efficiency.

[0048] The aforementioned identification information includes at least a data ID that identifies and locates the original reference data. In the reference database, the original reference data is associated with unique identification information to ensure traceability back to the original reference data. For example, the identification information may also include modality type, time attribute, source information, etc. A mapping relationship between vector space and physical storage is established through the identification information, so that each piece of original reference data has a corresponding reference data vector, an identification information, and a data index location.

[0049] In step S31 above, for any modal data (image data / text data), it is converted into a query vector using the corresponding encoder (image encoder / text encoder). Further, a nearest neighbor search is performed using a data index table, for example, by using semantic similarity or image similarity for search matching, to quickly determine multiple reference data vectors that match the query vector.

[0050] In step S33 above, since the reference data vector only contains semantic information and cannot be directly presented to the user or used for subsequent analysis, it is necessary to use the identification information to locate the original data in reverse. Specifically, each reference data vector is associated with unique identification information in the data index table. The system extracts the identification information based on the retrieved reference data vector, and then retrieves the complete target reference data from the storage layer of the reference database.

[0051] In this embodiment, the reference database is constructed as follows: S301: Obtain the historical database, wherein the historical database includes a historical image database and a historical text database; S303: Preprocess the historical data in any historical database, and generate a historical data vector corresponding to the preprocessed historical data. The above preprocessing operations are determined based on the modality type of the historical database. If the historical database is historical image data, the preprocessing operations are to normalize the size, correct the color, and filter the noise of the historical image data. If the historical database is historical text data, the preprocessing operations are to perform sentence segmentation and font conversion on the historical text data. Furthermore, the preprocessed historical data is encoded, that is, it is encoded into vector form by the corresponding encoder to obtain the historical data vector.

[0052] S305: Historical data is used as reference data and stored in the corresponding reference database, and historical data vectors are used as reference data vectors and stored in the data index table of the reference database. The reference databases include image databases and text databases.

[0053] In one embodiment, the image database is constructed as follows: historical image data in the historical image database is sequentially subjected to size normalization, color correction, and noise filtering. Size normalization involves adjusting all images to a uniform resolution (preferably 512×512 pixels) and using bicubic interpolation to preserve details, as shown below: , This represents the image after size normalization and preprocessing. For image scaling functions, ' ' indicates that the interpolation algorithm used is bicubic interpolation, and the color correction is based on the CIE LAB color space for color cast compensation, adjusting the white balance parameter δ∈[0.9,1.1]. The color correction formula is expressed as follows: Noise filtering employs the non-local means denoising (NL-means) algorithm, setting the search window to 7×7 and the similarity block window to 5×5. For the pixels in the image to be denoised... Its denoised value is expressed as : ,in, It is based on pixels A 7x7 search window centered on the search engine. It is the original intensity value of the pixel. It is a pixel. and pixels The weighting coefficients between them are expressed as ,in, Denotes the normalization factor, which is They are respectively based on and The pixel vectors of similar blocks (5×5 neighborhood) centered on the center. The smoothing parameter (i.e., the filtering strength) is a multiple of the noise standard deviation.

[0054] Furthermore, after preprocessing, image feature encoding is performed, and the Chinese-CLIP visual encoder is used for feature extraction. First, the normalized image is center-cropped and scaled as follows: , This represents the image after center cropping and scaling. This represents the image after preprocessing such as size normalization. Next, a 768-dimensional feature vector is extracted. , for The visual encoder performs L2 normalization on the feature vectors: , to be used as a reference data vector.

[0055] In another embodiment, the aforementioned text database is constructed as follows: historical text data in the historical text database undergoes font conversion, sentence segmentation detection, and text feature encoding processing sequentially. Font conversion uses the OpenCC library to convert traditional Chinese characters to simplified Chinese characters via a default mapping table. Sentence segmentation detection uses an ancient Chinese sentence segmentation model to predict punctuation in unpunctuated text fields. During text feature encoding, the input format is controlled to be '[CLS]+word segmentation result+[SEP]', and the maximum length is controlled to 512 tokens. The 768-dimensional feature vector at the [CLS] position is obtained through CLIP_TextEncoder encoding and then subjected to L2 normalization.

[0056] In the above embodiments, based on the reference data set We construct efficient Approximate Nearest Neighbor (ANN) index structures for retrieval, and generate codebook sets by minimizing distortion errors through k-means clustering learning. The quantizer Q is constructed to map the vectors to the codebook. Specifically, the k-means clustering objective function is expressed as: The vector quantization encoding function is expressed as: ,in, For reference data set The i-th reference data vector in the middle, For the j-th cluster center, the data ID of the reference data vector is finally associated to ensure that the encoding result can be traced back to the source data.

[0057] The technical solution provided in this embodiment achieves efficient mapping from semantic query to physical data acquisition through vector encoding, index retrieval, and identifier resolution. Specifically, during database construction, a data index table is established for each modality. Based on this index table, a high-dimensional vector approximate nearest neighbor search is performed to quickly determine matching reference data vectors, thereby retrieving the corresponding target reference data. This technical solution uses index and identifier information to reverse locate and retrieve complete target reference data, thus avoiding a full comparison of all vectors in a reference database of millions or even tens of millions of data points. This significantly reduces search complexity, improves query matching efficiency, and achieves efficient mapping from semantic query to physical data acquisition, further enhancing the efficiency and accuracy of historical verification.

[0058] In one implementation, please refer to Figure 3 The confidence screening of the reference dataset to determine the candidate dataset that matches the modal data includes the following steps: S51: For any reference set in the reference data set, determine the similarity between any two target reference data in the reference set; S53: Construct a node weight graph of the reference set, where the nodes of the node weight graph are the target reference data, and the edge weights of the node weight graph are the similarity between connected nodes. S55: Calculate the confidence score for any node, and propagate the confidence score to the node weight graph based on the confidence score results; S57: In the node weight graph after confidence propagation, select candidate reference data that meet the preset confidence threshold and store the candidate reference data into the candidate data set corresponding to the reference set.

[0059] The aforementioned reference data set includes a first reference set and a second reference set, the aforementioned candidate data set includes a first candidate set and a second candidate set, the modal types of the aforementioned first modal data, the first reference set and the first candidate set correspond, the modal types of the aforementioned second modal data, the second reference set and the second candidate set correspond, the first candidate set includes multiple image candidate reference data, and the second candidate set includes multiple text candidate reference data.

[0060] The aforementioned confidence score can be scored directly by domain experts through a human-computer interaction interface, or by evaluating the similarity between modal data and each target reference data, with an output value ranging from 0 to 1. The aforementioned node weight graph is used to model the relationships between each target reference data point. It determines the corresponding connection relationship by checking whether the similarity between each target reference data point meets a preset threshold, and uses this similarity as the edge weight, thus forming the node weight graph. The aforementioned confidence propagation can be understood as updating the confidence estimate of each node through a message passing mechanism between nodes. Even with only one node having a known confidence, the confidence of other nodes can be quickly determined based on the magnitude of the edge weights. The aforementioned confidence propagation process can also dynamically adjust the confidence based on edge connections. If a node is connected to multiple high-confidence nodes, its confidence should be increased accordingly; conversely, if it is only connected to low-confidence nodes, its confidence should be suppressed.

[0061] In step S53 above, a node weight graph is constructed based on the reference set type. If the reference set contains image data, the nodes in the node weight graph are image nodes, and the edge weights represent the similarity between image nodes. If the reference set contains text data, the nodes in the node weight graph are text nodes, and the edge weights represent the similarity between text nodes. Further, domain experts manually review the target reference data represented by any node to label its relevance and confidence level to the verification task, thereby performing confidence propagation and confidence filtering to determine candidate reference data (image candidate reference data / text candidate reference data). The aforementioned verification task is a verification topic proposed by the experts. It should be noted that the first and second reference sets have their own node weight graphs.

[0062] In one embodiment, confidence is passed in the following manner: experts assign initial confidence scores of 1–7 to any number of images or texts in the reference set. The initial confidence level is normalized and mapped to [0,1], represented as follows: Among them, the unrated images or texts are ordered to be... Initialize the above initial confidence level mapping into a confidence level vector. And iterate in the following manner: ,in, , representing the propagation coefficient, This is a similarity matrix transformed based on similarity scores. Let be the confidence vector for the current round t. Given the confidence vector updated in round t+1, calculate the maximum difference in each round. ,in, Let be the confidence level of the i-th node in the t-th round. Let be the confidence level of the i-th node in the (t+1)-th round, when the maximum difference is less than the threshold ( After convergence, take The smoothed confidence score for each node is used to complete the aforementioned confidence propagation. Optionally, based on the confidence propagation results, a comprehensive score is calculated for the image candidate reference data or text candidate reference data: ,in, Set a custom threshold (e.g., 0.5) to filter low-scoring data and generate a high-quality candidate data set. Optionally, support experts to make multiple rounds of interactive adjustments to the screening results to obtain a high-quality first candidate set and second candidate set.

[0063] The technical solution provided in this embodiment achieves refined screening of candidate data sets through formal modeling of node weight graphs and confidence propagation, significantly improving the reliability and accuracy of historical verification. Specifically, by constructing the correlation between reference data in the node weight graph model, and using confidence propagation to reasonably estimate and correct the confidence of unlabeled data, the accuracy and rationality of confidence assessment can be effectively improved, avoiding bias caused by a single score. By using threshold screening based on a combination of confidence and similarity, low-relevance data can be eliminated, resulting in a high-quality, highly relevant candidate data set, providing a reliable data foundation for subsequent historical verification analysis. This approach also takes into account the advantages of human expert experience and automated calculation, improving the accuracy and flexibility of the screening process, thereby further enhancing the reliability and accuracy of historical verification.

[0064] In one implementation, cross-modal semantic consistency matching is performed on a first candidate set and a second candidate set, and a graph-text association graph is constructed based on the matching results, including: The candidate reference data of the first and second candidate sets are paired up to form multiple image-text data pairs, which include image data and text data. Further, the semantic consistency of each image-text data pair is determined. Based on each image-text data pair and its corresponding semantic consistency, an image-text association graph is constructed. The nodes in the image-text association graph include image data and text data, and the edge weights are the semantic consistency of the corresponding image-text data pairs. These image-text data pairs serve as node pairs in the image-text association graph. It should be noted that the first and second candidate sets together determine an image-text association graph.

[0065] The first candidate set includes multiple image candidate reference data, and the second candidate set includes multiple text candidate reference data. Based on this, for any image candidate reference data, it is paired one-to-one with each text candidate reference data to form multiple image-text data pairs in the form of binary pairs. The image candidate reference data serves as the image data of the image-text data pair, and the text candidate reference data serves as the text data of the image-text data pair. The semantic consistency of the above image-text data pairs is used to quantify the degree of matching between the two types of modal data at the semantic level, thereby overcoming the limitations of single-modal retrieval and realizing explicit association between images and text.

[0066] In this context, the semantic consistency of image-text data pairs is a cross-modal similarity. Unlike unimodal similarity, which calculates distance within the same vector space, cross-modal similarity requires addressing the semantic alignment issue across heterogeneous data spaces. For example, a multimodal large-scale model can be used to determine the similarity of image-text data pairs. For instance, the CLIP model encodes image and text data into vector forms, maps them to the same space, and calculates cosine similarity or dot product as the similarity of the image-text data pairs. Through explicit pairwise pairing and similarity calculation, two isolated sets are merged into a unified association network, achieving cross-validation of images and text, thereby significantly improving the reliability and accuracy of historical verification.

[0067] In one embodiment, for any image-text data pair, the normalized log-likelihood value of the image candidate reference data and the text candidate reference data is used as its semantic consistency, and is represented as follows: ,in, Text transformed from candidate reference data sequence of (word elements), for The total number, The model represents the model in a given image. and the first i ( to Predict the next word in the case of ) The probability. Then the score obtained. Linear normalization to [0,1] is used to obtain the corresponding semantic consistency score. Further, image and text candidate reference data are presented as nodes, and connection lines between nodes are drawn based on their initial consistency scores. Optionally, the visual attributes of the connection lines (e.g., color, shade, thickness) map the level of semantic consistency between the image and text. Optionally, a graphical control is designed and provided to support experts in filtering nodes by time grouping at a macro level and performing fine-grained filtering of connection lines by setting a semantic consistency threshold.

[0068] The technical solution provided in this embodiment further refines the construction method of the image-text association graph. Specifically, candidate reference data in the first candidate set and the second candidate set are paired in pairs to form multiple image-text data pairs. A unified image-text association graph is constructed using image data and text data as nodes and the semantic consistency of corresponding image-text data pairs as edge weights. This technical solution, through explicit cross-modal pairing and semantic consistency measurement calculation, integrates the originally isolated image and text candidate data into an association network, realizing cross-validation and semantic alignment of images and text. This effectively overcomes the limitations of single-modal retrieval and significantly improves the reliability and accuracy of historical verification results. In addition, the visualized association graph and filtering functions facilitate experts to conduct macro-level control and refined analysis, enhancing the intuitiveness and controllability of the verification process.

[0069] In one implementation, edge weight correction of the text-image association graph includes the following steps: S71: Identify anomalous node pairs in the text-image association graph, update the edge weights of the anomalous node pairs to obtain optimized edge weights, and obtain supplementary text annotations for the anomalous node pairs. S73: Based on the optimized edge weights of abnormal node pairs and supplementary text annotations, the edge weights of other node pairs in the graph-text association graph are corrected to obtain an updated graph-text association graph.

[0070] The aforementioned anomalous node pairs are those in the text-image association graph where the edge weights deviate significantly from the expected values. The optimized edge weights, confirmed through manual review or reliability verification, are edge weight values ​​that better reflect the true strength of the association. The edge weights represent the semantic consistency between the text-image data pairs. The determination of these anomalous node pairs can also be achieved through manual evaluation by domain experts via a human-computer interface. These experts will provide optimized edge weights for anomalous node pairs and supplementary textual annotations, which may include anomaly analysis, correction criteria, and association direction. For example, a re-estimation can be performed based on the expert's binary judgment (correct or incorrect) combined with confidence levels. By introducing anomaly determination and correction by domain experts, the probability of verification bias can be effectively reduced.

[0071] In this embodiment, the aforementioned edge weight correction can be performed by globally correcting other node pairs in the text-image association graph, excluding anomalous nodes, based on the optimized edge weights and supplementary text annotations of the anomalous node pairs. Unlike local neighborhood propagation, global propagation does not limit the propagation range but instead establishes the semantic association strength between anomalous node pairs and other node pairs through semantic consistency, adjusting the edge weights of other node pairs accordingly. The aforementioned semantic association strength can be understood as the degree of similarity between anomalous node pairs and other node pairs in visual features or semantic content.

[0072] In this embodiment, based on step S73 above, the edge weight correction for other node pairs based on the optimized edge weights of abnormal node pairs and supplementary text annotations includes the following steps: S731: Obtain the first image data and first text data corresponding to the abnormal node pair, and obtain the second image data and second text data corresponding to other node pairs; S733: Determine the visual similarity between the first image data and the second image data, and determine the first semantic similarity between the first text data and the second text data, and determine the second semantic similarity between the supplementary text annotation and the second text data; S735: Based on visual similarity, first semantic similarity, and second semantic similarity, update the edge weights of other node pairs to complete the edge weight correction of other node pairs.

[0073] In one embodiment, edge weight correction is performed as follows: For anomalous node pairs identified by experts, the correction magnitude for the edge weights is calculated as follows: ,in, The initial edge weights are calculated as follows. The optimized edge weights are provided after manual review by experts, and the edge weights are updated according to the above correction range. For any other node pair, the semantic association strength between it and the corrected abnormal node pair is calculated, including visual similarity, first semantic similarity, and second semantic similarity. Among them, visual similarity... First semantic similarity Second semantic similarity ,in, For the image vector representation of other node pairs. The image vector representation of the abnormal node pairs. For the text vector representation of other node pairs, This is the text vector representation of the abnormal node pairs. Vector representation with expert annotations.

[0074] Furthermore, based on visual similarity, first semantic similarity, and second semantic similarity, edge weights of other node pairs are updated. The update direction of these other node pairs is determined by the following formula: The adjustment range is scaled according to the following formula: Optionally, to prevent numerical updates from saturating at similarity boundaries, additional logit and sigmoid constraints are applied as follows: Optionally, the above-mentioned text-image association graph is iteratively updated. In each iteration of the update, the edge weights are corrected. When a preset termination condition is met, the iteration stops and the result is output. The preset termination condition can be reaching the maximum number of iterations threshold, or the maximum change in the network edge weight update being less than the preset threshold. This globally optimizes the semantic consistency score of the entire text-image association graph until the result converges.

[0075] The technical solution provided in this embodiment improves the accuracy of textual and graphical association by correcting the edge weights of the graph. Specifically, firstly, domain experts identify anomalous node pairs whose edge weights deviate significantly from the expected values ​​through an interactive interface. Their edge weights are then updated to optimized weights, and supplementary text annotations containing anomaly analysis and correction criteria are added. Subsequently, based on the optimized edge weights and supplementary text annotations of these anomalous node pairs, the association strength between these pairs and other node pairs in terms of visual features and semantic content is calculated, including visual similarity, semantic similarity between texts, and semantic similarity between the supplementary text annotations and the text. This determines the direction and magnitude of edge weight updates for other node pairs, and applies constraint functions to prevent numerical saturation. Global edge weight correction is iteratively performed until convergence, thereby propagating the impact of local expert corrections to the entire graph, achieving global optimization of semantic consistency scoring, effectively reducing the probability of textual verification deviations, and improving the reliability and accuracy of historical verification.

[0076] This application provides an embodiment of a historical research and analysis system that applies the historical research and analysis method described in the first aspect above. The system features a human-computer interaction interface for information exchange with experts. This embodiment uses the Chinese classical musical instrument "pipa" as the object of analysis. First, experts enter the description "one or more women in traditional clothing are playing the pipa" into the text query input box of the interactive interface, which serves as the target data to be verified in this embodiment. Exemplarily, historical research is conducted in the following four stages: Phase 1: Refer to Figure 4(a) to determine the set of modal data and image reference data that match the target data.

[0077] Specifically, the system identifies the target data as text-based, designating it as the first modality, and triggers a data augmentation operation: it calls a predefined image generation prompt template, combining the target data with style keywords randomly sampled from a Chinese painting style library (such as "Tang Dynasty Dunhuang mural style," "Song Dynasty court painting style," and "Ming Dynasty literati painting style") to generate multiple images depicting a "woman playing the pipa" scene under different historical and artistic styles, serving as the second modality. Further, the second modality is used to search a pre-built image reference database, returning a batch of authentic paintings containing pipa images, including several Tang Dynasty murals. Through the aforementioned "text → image generation → image retrieval" strategy, a second reference set spanning multiple dynasties and containing multiple authentic pipa images is quickly constructed.

[0078] Phase 2: Please refer to Figure 4(b) to perform confidence screening on the second reference set to determine the second candidate set.

[0079] Specifically, through the interactive interface provided by the system, experts manually select local areas of the pipa from multiple images in the second reference set, and assign different confidence scores based on the holding direction (e.g., "tilted to the left," "held vertically," "tilted to the right"). For example, icons with a significant leftward tilt are labeled with high confidence. The confidence level can be indicated by color options; for instance, green indicates higher confidence, and red indicates lower confidence. The system receives the image features of these local areas and their corresponding expert confidence scores as "seed" samples. Using a built-in confidence propagation algorithm, the similarity between the features of these seed samples and the corresponding features of other images in the image reference set is calculated, and the confidence scores are propagated in a weighted manner to unlabeled images with similar features. Based on the comprehensive confidence scores after propagation, the second reference set is reordered and filtered, thereby highlighting the local feature pattern of "holding direction" throughout the second reference set, retaining only image data with different holding styles as the second candidate set.

[0080] Phase 3: Refer to Figure 4(c) to determine the first candidate set that matches the first modality data, and match the first candidate set with the second candidate set.

[0081] In a historical text database, historical text records related to "pipa" are retrieved to form a first reference set comprising multiple text data. This first reference set is then filtered based on confidence level to form a final reference set. For each image-text data pair formed by the first and second reference sets, a text-image association graph is constructed, using image-text data pairs as image-text node pairs and their semantic consistency as edge weights. By setting a semantic consistency threshold, image-text pairs with strong associations can be initially selected. For example, several "farewell poems" mentioning the pipa show a high degree of match with images depicting farewell scenes of pipa performance, revealing the "farewell" imagery often associated with the pipa in classical literature.

[0082] Phase 4: Please refer to Figure 4(d) to perform edge weight correction on the matched text-image association graph. In the corrected text-image association graph, determine the historical verification information.

[0083] For example, experts discovered a key document in the text-image association graph that states, "During the Zhenguan era, the pipa player Pei Luo'er abandoned the plectrum and used her hand, which is what we now call 'hand-plucking the pipa'," hinting at the historical evolution of pipa playing techniques from using a plectrum to hand-plucking. However, initial system calculations showed that this document had a low correlation with many later paintings depicting hand-plucking techniques. Researchers then manually increased the semantic consistency score between this document and several paintings explicitly showing hand-plucking of the pipa through the system's association correction interface. This involved edge weight correction and adding supplementary textual annotations such as, "This document records a technique innovation and should be strongly correlated with paintings depicting hand-plucking techniques."

[0084] Furthermore, the system uses this local correction (the corrected edge weights) as expert knowledge input, and based on the affinity between the corrected image and text features, automatically propagates the correction's impact to other node pairs semantically similar in their depiction of techniques. After correction, the updated image-text association graph shows that this document has established a stronger association link with a large number of paintings depicting hand-playing techniques from the Tang Dynasty onwards, clearly confirming and visually presenting the historical evolution of performance techniques.

[0085] Please see Figure 5 The second aspect of this application provides a historical research and analysis device, the device comprising: The data preparation unit 100 is used to acquire the target data to be verified and determine the modal data that matches the target data, wherein the modal data includes first modal data and second modal data; The data matching unit 200 is used to query target reference data that matches the modal data in a pre-built reference database to form a reference data set that matches the modal data, the reference data set including a first reference set and a second reference set; The data filtering unit 300 is used to perform confidence filtering on the reference data set to determine a candidate data set that matches the modality data, wherein the candidate data set includes a first candidate set and a second candidate set; The information acquisition unit 400 is used to perform cross-modal semantic consistency matching on the first candidate set and the second candidate set, construct a graph-text association graph based on the matching results, perform edge weight correction on the graph-text association graph, and determine the historical verification information of the target data based on the corrected graph-text association graph.

[0086] In one embodiment, the data preparation unit 100 is specifically used to acquire the target data to be verified, determine the modality type of the target data, determine whether the target data needs to be expanded based on the modality type, determine the target data as first modality data when it is determined that the target data needs to be expanded, determine the expanded data type of the target data, expand the target data based on the expanded data type to obtain second modality data, and split the target data into first modality data and second modality data when it is determined that the target data does not need to be expanded.

[0087] In one embodiment, the data matching unit 200 is specifically configured to, for any one of the first modal data and the second modal data, encode the modal data into a query vector, determine multiple reference data vectors that match the query vector according to a data index table, determine the identification information carried by each reference data vector, determine the corresponding target reference data in a reference database according to the identification information, store the target reference data corresponding to the first modal data into a first reference set, and store the target reference data corresponding to the second modal data into a second reference set.

[0088] In one embodiment, the data filtering unit 300 is specifically used to determine the similarity between any two target reference data in any reference set in the reference data set, construct a node weight graph of the reference set, wherein the nodes of the node weight graph are target reference data, the edge weights of the node weight graph are the similarity between connected nodes, perform confidence scoring on any node, perform confidence propagation on the node weight graph based on the confidence scoring results, filter out candidate reference data that meet the preset confidence threshold in the node weight graph after confidence propagation, and store the candidate reference data in the candidate data set corresponding to the reference set.

[0089] In one embodiment, the information acquisition unit 400 is specifically used to pair candidate reference data of the first candidate set and the second candidate set to form multiple image-text data pairs, each image-text data pair including image data and text data. The semantic consistency of each image-text data pair is determined. Based on each image-text data pair and its corresponding semantic consistency, an image-text association graph is constructed. The nodes in the image-text association graph include image data and text data, and the edge weight is the semantic consistency of the corresponding image-text data pair. Based on the corrected image-text association graph, the historical verification information of the target data is determined.

[0090] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0091] One embodiment of the historical research and analysis device in this application is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, or other devices that can provide the above functions.

[0092] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 6 Take a processor 10 as an example.

[0093] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0094] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0095] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0096] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0097] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0098] This application also provides a computer-readable storage medium. The methods described in this application can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the methods shown in the above embodiments are implemented.

[0099] The apparatus, module, or unit described in the above embodiments can be implemented by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0100] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer devices. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer devices according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0106] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0107] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

[0108] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A historical research and analysis method, characterized in that, The method includes: Obtain the target data to be verified, and determine the modal data that matches the target data, wherein the modal data includes first modal data and second modal data; In a pre-built reference database, target reference data that matches the modal data is queried to form a reference data set that matches the modal data. The reference data set includes a first reference set and a second reference set. The reference data set is subjected to confidence screening to determine a candidate data set that matches the modality data. The candidate data set includes a first candidate set and a second candidate set. Cross-modal semantic consistency matching is performed on the first candidate set and the second candidate set. A graph-text association graph is constructed based on the matching results. Edge weight correction is performed on the graph-text association graph. Based on the corrected graph-text association graph, the historical verification information of the target data is determined.

2. The method according to claim 1, characterized in that, The modal data that matches the target data includes: Determine the modality type of the target data, and based on the modality type, determine whether the target data needs to be augmented. When it is determined that the target data needs to be expanded, the target data is identified as first modal data, and the data type for expansion of the target data is determined. Based on the data type for expansion, the target data is expanded to obtain second modal data. When it is determined that the target data does not require data expansion, the target data is split into the first modal data and the second modal data.

3. The method according to claim 1, characterized in that, The reference database includes a data index table; in the pre-built reference database, querying target reference data that matches the modal data to form a set of reference data that matches the modal data includes: For any one of the first modal data and the second modal data, the modal data is encoded into a query vector, and multiple reference data vectors that match the query vector are determined according to the data index table; Determine the identification information carried by each of the reference data vectors, and determine the corresponding target reference data in the reference database based on the identification information; Store the target reference data corresponding to the first modal data into the first reference set, and store the target reference data corresponding to the second modal data into the second reference set.

4. The method according to claim 1 or 3, characterized in that, The reference database is constructed in the following manner: Obtain a historical database, wherein the historical database includes a historical image database and a historical text database; The historical data in any of the historical databases is preprocessed, and a historical data vector corresponding to the preprocessed historical data is generated. The historical data is used as reference data and stored in the corresponding reference database. The historical data vector is used as reference data vector and stored in the data index table of the reference database. The reference database includes an image database and a text database.

5. The method according to claim 1, characterized in that, The reference dataset is subjected to confidence screening to determine a candidate dataset that matches the modality data, including: For any reference set in the reference data set, determine the similarity between any two target reference data in the reference set; Construct a node weight graph of the reference set, wherein the nodes of the node weight graph are target reference data, and the edge weights of the node weight graph are the similarity between connected nodes; A confidence score is assigned to any of the nodes, and confidence propagation is performed on the node weight graph based on the confidence score results. In the node weight graph after confidence propagation, candidate reference data that meet the preset confidence threshold are selected and stored in the candidate data set corresponding to the reference set.

6. The method according to claim 1, characterized in that, Cross-modal semantic consistency matching is performed on the first candidate set and the second candidate set, and a graph-text association graph is constructed based on the matching results, including: The candidate reference data of the first candidate set and the second candidate set are paired up to form multiple image-text data pairs, which include image data and text data. Determine the semantic consistency of each of the image-text data pairs, and construct an image-text association graph based on each of the image-text data pairs and their corresponding semantic consistency. The nodes in the image-text association graph include the image data and the text data, and the edge weight is the semantic consistency of the corresponding image-text data pair.

7. The method according to claim 1, characterized in that, The edge weight correction for the text-image association graph includes: Identify abnormal node pairs in the text-image association graph, update the edge weights of the abnormal node pairs to obtain optimized edge weights, and obtain supplementary text annotations for the abnormal node pairs. Based on the optimized edge weights and supplementary text annotations of the abnormal node pairs, the edge weights of other node pairs in the graph-text association graph are corrected to obtain an updated graph-text association graph.

8. The method according to claim 7, characterized in that, Based on the optimized edge weights and supplementary text annotations of the anomalous node pairs, edge weight correction is performed on other node pairs in the graph-text association graph, including: Obtain the first image data and first text data corresponding to the abnormal node pair, and obtain the second image data and second text data corresponding to the other node pairs; Determine the visual similarity between the first image data and the second image data, determine the first semantic similarity between the first text data and the second text data, and determine the second semantic similarity between the supplementary text annotation and the second text data; Based on the visual similarity, the first semantic similarity, and the second semantic similarity, the edge weights of the other node pairs are updated to complete the edge weight correction of the other node pairs.

9. A historical research and analysis device, characterized in that, The device includes: A data preparation unit is used to acquire target data to be verified and determine modal data that matches the target data, wherein the modal data includes first modal data and second modal data; A data matching unit is used to query target reference data that matches the modal data in a pre-built reference database to form a reference data set that matches the modal data, the reference data set including a first reference set and a second reference set; A data filtering unit is used to perform confidence filtering on the reference data set to determine a candidate data set that matches the modality data, wherein the candidate data set includes a first candidate set and a second candidate set; The information acquisition unit is used to perform cross-modal semantic consistency matching on the first candidate set and the second candidate set, construct a graph-text association graph based on the matching results, perform edge weight correction on the graph-text association graph, and determine the historical verification information of the target data based on the corrected graph-text association graph.

10. A computer device, characterized in that, include: A memory and a processor are interconnected, the memory stores computer instructions, and the processor executes the historical research and analysis method according to any one of claims 1 to 8 by executing the computer instructions.

Images