Multimodal-based cultural relic text image matching method and device

Abstract

The application provides a kind of based on multi-modal relic text image matching method and device, the method includes: based on the relic data of each candidate relic, the semantic features of each candidate relic are extracted, and based on the relic data of target relic, the semantic features of target relic are extracted;Based on the similarity between the semantic features of each candidate relic and the semantic features of target relic, determine the initial matching relic from each candidate relic;Based on the semantic features of initial matching relic, the description features of initial matching relic are extracted, and based on the description features of initial matching relic, the candidate subgraph of initial matching relic is constructed;Based on the similarity between the candidate subgraph of initial matching relic and the template subgraph of target relic, determine the matching relic matched with target relic from initial matching relic.The application can realize network relic data resource to official relic data resource text-image, image-image and text-text high-precision matching correlation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for matching text and images of cultural relics based on multimodality. Background Technology

[0002] Public cultural resources are crucial for the effective operation of public cultural service cloud platforms, and cultural relic data resources, as an important component of public culture, play a vital role in existing public cultural services. Currently, cultural relic data resources largely rely on local public cultural institutions, lacking support from non-official resources. Furthermore, a wealth of high-quality cultural relic data resources created by individuals and businesses exist online. Establishing connections between these resources and official public cultural relic data resources can effectively improve the diversity of cultural resources and the service quality of cultural service cloud platforms. However, most online text and image data resources of cultural relics lack precise tags, making it difficult to establish links with existing public cultural relic data resources.

[0003] Joint text and image training techniques, such as CLIP (Contrastive Language-Image Pre-Training), are commonly used to jointly extract and associate text and image features, which can solve the matching problem between text and image data of cultural relics to some extent. However, matching and associating text and image data of cultural relics often requires precision down to a specific cultural relic. Furthermore, cultural relic data themselves have a high degree of similarity, and abstract semantic features cannot achieve such high-precision matching between text and image data of cultural relics. Summary of the Invention

[0004] This invention provides a method and apparatus for matching text images of cultural relics based on multimodality, in order to overcome the deficiencies in the prior art.

[0005] This invention provides a multimodal method for matching text images of cultural relics, comprising:

[0006] Based on the cultural relic data of each candidate cultural relic, semantic features of each candidate cultural relic are extracted; and based on the cultural relic data of the target cultural relic, semantic features of the target cultural relic are extracted; the cultural relic data includes text data and / or image data.

[0007] Based on the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact, an initial matching artifact is determined from each candidate artifact.

[0008] Based on the semantic features of the initially matched artifact, the descriptive features of the initially matched artifact are extracted, and based on the descriptive features of the initially matched artifact, a candidate sub-graph of the initially matched artifact is constructed.

[0009] Based on the similarity between the candidate sub-images of the initial matching artifact and the template sub-image of the target artifact, matching artifacts that match the target artifact are determined from the initial matching artifacts.

[0010] According to the present invention, a multimodal text-image matching method for cultural relics is provided, wherein the semantic features include text features and image features;

[0011] The extraction of semantic features from the cultural relic data of each candidate cultural relic, and the extraction of semantic features from the cultural relic data of the target cultural relic, include:

[0012] The image data of each candidate cultural relic is input into the image coding model to obtain the image features of each candidate cultural relic output by the image coding model; and the image data of the target cultural relic is input into the image coding model to obtain the image features of the target cultural relic output by the image coding model.

[0013] The text data of each candidate cultural relic is input into the text encoding model to obtain the text features of each candidate cultural relic output by the text encoding model; and the text data of the target cultural relic is input into the text encoding model to obtain the text features of the target cultural relic output by the text encoding model.

[0014] The image encoding model and the text encoding model constitute a semantic feature extraction model, which is trained based on the sample text data and sample image data of the sample cultural relics.

[0015] According to the multimodal text-image matching method for cultural relics provided by the present invention, the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic is determined based on the following steps:

[0016] Image similarity is determined based on the cosine distance between the image features of each candidate artifact and the image features of the target artifact.

[0017] The text similarity is determined based on the cosine distance between the text features of each candidate cultural relic and the text features of the target cultural relic;

[0018] Based on the image similarity and / or the text similarity, the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic is determined.

[0019] According to the present invention, a multimodal text-image matching method for cultural relics is provided, wherein extracting descriptive features of the initially matched cultural relics based on their semantic features includes:

[0020] Construct the main statement of the initially matched cultural relic, and construct the query statement of the initially matched cultural relic based on the descriptive dictionary;

[0021] Extract the main features of the initially matched cultural relic from the main statement, and extract the query features of the initially matched cultural relic from the query statement;

[0022] The semantic features of the initially matched cultural relic are encoded to obtain the encoded features of the initially matched cultural relic;

[0023] Based on the query features and the subject features, the encoded features are decoded to obtain the description features.

[0024] According to the multimodal text-image matching method for cultural relics provided by the present invention, the similarity between the candidate sub-image of the initially matched cultural relic and the template sub-image of the target cultural relic is determined based on the following steps:

[0025] The similarity of the main nodes is determined based on the cosine distance between the main nodes of the candidate subgraph and the main nodes of the template subgraph.

[0026] The similarity of descriptive word nodes is determined based on the relationship weight between the descriptive word nodes of the candidate subgraph and the corresponding descriptive word nodes in the template subgraph, as well as the probability of each predicted word in the descriptive word nodes of the candidate subgraph.

[0027] Based on the similarity of the main nodes and the similarity of the descriptive node, the similarity between the candidate sub-image of the initially matched cultural relic and the template sub-image of the target cultural relic is determined.

[0028] According to the multimodal text-image matching method for cultural relics provided by the present invention, the step of extracting semantic features of each candidate cultural relic based on the cultural relic data of each candidate cultural relic further includes:

[0029] The cultural relic data of each candidate cultural relic and the cultural relic data of the target cultural relic are preprocessed to obtain the preprocessed cultural relic data of each candidate cultural relic and the preprocessed cultural relic data of the target cultural relic; the preprocessing includes keyframe detection, image subject detection and text preprocessing.

[0030] According to a multimodal text-image matching method for cultural relics provided by the present invention, the method further includes, after determining the matching cultural relics that match the target cultural relic from the initial matched cultural relics, the method further includes:

[0031] The cultural relic data of the matched cultural relic is associated with the data resource database of the target cultural relic.

[0032] The present invention also provides a multimodal artifact text-image matching device, comprising:

[0033] The extraction unit is used to extract semantic features of each candidate cultural relic based on the cultural relic data of each candidate cultural relic, and to extract semantic features of the target cultural relic based on the cultural relic data of the target cultural relic; the cultural relic data includes text data and / or image data.

[0034] The determining unit is used to determine the initial matching artifact from the candidate artifacts based on the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact;

[0035] The construction unit is used to extract the descriptive features of the initially matched cultural relic based on its semantic features, and to construct a candidate sub-graph of the initially matched cultural relic based on its descriptive features.

[0036] A matching unit is used to determine, based on the similarity between the candidate sub-images of the initially matched artifacts and the template sub-images of the target artifact, a matching artifact that matches the target artifact from the initially matched artifacts.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal text-image matching method for cultural relics as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal text-image matching method for cultural relics as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal text-image matching method for cultural relics as described above.

[0040] The present invention provides a multimodal text-image matching method and apparatus for cultural relics. Based on the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic, an initial matching cultural relic is determined from each candidate cultural relic. Based on the semantic features of the initial matching cultural relic, descriptive features such as shape, color, and text subject are further extracted to characterize the initial matching cultural relic. Based on the descriptive features of the initial matching cultural relic, a candidate sub-image of the initial matching cultural relic is constructed. Thus, based on the similarity between the candidate sub-image of the initial matching cultural relic and the template sub-image of the target cultural relic, the matching cultural relic that matches the target cultural relic can be accurately determined from the initial matching cultural relic. This achieves high-precision text-image, image-image, and text-text matching and association between online cultural relic data resources and official cultural relic data resources. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the multimodal text-image matching method for cultural relics provided by the present invention.

[0043] Figure 2 This is a flowchart illustrating the semantic feature extraction model training method provided by the present invention;

[0044] Figure 3 This is a flowchart illustrating the feature extraction method provided by the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of the multimodal cultural relic text image matching device provided by the present invention;

[0046] Figure 5 This is a schematic diagram of the structure of the multimodal cultural relic text image matching system provided by the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] CLIP is often used to jointly extract and associate text and image features, which can solve the problem of matching text and image data of cultural relics to a certain extent. However, the matching and association of text and image data of cultural relics often needs to be precise down to a specific cultural relic. Secondly, the cultural relic data themselves have a high degree of similarity, and abstract semantic features cannot achieve such high-precision matching of text and image data of cultural relics.

[0050] To address this issue, this invention provides a multimodal method for matching text and images of cultural relics. Based on semantic features, this method combines information from cultural relic data from multiple dimensions, extracting more specific information such as shape, color, and text subject from text and image data. This enables high-precision matching and association of text-image, image-image, and text-text between online and official cultural relic data resources.

[0051] Figure 1 This is a flowchart illustrating the multimodal text-image matching method for cultural relics provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0052] Step 110: Extract semantic features of each candidate cultural relic based on its cultural relic data, and extract semantic features of the target cultural relic based on its cultural relic data; the cultural relic data includes text data and / or image data.

[0053] Here, candidate cultural relics can be understood as online cultural relics, that is, unofficial cultural relics. The cultural relic data of candidate cultural relics can be understood as a wealth of high-quality cultural relic data resources created by individuals and enterprises online. The semantic features of candidate cultural relics are used to characterize the historical, cultural, and artistic information they contain.

[0054] The target cultural relic can be understood as an official cultural relic, and the data on the target cultural relic can be understood as official cultural relic data. The semantic features of the target cultural relic are used to characterize the historical, cultural, and artistic information contained within it.

[0055] The cultural relic data includes text data and image data. For candidate cultural relics, the cultural relic data can be text data, image data and video data; for target cultural relics, the cultural relic data can be text data and image data.

[0056] As an optional embodiment, semantic features of each candidate cultural relic can be extracted from the cultural relic data of each candidate cultural relic based on a feature extraction model, and semantic features of the target cultural relic can be extracted from the cultural relic data of the target cultural relic.

[0057] Step 120: Based on the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic, determine the initial matching cultural relic from each candidate cultural relic.

[0058] Specifically, the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact is used to characterize the matching correlation between the candidate artifact and the target artifact. The higher the similarity, the higher the matching correlation between the candidate artifact and the target artifact, and thus the higher the probability that the candidate artifact is the initial matching artifact. Optionally, candidate artifacts with a similarity greater than a threshold can be used as initial matching artifacts. Multiple artifacts can be used as initial matching artifacts.

[0059] It should be noted that, if

[0060] Step 130: Based on the semantic features of the initially matched cultural relics, extract the descriptive features of the initially matched cultural relics, and construct candidate subgraphs of the initially matched cultural relics based on the descriptive features of the initially matched cultural relics.

[0061] Specifically, considering the high similarity of cultural relic data, the abstract semantic features extracted from cultural relic data cannot achieve high-precision matching of cultural relic text and images. In other words, the initial matching of cultural relic cannot achieve high-precision matching with the target cultural relic.

[0062] To address this, embodiments of the present invention extract descriptive features from the initially matched cultural relics based on their semantic features. These descriptive features characterize the name (i.e., main information) and descriptive word information (i.e., descriptive information) of the initially matched cultural relics. In other words, compared to abstract semantic features, descriptive features can further refine the granular representation of the information contained in the cultural relic data.

[0063] After extracting the descriptive features, a candidate subgraph for the initial matching artifact is constructed based on these features. This candidate subgraph can be understood as a network graph centered on the initial matching artifact. It consists of a main node (corresponding to main information) and several descriptive word nodes (corresponding to descriptive information). Each descriptive word node contains a matching weight, and the edges between descriptive word nodes contain a relation weight. The matching weight represents the probability that the descriptive word of the corresponding descriptive word node appears in the initial matching artifact, while the relation weight represents the probability that the two descriptive words of the two corresponding descriptive word nodes appear simultaneously.

[0064] As an optional implementation, the candidate subgraph is constructed based on the following method:

[0065] First, obtain the descriptive subgraph (i.e., template subgraph) corresponding to the given official cultural relic resource in the cultural relic description knowledge graph. This subgraph consists of a cultural relic subject node and several descriptive word nodes. Each descriptive word node also contains a matching weight, and the edges between descriptive words contain a relation weight. Second, construct candidate subgraphs based on the descriptive features. The subject information name serves as the cultural relic subject node, and the descriptive information and the descriptive word nodes of the template subgraph are in one-to-one correspondence. At this point, each candidate subgraph's descriptive word node can contain three predicted descriptive words (descriptive word and corresponding predicted probability).

[0066] Step 140: Based on the similarity between the candidate sub-images of the initial matched artifacts and the template sub-images of the target artifacts, determine the matching artifacts that match the target artifacts from the initial matched artifacts.

[0067] Specifically, the template subgraph for a target cultural relic refers to a network graph centered on the target cultural relic. This template subgraph can be constructed based on experiential knowledge of commonly used descriptive terms for cultural relics and a cultural relic database from cultural venues. The template subgraph also describes the main information and descriptive information of the target cultural relic. That is, the template subgraph consists of a main node (corresponding to main information) and several descriptive node nodes (corresponding to descriptive information) for the target cultural relic. The main information includes the name of the cultural relic, and the descriptive information includes descriptive terms for the cultural relic, such as shape descriptions, color descriptions, and the dynasty to which it belongs. Multiple template subgraphs for target cultural relics constitute a knowledge graph for cultural relic description, where the main nodes are not interconnected, but the descriptive nodes are connected pairwise.

[0068] Furthermore, each descriptor node in the template subgraph contains a matching weight, and the edges between descriptor nodes contain a relation weight. The matching weight represents the probability that the descriptor of the corresponding descriptor node appears in the target artifact, while the relation weight represents the probability that the two descriptors of the corresponding two descriptor nodes appear simultaneously.

[0069] For example, let A and B be any two distinct description nodes in a knowledge graph of cultural relics descriptions. The number of cultural relics containing both description nodes is AB_and, and the number of cultural relics containing at least one description node is AB_or. Therefore, the relational weight between the two description nodes is AB_and / AB_or. As another example, let A be any description node in the knowledge graph of cultural relics descriptions, the number of cultural relics containing this description node is A1, and the total number of cultural relics is S. Therefore, the matching weight of the description node is A1 / S.

[0070] Furthermore, the similarity between the candidate sub-image of the initial matching artifact and the template sub-image of the target artifact is used to characterize the degree of matching between the candidate and template sub-images. The higher the similarity, the higher the degree of matching between the candidate and template sub-images, and thus the higher the probability that the initial matching artifact is a matching artifact. Optionally, initial matching artifacts with a similarity greater than a threshold can be used as the final matching artifacts.

[0071] The multimodal text-image matching method for cultural relics provided in this invention determines the initial matching cultural relic from the candidate cultural relics based on the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic. Based on the semantic features of the initial matching cultural relic, descriptive features such as shape, color, and text content are further extracted to characterize the initial matching cultural relic. Candidate sub-images of the initial matching cultural relic are constructed based on these descriptive features. Thus, based on the similarity between the candidate sub-images of the initial matching cultural relic and the template sub-images of the target cultural relic, the matching cultural relic that matches the target cultural relic can be accurately determined from the initial matching cultural relic. This achieves high-precision text-image, image-image, and text-text matching and association between online cultural relic data resources and official cultural relic data resources.

[0072] Based on the above embodiments, semantic features include text features and image features;

[0073] Based on the cultural relic data of each candidate cultural relic, semantic features of each candidate cultural relic are extracted; and based on the cultural relic data of the target cultural relic, semantic features of the target cultural relic are extracted, including:

[0074] The image data of each candidate cultural relic is input into the image coding model to obtain the image features of each candidate cultural relic output by the image coding model; the image data of the target cultural relic is input into the image coding model to obtain the image features of the target cultural relic output by the image coding model.

[0075] The text data of each candidate cultural relic is input into the text encoding model to obtain the text features of each candidate cultural relic output by the text encoding model; the text data of the target cultural relic is input into the text encoding model to obtain the text features of the target cultural relic output by the text encoding model.

[0076] The image encoding model and the text encoding model constitute the semantic feature extraction model, which is trained based on the sample text data and sample image data of the sample cultural relics.

[0077] Specifically, the image coding model is used to extract image features from image data, and the text coding model is used to extract text features from text data. The image coding model and the text coding model together constitute a semantic feature extraction model, which can be trained based on sample text data and sample image data of the sample artifacts. Specifically, the semantic feature extraction model can be built based on CLIP. The sample text data and sample image data of the sample artifacts constitute sample pairs, including positive sample pairs and negative sample pairs. Positive sample pairs can be existing paired sample text data and sample image data obtained from a cultural venue artifact data resource database. Negative sample pairs can be obtained by randomly changing the sample image data in the positive sample pairs, or by randomly changing one-third of the sample text data in the positive sample pairs.

[0078] Figure 2 This is a flowchart illustrating the semantic feature extraction model training method provided by the present invention, as shown below. Figure 2 As shown, sample image data is input into an image encoding model to obtain sample image features; sample text data is input into a text encoding model to obtain sample text features; the contrast loss is determined based on the difference between sample image features and sample text features, and the parameters of the semantic feature extraction model are updated based on the contrast loss until the convergence condition is met, thus obtaining the semantic feature extraction model.

[0079] Based on any of the above embodiments, the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic is determined based on the following steps:

[0080] Image similarity is determined based on the cosine distance between the image features of each candidate artifact and the image features of the target artifact.

[0081] The text similarity is determined based on the cosine distance between the text features of each candidate cultural relic and the text features of the target cultural relic;

[0082] Based on image similarity and / or text similarity, the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic is determined.

[0083] Specifically, image similarity is used to characterize the degree of similarity between the image features of each candidate artifact and the image features of the target artifact. It can be characterized by the cosine distance between the image features of each candidate artifact and the image features of the target artifact.

[0084] Text similarity is used to characterize the degree of similarity between the text features of each candidate artifact and the text features of the target artifact. It can be represented by the cosine distance between the text features of each candidate artifact and the text features of the target artifact.

[0085] After determining image similarity and text similarity, the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact is determined based on the image similarity and text similarity. Optionally, the image similarity and text similarity can be weighted and fused based on preset weights to obtain the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact. The weights corresponding to image similarity and text similarity can both be 0.5, or can be set according to actual conditions. This embodiment of the invention does not impose specific limitations on this.

[0086] It should be noted that, when only text data is available, text similarity can be directly used as the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact. Similarly, when only image data is available, image similarity can be directly used as the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact.

[0087] Based on any of the above embodiments, based on the semantic features of the initially matched cultural relic, the descriptive features of the initially matched cultural relic are extracted, including:

[0088] Construct the main statement for initially matching cultural relics, and construct the query statement for initially matching cultural relics based on the descriptive dictionary;

[0089] Extract the main features of the initial matching artifact from the main body statement, and extract the query features of the initial matching artifact from the query statement;

[0090] The semantic features of the initially matched cultural relics are encoded to obtain the encoded features of the initially matched cultural relics;

[0091] Based on the query features and subject features, the encoded features are decoded to obtain the descriptive features.

[0092] Specifically, the descriptive features include the name of the cultural relic (i.e., the main information) and the prediction of descriptive terms (i.e., the descriptive information). The name of the cultural relic can be guided by the main statement (main query) and the query statement (descriptive query).

[0093] The query consists of 20 main queries, arranged at the beginning of the query list, used to guide the retrieval of cultural relic names, corresponding to 20 words in the output. The description query comprises multiple tokens, each token guiding a specific descriptive feature to predict the classification result. To ensure semantic connections between tokens in the description query, the required tokens are constructed using the output features of the specific query statement.

[0094] The token construction method for describing queries is as follows: Query statements are constructed based on a descriptive dictionary, such as "What is the age of this artifact?", "What is the main color of this artifact?", "What is this artifact made of?", "How many legs does this artifact have?", etc., totaling thousands of query statements. The text encoding model in the semantic feature extraction model is used to calculate the features (i.e., query features, query tokens) of each query statement. In actual implementation, the corresponding query token is selected based on the specific descriptive words of the artifact. A single data description query contains approximately dozens of tokens. The descriptive words of the artifact are predicted based on the output features. To improve the stability of the prediction, the top three predictions with the highest probabilities for each descriptive word are taken as the prediction results and named the description information.

[0095] Figure 3 This is a flowchart illustrating the feature extraction method provided by the present invention, as shown below. Figure 3 As shown, the encoding module can be a Transformer-based encoding module, and the decoding module can be a Transformer-based decoding module. In the encoding stage, the input is the initial semantic features (i.e., image features and text features) of the matched artifacts obtained based on the semantic feature extraction model. Simultaneously, category encoding is used to label inputs with only image features or only text features. The positional encoding mode adopts the general Transformer positional encoding method. In the decoding stage, masking decoding is used to highlight the sequential order of descriptive features. The final descriptive features include both subject information and descriptive information.

[0096] Based on any of the above embodiments, the similarity between the candidate sub-image of the initially matched cultural relic and the template sub-image of the target cultural relic is determined based on the following steps:

[0097] The similarity of main nodes is determined based on the cosine distance between the main nodes of the candidate subgraph and the main nodes of the template subgraph.

[0098] The similarity of descriptive word nodes is determined based on the relationship weight between the descriptive word nodes in the candidate subgraph and the corresponding descriptive word nodes in the template subgraph, as well as the probability of each predicted word in the descriptive word nodes of the candidate subgraph.

[0099] Based on the similarity of the main node and the similarity of the descriptive node, the similarity between the candidate sub-image of the initial matching cultural relic and the template sub-image of the target cultural relic is determined.

[0100] Specifically, the similarity between the candidate sub-image of the initial matching artifact and the template sub-image of the target artifact includes the similarity of the main nodes and the similarity of the descriptive node. The similarity of the main nodes can be determined by calculating the feature cosine distance between the main nodes of the candidate sub-image and the main nodes of the template sub-image. For example, the main features of the main nodes of the candidate sub-image and the main features of the main nodes of the template sub-image can be extracted by a semantic feature extraction model, and the cosine distance between the two main features can be calculated. When the two main features are the same, the cosine distance is 1, and when the cosine distance is less than 0, it is 0 by default.

[0101] Furthermore, the similarity of descriptive node nodes can be determined based on the relationship weights between descriptive node nodes in the candidate subgraph and their corresponding descriptive node nodes in the template subgraph, as well as the probabilities of each predicted word in the descriptive node nodes of the candidate subgraph. For example, suppose any descriptive node in the candidate subgraph is B, and its three predicted words are B1, B2, and B3, with corresponding prediction probabilities a1, a2, and a3. The descriptive node in the template subgraph corresponding to B is A. Then, the similarity between the two corresponding descriptive node nodes is M = (B1,A)×a1 + (B2,A)×a2 + (B2,A)×a2, where (B1,A) represents the relationship weight between descriptive word A and descriptive word B1 in the cultural relic knowledge graph. Based on the above method, the similarity of all descriptive node nodes is calculated. After weighted summation of all similarities based on the matching weights, the result is normalized by dividing by the sum of the matching weights to obtain the final descriptive node similarity.

[0102] As an optional embodiment, after determining the similarity of the main node and the similarity of the descriptive node, the main similarity and the descriptive similarity are added together and then divided by 2 to obtain the similarity between the candidate sub-image of the initially matched cultural relic and the template sub-image of the target cultural relic.

[0103] Based on any of the above embodiments, semantic features of each candidate cultural relic are extracted based on the cultural relic data of each candidate cultural relic, and the process further includes:

[0104] The cultural relic data of each candidate cultural relic and the cultural relic data of the target cultural relic are preprocessed to obtain the preprocessed cultural relic data of each candidate cultural relic and the preprocessed cultural relic data of the target cultural relic. The preprocessing includes keyframe detection, image subject detection and text preprocessing.

[0105] Specifically, preprocessing includes keyframe detection of the video, subject detection of the image, and text preprocessing.

[0106] The specific processes for video keyframe detection and image artifact subject detection are as follows:

[0107] Using image data from an existing cultural venue artifact database, a artifact detection model is trained using the SSD (Single Shot MultiBoxDetector) object detection framework. This model determines whether a video frame or image contains an artifact image, and based on the detection bounding box, assesses the appropriateness of the artifact's location and display size. Optionally, the selection criteria are as follows: the detection bounding box should be more than 0.1 times the width and height of the corresponding image or video frame. Under this criterion, the image and video frame with the highest score are selected. The score is calculated as: 0.9 × (detection bounding box area / video image area) - 0.1 × (distance from the center of the detection bounding box to the center of the video image / video image area)^0.5.

[0108] Furthermore, text preprocessing includes preprocessing of the descriptive text of the cultural relics and preprocessing of the text within the image. The specific process is as follows: First, OCR technology is used to obtain the text information on the image and add it to the descriptive text of the cultural relics. Based on the descriptive text data of the cultural relics database and general knowledge of cultural relics descriptions, a dictionary of descriptive terms for cultural relics is constructed, mainly including material descriptive terms (such as stone tools, jade tools, bronze tools, bone tools, etc.), shape descriptive terms (round, three-legged, square, etc.), color descriptive terms, time and dynasty descriptive terms, etc. Based on the descriptive term dictionary and the TF-IDF algorithm, the text data is segmented, and based on the distribution of descriptive terms, the text description sentences containing the most descriptive terms are selected (all sentences are arranged from most to least few according to the proportion of descriptive terms, at least 5 sentences are selected, and it is ensured that every different descriptive term is included in the selected sentences).

[0109] Based on any of the above embodiments, after determining the matching artifacts that match the target artifact from the initial matched artifacts, the method further includes:

[0110] The matching artifact data is linked to the target artifact's data resource database.

[0111] Specifically, matching cultural relics can be understood as cultural relics that are matched with official cultural relics. After identifying the matching cultural relics, their data is linked to the target cultural relic's data resource database. Here, the data resource database can be understood as the cultural venue's cultural relic data resource database.

[0112] The multimodal text image matching device for cultural relics provided by the present invention will be described below. The multimodal text image matching device for cultural relics described below can be referred to in correspondence with the multimodal text image matching method for cultural relics described above.

[0113] Based on any of the above embodiments Figure 4 This is a schematic diagram of the structure of the multimodal cultural relic text image matching device provided by the present invention, as shown below. Figure 4 As shown, the device includes:

[0114] Extraction unit 410 is used to extract semantic features of each candidate cultural relic based on the cultural relic data of each candidate cultural relic, and to extract semantic features of the target cultural relic based on the cultural relic data of the target cultural relic; the cultural relic data includes text data and / or image data.

[0115] The determining unit 420 is used to determine the initial matching cultural relic from each candidate cultural relic based on the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic.

[0116] Construction unit 430 is used to extract the descriptive features of the initially matched cultural relics based on the semantic features of the initially matched cultural relics, and to construct candidate sub-graphs of the initially matched cultural relics based on the descriptive features of the initially matched cultural relics.

[0117] The matching unit 440 is used to determine the matching artifacts that match the target artifacts from the initial matching artifacts based on the similarity between the candidate sub-images of the initial matching artifacts and the template sub-images of the target artifacts.

[0118] Based on any of the above embodiments Figure 5 This is a schematic diagram of the structure of the multimodal cultural relic text image matching system provided by the present invention, as shown below. Figure 5 As shown, the system includes a preprocessing module, a preliminary data resource matching module, and a precise data resource matching module.

[0119] The preprocessing module is used to preprocess the acquired text, image, and video data resources related to cultural relics. Preprocessing includes keyframe detection in videos, subject detection in images, and text preprocessing. The data processed by the preprocessing module includes unofficial cultural relic data resources from the internet (i.e., candidate cultural relic data) and official cultural relic data resources (i.e., target cultural relic data).

[0120] The preliminary data resource matching module is used to quickly filter out candidate cultural relics data that are semantically similar to the target cultural relic data based on the semantic feature extraction model, and use the semantically similar candidate cultural relics as the initial matching cultural relics.

[0121] The data resource precise matching module is used to further obtain the descriptive features of the initial matching cultural relics, construct candidate subgraphs based on the descriptive features, determine the final matching cultural relics based on the similarity between the candidate subgraphs and the template subgraphs (i.e. the subgraphs corresponding to the target cultural relics in the cultural relic description knowledge graph), and associate the cultural relic data of the matching cultural relics with the cultural relic data resource library to achieve precise matching from candidate cultural relics to target cultural relics.

[0122] Therefore, in this embodiment of the invention, official cultural relics are used as target cultural relics, and non-official cultural relics obtained from the internet are used as candidate cultural relics. The data resource preliminary matching module uses semantic information to perform preliminary screening and determine the initial matching cultural relics. Then, the data resource precise matching module obtains detailed descriptive features and constructs candidate sub-graphs. Compared with the template sub-graph, the similarity between the sub-graphs is calculated to achieve precise matching. Finally, the obtained matching association results are added to the cultural venue cultural relic data resource database.

[0123] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 610, a memory 620, a communication interface 630, and a communication bus 640, wherein the processor 610, the memory 620, and the communication interface 630 communicate with each other through the communication bus 640. The processor 610 can invoke logical instructions in the memory 620 to execute a multimodal text-image matching method for cultural relics. This method includes: extracting semantic features of each candidate cultural relic based on its data, and extracting semantic features of the target cultural relic based on its data; the cultural relic data includes text data and / or image data; determining an initial matching cultural relic from the candidate cultural relic based on the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic; extracting descriptive features of the initial matching cultural relic based on its semantic features, and constructing a candidate sub-image of the initial matching cultural relic based on its descriptive features; and determining a matching cultural relic that matches the target cultural relic from the initial matching cultural relic based on the similarity between the candidate sub-image of the initial matching cultural relic and the template sub-image of the target cultural relic.

[0124] Furthermore, the logical instructions in the aforementioned memory 620 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the multimodal cultural relic text-image matching method provided by the above methods, the method comprising: extracting semantic features of each candidate cultural relic based on the cultural relic data of each candidate cultural relic, and extracting semantic features of the target cultural relic based on the cultural relic data of the target cultural relic; the cultural relic data comprising text data and / or image data; determining an initial matching cultural relic from each candidate cultural relic based on the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic; extracting descriptive features of the initial matching cultural relic based on the semantic features of the initial matching cultural relic, and constructing a candidate sub-image of the initial matching cultural relic based on the descriptive features of the initial matching cultural relic; and determining a matching cultural relic matching the target cultural relic from the initial matching cultural relic based on the similarity between the candidate sub-image of the initial matching cultural relic and the template sub-image of the target cultural relic.

[0126] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is implemented to perform the above-described multimodal text-image matching methods for cultural relics. The method includes: extracting semantic features of each candidate cultural relic based on its cultural relic data, and extracting semantic features of the target cultural relic based on its cultural relic data; the cultural relic data includes text data and / or image data; determining an initial matching cultural relic from the candidate cultural relic based on the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic; extracting descriptive features of the initial matching cultural relic based on its semantic features, and constructing a candidate sub-image of the initial matching cultural relic based on its descriptive features; and determining a matching cultural relic matching the target cultural relic from the initial matching cultural relic based on the similarity between the candidate sub-image of the initial matching cultural relic and the template sub-image of the target cultural relic.

[0127] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal method for matching text and images of cultural relics, characterized in that, include: Based on the cultural relic data of each candidate cultural relic, semantic features of each candidate cultural relic are extracted; and based on the cultural relic data of the target cultural relic, semantic features of the target cultural relic are extracted; the cultural relic data includes text data and / or image data. Based on the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact, an initial matching artifact is determined from each candidate artifact. Based on the semantic features of the initially matched cultural relic, the descriptive features of the initially matched cultural relic are extracted, and based on the descriptive features of the initially matched cultural relic, a candidate sub-graph of the initially matched cultural relic is constructed; the descriptive features include the cultural relic name and descriptive word prediction; Based on the similarity between the candidate sub-images of the initial matched artifacts and the template sub-images of the target artifacts, matching artifacts that match the target artifacts are determined from the initial matched artifacts; The step of extracting descriptive features of the initially matched artifact based on its semantic features includes: Construct the main statement of the initially matched cultural relic, and construct the query statement of the initially matched cultural relic based on the descriptive dictionary; Extract the main features of the initially matched cultural relic from the main statement, and extract the query features of the initially matched cultural relic from the query statement; The semantic features of the initially matched cultural relic are encoded to obtain the encoded features of the initially matched cultural relic; Based on the query features and the subject features, the encoded features are decoded to obtain the description features; The similarity between the candidate sub-image of the initially matched artifact and the template sub-image of the target artifact is determined based on the following steps: The similarity of the main nodes is determined based on the cosine distance between the main nodes of the candidate subgraph and the main nodes of the template subgraph. The similarity of descriptive word nodes is determined based on the relationship weight between the descriptive word nodes of the candidate subgraph and the corresponding descriptive word nodes in the template subgraph, as well as the probability of each predicted word in the descriptive word nodes of the candidate subgraph. Based on the similarity of the main nodes and the similarity of the descriptive node, the similarity between the candidate sub-image of the initially matched cultural relic and the template sub-image of the target cultural relic is determined.

2. The multimodal text-image matching method for cultural relics according to claim 1, characterized in that, The semantic features include text features and image features; The extraction of semantic features from the cultural relic data of each candidate cultural relic, and the extraction of semantic features from the cultural relic data of the target cultural relic, include: The image data of each candidate cultural relic is input into the image coding model to obtain the image features of each candidate cultural relic output by the image coding model; and the image data of the target cultural relic is input into the image coding model to obtain the image features of the target cultural relic output by the image coding model. The text data of each candidate cultural relic is input into the text encoding model to obtain the text features of each candidate cultural relic output by the text encoding model; and the text data of the target cultural relic is input into the text encoding model to obtain the text features of the target cultural relic output by the text encoding model. The image encoding model and the text encoding model constitute a semantic feature extraction model, which is trained based on the sample text data and sample image data of the sample cultural relics.

3. The multimodal text-image matching method for cultural relics according to claim 2, characterized in that, The similarity between the semantic features of each candidate artifact and the semantic features of the target artifact is determined based on the following steps: Image similarity is determined based on the cosine distance between the image features of each candidate artifact and the image features of the target artifact. The text similarity is determined based on the cosine distance between the text features of each candidate cultural relic and the text features of the target cultural relic; Based on the image similarity and / or the text similarity, the similarity between the semantic features of each candidate cultural relic and the semantic features of the target cultural relic is determined.

4. The multimodal text-image matching method for cultural relics according to any one of claims 1 to 3, characterized in that, The process of extracting semantic features from the cultural relic data of each candidate artifact, prior to which includes: The cultural relic data of each candidate cultural relic and the cultural relic data of the target cultural relic are preprocessed to obtain the preprocessed cultural relic data of each candidate cultural relic and the preprocessed cultural relic data of the target cultural relic; the preprocessing includes keyframe detection, image subject detection and text preprocessing.

5. The multimodal text-image matching method for cultural relics according to any one of claims 1 to 3, characterized in that, The step of determining a matching artifact from the initial matched artifacts that matches the target artifact further includes: The cultural relic data of the matched cultural relic is associated with the data resource database of the target cultural relic.

6. A multimodal artifact text-image matching device, characterized in that, include: The extraction unit is used to extract semantic features of each candidate cultural relic based on the cultural relic data of each candidate cultural relic, and to extract semantic features of the target cultural relic based on the cultural relic data of the target cultural relic; the cultural relic data includes text data and / or image data. The determining unit is used to determine the initial matching artifact from the candidate artifacts based on the similarity between the semantic features of each candidate artifact and the semantic features of the target artifact; The construction unit is used to extract the descriptive features of the initially matched cultural relic based on its semantic features, and to construct a candidate sub-graph of the initially matched cultural relic based on its descriptive features; the descriptive features include the cultural relic name and descriptive word prediction. A matching unit is configured to determine, based on the similarity between the candidate sub-images of the initially matched artifacts and the template sub-images of the target artifact, a matching artifact that matches the target artifact from the initially matched artifacts; The step of extracting descriptive features of the initially matched artifact based on its semantic features includes: Construct the main statement of the initially matched cultural relic, and construct the query statement of the initially matched cultural relic based on the descriptive dictionary; Extract the main features of the initially matched cultural relic from the main statement, and extract the query features of the initially matched cultural relic from the query statement; The semantic features of the initially matched cultural relic are encoded to obtain the encoded features of the initially matched cultural relic; Based on the query features and the subject features, the encoded features are decoded to obtain the description features; The similarity between the candidate sub-image of the initially matched artifact and the template sub-image of the target artifact is determined based on the following steps: The similarity of the main nodes is determined based on the cosine distance between the main nodes of the candidate subgraph and the main nodes of the template subgraph. The similarity of descriptive word nodes is determined based on the relationship weight between the descriptive word nodes of the candidate subgraph and the corresponding descriptive word nodes in the template subgraph, as well as the probability of each predicted word in the descriptive word nodes of the candidate subgraph. Based on the similarity of the main nodes and the similarity of the descriptive node, the similarity between the candidate sub-image of the initially matched cultural relic and the template sub-image of the target cultural relic is determined.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal cultural relic text image matching method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal text-image matching method for cultural relics as described in any one of claims 1 to 5.

Images