A method for constructing multimodal intelligent knowledge cards and self-organizing links by topic.

Abstract

This invention relates to artificial intelligence and big data processing technologies, aiming to provide a method for constructing multimodal intelligent knowledge cards and self-organizing links by topic. The method includes: collecting data from multiple sources, preprocessing it to extract multimodal information, and constructing several standard structured data objects as knowledge cards; encapsulating the data in a unified format to build a knowledge card database; constructing multidimensional hybrid embedding vectors based on the attributes of the data in the knowledge card database using different knowledge cards; establishing a multimodal semantic adaptive topology network under spatiotemporal constraints to achieve self-organizing links by topic; storing the knowledge cards as nodes in a backend knowledge graph and establishing connections with the frontend display / interactive interface. This invention can be used for intelligent processing of data extraction, structured encapsulation, and dynamic association. Through vector space alignment, it can discover implicit knowledge and potential connections, achieving dynamic evolution and adaptability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and big data processing technology, specifically to a method for constructing multimodal intelligent knowledge cards and self-organizing links by topic. Background Technology

[0002] Currently, cultural tourism digital display systems and online museums are undergoing a transformation from simple "resource digitization" to in-depth "knowledge intelligence." This can be summarized as in-depth technology application and diversified experience formats. For example, immersive experiences are being built through technology-driven approaches. VR / AR / MR, 3D scanning, and high-definition modeling technologies are widely used in digital display systems, shifting the presentation of knowledge from static browsing to dynamic interaction. AI technology is being used to create the role of "data curator," providing customized explanations for different user groups. Content innovation is shifting from "having form" to "having soul," exploring IP-based operations; by combining cultural relic knowledge with modern creativity, the focus is shifting from "viewing cultural relics" to "playing with culture." Enhanced narrative transforms historical knowledge into easily understandable visual content and interactive games, effectively increasing user engagement and the breadth of cultural dissemination. These examples of transformation are largely based on changes in external display methods. The completeness of cultural tourism digital display systems and online museums still depends on the data processing of exhibits, cultural relics, and landscapes by each museum, exhibition hall, or cultural tourism scenic spot.

[0003] However, many current cultural and tourism digital display systems or online museums still have the following pain points: (1) Data silos exist: The data formats of various museums and encyclopedia websites are not uniform, multimodal data (images, text, audio, and video) are fragmented, and there is a lack of unified semantic understanding. (2) Single association dimension: The existing associations are mostly based on simple keyword matching (such as tag matching), and lack automatic association based on deep semantics and "themes". For example, it is impossible to automatically associate "blue and white porcelain" with "foreign trade in the Ming Dynasty" through potential semantic themes. (3) Static connection relationship: Once the knowledge base is established, its connection relationship is often static and lacks the ability to automatically reorganize and adaptively adjust weights as new data is introduced.

[0004] Therefore, it is necessary to propose new solutions to address the aforementioned problems. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a method for constructing multimodal intelligent knowledge cards and self-organizing links by topic.

[0006] To solve the technical problem, the solution of the present invention is:

[0007] A method for constructing multimodal intelligent knowledge cards and self-organizing links by topic is provided, including:

[0008] Data in text, image, and video formats was collected from multiple sources, and the raw data was preprocessed.

[0009] Multimodal information is extracted from the preprocessed data, and several standard structured data objects are constructed as knowledge cards based on this information. The data objects are then encapsulated in a unified format to build a knowledge card database.

[0010] Based on the attributes of data in the knowledge card database, multidimensional hybrid embedding vectors are constructed using different knowledge cards; a multimodal semantic adaptive topology network under spatiotemporal constraints is established to achieve topic-based self-organizing links.

[0011] Knowledge cards are stored as nodes in the knowledge graph in the backend, and a connection is established between the knowledge graph and the frontend display / interactive interface.

[0012] As a preferred embodiment of the present invention, the data collection from multiple sources includes at least: data extraction from local collections of text, images, and videos of cultural and tourism, cultural relics, historical or art collection institutions; or, obtaining text, images, and video data from specified website page links or databases via the Internet using web crawling technology; the data preprocessing includes: binding the text extracted from the original data to images and / or videos related to its source in terms of location, and establishing original correspondence groups between text fragments and media files.

[0013] As a preferred embodiment of the present invention, the multimodal information extraction includes:

[0014] Identify basic information from the text, including: entities and related people, events, dates, and organizations;

[0015] Multiple basic triplet relationships are formed using the basic information of entity objects;

[0016] Use ResNet or ViT models to extract features from image / video frames.

[0017] As a preferred embodiment of the present invention, the data object serving as a knowledge card encapsulates a unique identifier, entity type, and name with structured attributes, and encapsulates relation edges, spatiotemporal profiles, multimodal content data, and vector embeddings through a standardized format.

[0018] The attributes of the relation edges include: the target knowledge card ID to which it points, the type of relation, the weight of the relation, the contextual description of the relation, and the data source or origin information; the attributes of the spatiotemporal profile include: normalized time, time span, geographic coordinates, and semantic location; the attributes of the multimodal content data include: text summary, image list, video stream, audio guide, and 3D model; the attributes of the vector embedding include: unified vector representation, semantic vector, temporal vector, and spatial vector.

[0019] As a preferred embodiment of the present invention, the construction of the multidimensional hybrid embedding vector specifically includes:

[0020] (1) Semantic and visual feature extraction: The semantic and visual features in the knowledge card are extracted using a pre-trained multimodal model, and a weighted fusion high-dimensional vector is output;

[0021] (2) Time continuity encoding: The time tags in the knowledge cards are continuously encoded and mapped to continuous vectors to calculate the time distance;

[0022] (3) Geospatial coding: Map latitude and longitude or location names to spherical coordinate vectors so that the proximity of geographic space is preserved in the vector space;

[0023] (4) Assign learnable hyperparameters to the vectors obtained by the above operations, and then perform vector synthesis.

[0024] As a preferred embodiment of the present invention, a topological network with linked relationships is constructed using the constructed vectors by performing the following operations:

[0025] (1) To address the issue of multi-source heterogeneity of data, fuzzy ontology alignment logic is introduced; specifically including:

[0026] Based on the relationships extracted from multimodal information, deterministic hard links are established between different knowledge cards; a thesaurus is constructed to determine whether different knowledge cards point to the same entity through name similarity and attribute overlap; hierarchical reasoning is performed based on the constructed vectors to determine propagation links or complete explicit links;

[0027] (2) A dynamic subgraph generation method based on the gravity model to uncover unlabeled connections between knowledge cards; specifically including:

[0028] Each knowledge card is treated as a point mass in a vector space, and the semantic attraction between different knowledge cards is defined. A critical value for semantic attraction is set, and local clusters are formed through dynamic clustering. A time direction vector is introduced into the local clusters to detect the evolution trend of the vector over time. Based on the joint assumption of entity continuity and time decay, a narrative chain with cause and effect is constructed.

[0029] (3) After completing the link relationship construction, calculate the weight of each knowledge card relationship edge in the topology network.

[0030] As a preferred embodiment of the present invention, reinforcement learning is performed based on user behavior to achieve adaptive evolution of the topological network; specifically including:

[0031] (1) When a user browses a knowledge card in the knowledge graph through the front-end display and interactive interface, if the user continues to browse another knowledge card according to the guidance within the specified time, the link between the two is considered to be valid.

[0032] (2) If a user browses a knowledge card for more than the set time, or exits before the set time is reached after clicking, a reward or penalty will be given according to the preset rules, and the weight of the relationship edge of the knowledge card will be updated.

[0033] (2) Periodically check the knowledge graph. If the weight of a relation edge of a certain implicit link in the topology network is lower than the set threshold, then disconnect the connection.

[0034] (3) If more than a set number of users continue to search for another knowledge card after browsing a certain knowledge card, the relationship edge attributes in the two knowledge cards will be automatically updated to establish a new link relationship between them, even if there is no initial link between them.

[0035] As a preferred embodiment of the present invention, when a user performs a query or search in the knowledge graph through a front-end display and interactive interface:

[0036] (1) If the target content is associated with a knowledge card, while returning the knowledge card, multiple neighbor cards with a weight higher than the set value of the relation edge linked to it are also returned through random walk or breadth-first search, thus forming a recommendation list;

[0037] (2) Based on the topic description input by the user, select matching knowledge cards from the knowledge card library, and generate a special exhibition page by calculating the topological relationship between the cards in real time.

[0038] As a preferred embodiment of the present invention, the special exhibition page is generated in the following manner:

[0039] (1) Based on the text input by the user, insert topic anchors into the existing knowledge graph by means of intent vectorization and injection of virtual nodes;

[0040] (2) Based on the maximum boundary relevance, select knowledge cards that meet the similarity requirements between the input topic and the selected set;

[0041] (3) Build links between the selected knowledge cards to form a story logic;

[0042] (4) Analyze the text of the knowledge cards using a large language model and automatically generate exhibition copy based on the input text;

[0043] (5) Determine the topology structure presented on the front-end display and interactive interface based on the temporal and spatial distribution characteristics of the knowledge cards.

[0044] The present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the aforementioned method for constructing multimodal intelligent knowledge cards and self-organizing links by topic.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] 1. Based on knowledge graph, multimodal fusion and natural language processing technologies, this invention can be used for intelligent processing of data extraction, structured encapsulation and dynamic association in cultural tourism and exhibition scenarios of cultural relics, history, art and other fields.

[0047] 2. This invention has a high degree of automation, which greatly reduces the manual cost of constructing cultural tourism knowledge graphs and can handle massive amounts of heterogeneous data;

[0048] 3. This invention can discover implicit knowledge. Through vector space alignment, it can discover potential connections across eras and materials (such as "Song Dynasty landscape paintings" and "Tang Dynasty frontier poems" being automatically associated by the algorithm because they express the same theme of "loneliness").

[0049] 4. This invention enables a multimodal experience. Knowledge cards are no longer dry text, but rich media objects that include 3D and video. Furthermore, objects can be associated with each other through visual features (such as automatically associating different objects based on "decorative style").

[0050] 5. This invention enables dynamic evolution. The link relationships are not hard-coded, but automatically adjust the weights as new data is added and user behavior feedback is received, thus possessing adaptability. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of multimodal feature fusion.

[0052] Figure 2 This is a schematic diagram of the data structure for a knowledge card object.

[0053] Figure 3 This is a diagram of the overall system architecture.

[0054] Figure 4 This is a flowchart of a topic-based self-organizing link algorithm. Detailed Implementation

[0055] First, it should be noted that this invention relates to database technology, specifically an application of computer technology in the field of information security. The implementation of this invention involves the application of multiple software functional modules. The applicant believes that, after carefully reading the application documents and accurately understanding the implementation principles and objectives of this invention, and in conjunction with existing publicly known technologies, those skilled in the art can fully utilize their software programming skills to implement this invention. All references made in this application fall within this scope, and the applicant will not list them all further.

[0056] Those skilled in the art will understand that, besides implementing a portion of the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, enabling the system and its various devices, modules, and units to function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered both software modules implementing the method and structures within the hardware component.

[0057] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] Part 1: Overview of the Implementation Schemes of the Invention

[0059] The multimodal intelligent knowledge card construction and topic-based self-organizing linking method proposed in this invention focuses on encapsulating relevant information of cultural and tourism entities into "knowledge card" objects. It then calculates explicit and implicit relationships between cards in a multimodal vector space through topic-based self-organizing links, thereby achieving automatic growth and dynamic reorganization of the knowledge network. The method includes:

[0060] 1. Collect data in text, image, and video formats from multiple sources, and preprocess the raw data;

[0061] The data collection from multiple sources includes at least: data extraction from local collections of text, images, and videos of cultural and tourism, cultural relics, historical, or art collection institutions; or, obtaining text, image, and video data from specified website page links or databases through web crawling technology.

[0062] The data preprocessing includes: binding the text extracted from the raw data to the images and / or videos related to its source in terms of location, and establishing the original correspondence between text fragments and media files.

[0063] 2. Extract multimodal information from the preprocessed data, and construct several standard structured data objects as knowledge cards based on this information; encapsulate the data objects in a unified format to build a knowledge card database;

[0064] The multimodal information extraction includes: identifying basic information from text, including: entity objects and related people, events, dates and organizations; using the basic information of entity objects to form multiple basic triplet relationships; and using ResNet or ViT models to extract features from image / video frames.

[0065] 3. Based on the attributes of data in the knowledge card database, construct multidimensional hybrid embedding vectors using different knowledge cards; establish a multimodal semantic adaptive topology network under spatiotemporal constraints to achieve topic-based self-organizing links;

[0066] In the data objects serving as knowledge cards, unique identifiers, entity types, and names are encapsulated with structured attributes, and relation edges, spatiotemporal profiles, multimodal content data, and vector embeddings are encapsulated using standardized formats. The attributes of the relation edges include: the target knowledge card ID, the type of relation, the weight of the relation, the contextual description of the relation, and data source or origin information. The attributes of the spatiotemporal profiles include: normalized time, time span, geographic coordinates, and semantic location. The attributes of the multimodal content data include: text summary, image list, video stream, audio guide, and 3D model. The attributes of the vector embeddings include: unified vector representation, semantic vector, temporal vector, and spatial vector.

[0067] The construction of the multidimensional hybrid embedding vector specifically includes:

[0068] (1) Semantic and visual feature extraction: Use a pre-trained multimodal model to extract semantic and visual features from the knowledge card and output a weighted fusion high-dimensional vector; (2) Temporal continuity encoding: Encode the time tags in the knowledge card continuously and map them into continuous vectors to calculate the time distance; (3) Geospatial encoding: Map latitude and longitude or place name into spherical coordinate vectors so that the proximity of geospatial space is maintained in the vector space; (4) Assign learnable hyperparameters to the vectors obtained by the above operations and then synthesize the vectors.

[0069] Using the constructed vectors, construct a topological network with linked relationships by performing the following operations:

[0070] (1) To address the issue of multi-source heterogeneity of data, fuzzy ontology alignment logic is introduced; specifically including:

[0071] Based on the relationships extracted from multimodal information, deterministic hard links are established between different knowledge cards; a thesaurus is constructed to determine whether different knowledge cards point to the same entity through name similarity and attribute overlap; hierarchical reasoning is performed based on the constructed vectors to determine propagation links or complete explicit links;

[0072] (2) A dynamic subgraph generation method based on the gravity model to uncover unlabeled connections between knowledge cards; specifically including:

[0073] Each knowledge card is treated as a point mass in a vector space, and the semantic attraction between different knowledge cards is defined. A critical value for semantic attraction is set, and local clusters are formed through dynamic clustering. A time direction vector is introduced into the local clusters to detect the evolution trend of the vector over time. Based on the joint assumption of entity continuity and time decay, a narrative chain with cause and effect is constructed.

[0074] (3) After completing the link relationship construction, calculate the weight of each knowledge card relationship edge in the topology network.

[0075] The present invention further includes: performing reinforcement learning based on user behavior to achieve adaptive evolution of the topological network;

[0076] (1) When a user browses a knowledge card in the knowledge graph through the front-end display and interactive interface, if the user continues to browse another knowledge card within the specified time according to the guidance, the link between the two is considered to be valid; (2) If the user browses a knowledge card for a longer time than the set time, or exits after clicking for less than the set time, a reward or penalty is given according to the preset rules, and the weight of the relation edge of the knowledge card is updated; (2) The knowledge graph is checked periodically. If the weight of the relation edge of a certain implicit link in the topology network is lower than the set threshold, the connection is disconnected; (3) If more than a set number of users continue to search for another knowledge card after browsing a knowledge card, even if there is no initial link between the two, the relation edge attributes in the two knowledge cards are automatically updated, and a new link relationship is established between the two.

[0077] 4. Store knowledge cards as nodes in the knowledge graph in the backend, and establish a connection between the knowledge graph and the frontend display / interactive interface.

[0078] When a user queries or searches the knowledge graph through the front-end display and interactive interface:

[0079] (1) If the target content is associated with a knowledge card, while returning the knowledge card, multiple neighbor cards with a weight higher than the set value of the relation edge linked to it are also returned through random walk or breadth-first search, thus forming a recommendation list;

[0080] (2) Based on the user-input topic description, select matching knowledge cards from the knowledge card library, and generate a thematic exhibition page by calculating the topological relationships between the cards in real time. Specifically, the thematic exhibition page is generated in the following way:

[0081] (2.1) Based on the text input by the user, insert topic anchors into the existing knowledge graph by means of intent vectorization and injection of virtual nodes;

[0082] (2.2) Based on the maximum boundary relevance, select knowledge cards that meet the similarity requirements between the input topic and the selected set;

[0083] (2.3) Build links between the selected knowledge cards to form a story logic;

[0084] (2.4) Analyze the text of the knowledge cards using a large language model and automatically generate exhibition copy based on the input text;

[0085] (2.5) Determine the topology structure presented on the front-end display and interactive interface based on the temporal and spatial distribution characteristics of the knowledge cards.

[0086] Based on the understanding of those skilled in the art, the multi-agent collaborative guided tour method described above is entirely based on computer technology, and the entire implementation process includes data acquisition, organization, calculation, and result display.

[0087] The implementation of this technology can be embodied in an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for constructing multimodal intelligent knowledge cards and self-organizing links by topic as described above.

[0088] Part Two: A Specific Application Example

[0089] The following detailed explanation of the implementation scheme of the present invention will be based on a specific case implemented in a museum.

[0090] This method mainly includes four core steps: data acquisition and preprocessing (S1), multimodal information extraction and card instantiation (S2), construction of topic-based self-organizing links (S3), and interactive dynamic updating and retrieval (S4).

[0091] S1. Data Acquisition and Preprocessing

[0092] Data in the form of text, images, and videos were collected from multiple sources, and the raw data was preprocessed.

[0093] The raw data for this invention can come from various sources, such as: extracting data from local collections of text, images, and videos from cultural tourism, cultural relics, historical, or art collection institutions; or obtaining text, image, and video data from specified website page links or databases via web crawling technology. After obtaining the raw data, the extracted text is bound to its source-related multimodal content, such as images and / or audio / video, to establish original correspondences between text fragments and media files.

[0094] The following example illustrates how data can be collected via the internet:

[0095] 1. Targeted data collection: Configure a list of URLs for the target museum's API interface and authoritative cultural and tourism websites (such as the Palace Museum, Baidu Encyclopedia, and Wikipedia).

[0096] 2. DOM parsing and noise reduction: Use an HTML parser to extract the main text, metadata (Meta tags), image links, and video stream addresses, and remove advertisements and navigation bar noise.

[0097] 3. Alignment preprocessing: The extracted text is positioned and bound to the surrounding images / videos to establish the original correspondence group of [text fragment, media file].

[0098] S2. Multimodal Information Extraction and Knowledge Card Instantiation

[0099] Multimodal information is extracted from the preprocessed data, and several standard structured data objects are constructed as knowledge cards based on this information. The data objects are then encapsulated in a unified format to build a knowledge card database.

[0100] This step aims to transform unstructured data into structured data objects, which this invention calls "knowledge cards." The example processing flow is as follows:

[0101] 1. Entity Recognition (NER):

[0102] Identify basic information from the text, including: entities and related people, events, dates, and organizations.

[0103] This invention uses a BERT-CRF model finely tuned through a corpus in the cultural and tourism field to identify from text: cultural relics as entities, as well as people, events, times, organizations, etc. related to these entities.

[0104] 2. Relation Extraction (RE):

[0105] Multiple basic triplet relationships are formed using the basic information of entity objects.

[0106] Using existing mature joint extraction models, we extract many-to-many basic triple relationships between entities, such as: <Along the River During the Qingming Festival, Author, Zhang Zeduan>, <Along the River During the Qingming Festival, Belongs to, Northern Song Dynasty>.

[0107] 3. Visual content analysis:

[0108] Use ResNet or ViT models to extract features from image / video frames.

[0109] This invention uses ResNet or ViT models to extract features from image / video frames, identify object labels (such as "ceramics", "horse", "landscape"), or extract style features (such as "ink painting", "oil painting", "bronze").

[0110] 4. Card object encapsulation:

[0111] In this invention, the core data object is defined as a Knowledge Card, which carries the extracted multimodal information. Within the Knowledge Card data object, unique identifiers, entity types, and names are encapsulated with structured attributes, and relational edges, spatiotemporal profiles, multimodal content data, and vector embeddings are encapsulated using a standardized format.

[0112] by Figure 2 For example, this diagram depicts a structured "knowledge card object," where the knowledge card (KnowledgeCard) is the core, representing a node or entity. Relation edges define the connections between nodes, used to link different nodes together to form a network; a knowledge card (source node) can have multiple relation edges (edges pointing to other nodes). KnowledgeCard aggregates the following three auxiliary fields through "composition" relationships: Spatiotemporal Profile, Multimodal Payload, and Vector Embeddings. These are used to enrich the connotation of the knowledge card from the three dimensions of spatiotemporal, multimedia, and AI vectorization, respectively, enabling it to carry multi-dimensional information that can be understood by humans and processed efficiently by machines.

[0113] As a specific example, the attributes of a knowledge card in the diagram include: `card_id` (unique identifier); `name` (entity name); `entity_type` (entity type, such as person, place, event, etc.); `attributes` (stores additional attribute information in key-value pair form); `interaction_score` (interaction score, possibly used to measure the entity's popularity or importance); and `update_weight()` (meaning the knowledge card can dynamically adjust its importance based on the updated weight). The attributes of a relation edge include: `target_card_id` (target knowledge card ID); `link_type` (relation type, such as "located in", "contains", "is the founder of..."); `weight` (relation weight, indicating the strength of the association); `context_desc` (context description of the relationship); and `provenance` (data source or origin information). A spatiotemporal profile provides a spatiotemporal context for the entity, and its attributes include: `normalized_time` (normalized time); `time_span` (time span); `geo_coords` (geographic coordinates); and `semantic_location` (semantic location, such as "place of origin" or "collection location"). Multimodal payload provides entities with multimedia content, processing not only text data but also supporting the fusion of images, videos, audio, and 3D models. Its attributes include: text_summary (text summary); images (image list); video_stream (video stream); audio_guide (audio guide); and model_3d (3D model). Vector embeddings provide entities with machine learning / vectorized representations for AI retrieval, recommendation systems, or graph neural network computations, allowing systems to perform similarity searches based on semantics, time, or space. Its attributes include: unified_vector (unified vector representation); semantic_vec (semantic vector); temporal_vec (temporal vector); and spatial_vec (spatial vector).

[0114] Here is an example of a specific application:

[0115] The KnowledgeCard includes a unique identifier (card id) and entity type to distinguish different cultural objects; as well as a name and attribute set, supporting extended storage of structured information such as author, year, and location. Multimedia content is encapsulated through a multimodal payload field, integrating links to multimodal resources such as text, images, videos, and 3D models, achieving resource decoupling and on-demand loading. Crucially, this object contains a semantic vector field, composed of a vector array generated by a natural language model, used to express the semantic features of the card content, supporting intelligent retrieval based on semantic similarity.

[0116] The KnowledgeCard is encapsulated in JSON format, making it suitable for unified data representation and efficient interaction in distributed systems.

[0117] S3. Topic-based Self-organizing Link Algorithm

[0118] Based on the attributes of data in the knowledge card database, multidimensional hybrid embedding vectors are constructed using different knowledge cards; a multimodal semantic adaptive topology network under spatiotemporal constraints is established to achieve topic-based self-organizing links.

[0119] This is the core of the invention, used to establish many-to-many relationships between cards (i.e., constructing a multimodal semantic adaptive topology network under spatiotemporal constraints). Unlike traditional static knowledge graphs, this invention introduces a spatiotemporal decay factor and a user behavior feedback loop, enabling the connections between cards to dynamically evolve with time and interaction.

[0120] 3.1 Multimodal Spatio-Temporal Embedding

[0121] In order to calculate the similarity between different knowledge cards and solve the problem of the difficulty in integrating heterogeneous "visual", "text", "time" and "space" data in cultural and tourism data, this invention proposes a new approach of constructing "multidimensional hybrid embedding vectors".

[0122] 3.1.1 Semantic and Visual Feature Extraction:

[0123] We use a pre-trained multimodal model to extract semantic and visual features from knowledge cards and output a weighted fusion high-dimensional vector.

[0124] This example uses a pre-trained CLIP (Contrastive Language-Image Pre-training) model, or a similar multimodal model. CLIP, a multimodal model released by OpenAI, acts like a "text-image translator," understanding both images and text simultaneously. Through contrastive learning, it is trained on massive amounts of image-text pairs, mapping images and text to the same vector space. This allows for accurate image-text matching, zero-shot transfer learning, and the ability to recognize new categories without fine-tuning, making it a cornerstone for applications such as AI painting and image retrieval.

[0125] During the pre-training process of the multimodal model, a "fine-grained cultural and tourism feature mask" is introduced, which is used by inputting a text summary of the card and a representative image. When extracting image features, an attention mechanism is added to focus specifically on features that are strongly related to cultural and tourism content, such as "patterns", "materials" (e.g., the luster of bronze and jade) and "brushstrokes" (e.g., axe-cut texture), rather than general object recognition.

[0126] The multimodal model outputs a weighted, fused high-dimensional vector. ;

[0127]

[0128] in, A high-dimensional vector of features for the text summary of knowledge cards; A high-dimensional vector of features for representative images in the knowledge card; These are learnable hyperparameters; This refers to the modal weighting coefficient.

[0129] 3.1.2 Temporal Continuity Coding

[0130] The "dynasty" or "year" in the data is not just used as a label. This invention proposes to encode the time labels in the knowledge card continuously and map them into a continuous vector to calculate the time distance.

[0131] A variant of sinusoidal positional encoding is employed to achieve relative position awareness and smooth evolution representation in the time dimension.

[0132] in, Encode the time location vector; For the normalized year, The time dimension.

[0133] 3.1.3 Geospatial Coding

[0134] Map latitude and longitude or location names to spherical coordinate vectors By using the Sphere2Vec algorithm, proximity in geospatial space is maintained in vector space.

[0135] 3.1.4 Vector Composition (Card Representation)

[0136] The vectors obtained from the above operations are assigned learnable hyperparameters, and then vectors are synthesized.

[0137] The specific formula is as follows:

[0138]

[0139] in, This is the comprehensive representation vector of the identification card; This refers to vector composition; As learnable hyperparameters, they are dynamically adjusted for different query scenarios (such as "finding similar paintings" which focuses on semantics, and "finding nearby cultural relics" which focuses on geography).

[0140] 3.2 Explicit Linking & Ontology Alignment

[0141] Using the vectors already constructed, build a topological network with linked relationships by performing the following operations.

[0142] (1) To address the issue of multi-source heterogeneity of data (such as inconsistent descriptions of the same event by different museums), fuzzy ontology alignment logic is introduced; specifically including:

[0143] To address the issue of multi-source and heterogeneous cultural and tourism data, a fuzzy ontology alignment logic is introduced.

[0144] 3.2.1 Deterministic Rule Linking

[0145] Based on the triplet relations extracted from multimodal information, deterministic hard links are established between different knowledge cards.

[0146] For example, if the attributes of knowledge card A contain the ID of knowledge card B (such as author or dynasty), then a relation edge is established. If knowledge cards A and B are by the same author, that is... Then establish an edge Then, the initial confidence level. .

[0147] 3.2.2 Alias ​​and Fuzzy Alignment (Alias ​​Resolution)

[0148] We construct a synonym graph for the cultural and tourism sector and determine whether different knowledge cards point to the same entity based on name similarity and attribute overlap.

[0149] For example, if The label is "Houmuwu Ding". If the tag is "Simuwu Ding", then calculate the name similarity. and attribute overlap .

[0150] For example, if That is, comprehensive similarity score Greater than the alias determination threshold If they point to the same entity, then it is determined that they are linked to the same entity, or a "heterogeneous association" edge is established.

[0151] 3.2.3 Hierarchical Inference

[0152] Based on the constructed vectors, perform hierarchical reasoning to determine propagation links or complete explicit links.

[0153] For example, if "Kangxi" and "Kangxi" "Qing Dynasty" automatically inherits the attributes of its parent.

[0154] For artifacts lacking dates, attribute propagation is performed using the known dates of their "author" or "excavated tomb" to complete the explicit links.

[0155] 3.3 Implicit Theme Self-Organization

[0156] A dynamic subgraph generation method based on a gravity model is used to uncover unlabeled connections between knowledge cards.

[0157] This part is the "brain" of the topic-based self-organizing linking method, used to discover connections that are not explicitly labeled by humans. To this end, this invention proposes an innovative dynamic subgraph generation method based on a gravity model, specifically including:

[0158] 3.3.1 Define the “Theme Gravitational Field”:

[0159] Each knowledge card is treated as a point mass in a vector space, and the semantic attraction between different knowledge cards is defined.

[0160] Treat each card as a point mass in a vector space. Define two cards. and The "semantic attraction" between them :

[0161]

[0162] in, It is an environmental adjustment factor (Gravitational Constant / Tuning Factor) used to regulate the overall gravitational strength; and The quality of a card is determined by its popularity (number of views), authority (source museum level), and information richness (volume of multimodal data). Let be the Euclidean distance between two cards in the multimodal vector space; the more important and similar the cards are, the greater their attraction to each other, and the easier it is for them to self-organize together.

[0163] 3.3.2 Dynamic Clustering & Drift Detection:

[0164] A threshold for semantic gravity is set, and local clusters are formed through dynamic clustering. A time direction vector is introduced into the local clusters to detect the evolution trend of the vector over time.

[0165] This invention does not pre-determine the number of cluster centers and employs an incremental DBSCAN or Leiden algorithm. When a new card is added, if... If the threshold is exceeded, a new local cluster (Topic Cluster) is formed. For cultural and tourism scenarios, a time direction vector is introduced. Within a cluster, the evolution trend of the detection vector over time.

[0166] Example: Calculate the feature vector offset from "Yuan" to "Qing" in the "Blue and White Porcelain" cluster. If a card from an unknown era is found to be located within the evolutionary trajectory... The system automatically predicts the age of the style and establishes a logical link for its "style evolution".

[0167] 3.3.3 Narrative Chain Construction:

[0168] Based on the joint assumptions of entity continuity and time decay, a narrative chain with cause and effect is constructed.

[0169] For historical events, the algorithm seeks out the chain of "cause and effect." The construction of the narrative chain is not merely a chronological ordering, but rather based on the joint assumptions of "entity continuity" and "temporal decay." This invention abandons the traditional knowledge graph method of establishing connections solely through "text similarity" or "manual rules," combining decay models from physics with narrative logic from history. It automatically filters out invalid noise that is "accidentally similar but spans too large a time frame," accurately capturing the contextual thread of historical evolution, and achieving intelligent narrative organization of cultural and tourism knowledge cards.

[0170] Entity continuity hypothesis: If two events share key participants (such as the same general or the same national treasure), there is a high probability that there is a causal relationship between them.

[0171] The time decay hypothesis states that the causal influence between events decays exponentially as the time interval increases (i.e., what happened yesterday is more likely to lead to today's results, while what happened 100 years ago usually has an indirect impact on today).

[0172] The specific implementation process is illustrated below:

[0173] Define event (Precedence) to the event Causal strength score of (consequences) as follows:

[0174]

[0175] in,

[0176] : Representing events respectively and events The key entity set extracted (including people, places, important objects, and organizations);

[0177] : Representing events respectively and events The time of occurrence (normalized timestamp);

[0178] : Causal indicator function. If ( What happened After that, the value is 1; otherwise, it is 0. This ensures the temporal unidirectionality of causality.

[0179] This factor is used to quantify "who passed on the influence";

[0180] This factor is based on the "forgetting curve" principle, simulating the dissipation of historical influence.

[0181]

[0182] in, Historical influence half-life coefficient. This is an adjustable hyperparameter used to control the rate of decay over time.

[0183] Dynamic adjustment: Different types of history can be loaded for different types of history (such as "war history" which has a fast pace, and "art history" which evolves slowly). Value. War history. Larger (emphasizing immediate reaction), art history Less significant (allowing for generational influence, such as the "Renaissance" reviving the "Ancient Greek" style).

[0184] Short-term strong correlation: if When the interval between the "July 7th Incident" and the "fall of Beijing and Tianjin" is very small (e.g., only 20 days), the index term is close to 1, and the causal score mainly depends on the entity association. The algorithm tends to judge it as a "direct consequence".

[0185] Long-term weak association: if Larger values ​​(e.g., from "the Qin Dynasty's unification in 221 BC" to "the fall of the Eastern Han Dynasty in 220 AD"), with several terms approaching zero. Even if both are in China (the same geographical entity), the algorithm will determine that the causal relationship is extremely weak, avoiding the creation of absurd "butterfly effect" links.

[0186] 3.4 Calculation of Comprehensive Weights and

[0187] After the link relationships are constructed, the weights of the relationship edges of each knowledge card in the topological network are calculated comprehensively.

[0188] Place any two cards Edge weights between Defined as:

[0189] in,

[0190] Multimodal semantic cosine similarity;

[0191] Temporal proximity score, defined as (Gaussian kernel function, For time window hyperparameters; (These are the time position encoding vectors for cards A and B, respectively).

[0192] Geographic spatial proximity;

[0193] User interaction feedback factors (key to self-organizing evolution);

[0194] In the formula, These are semantic weight factor, temporal relevance weight, spatial distance weight, and interactive feedback enhancement weight, respectively. For normalization, the final edge weights can be determined. The value is limited to the range [0,1]. This quantization value is used to characterize the strength of the association between cards, facilitating unified calculation in subsequent path planning and knowledge recommendation algorithms.

[0195] 3.5 Adaptive Evolution

[0196] The present invention further utilizes reinforcement learning based on user behavior to achieve adaptive evolution of the topology network.

[0197] The self-organizing evolution mechanism is explained in detail below:

[0198] 3.5.1 Click to transmit:

[0199] If a user browses a knowledge card in a knowledge graph through the front-end display and interactive interface, and then continues to browse another knowledge card within a specified time according to the instructions, the link between the two is considered valid.

[0200] For example, when a user is browsing cards Afterwards, Clicked the card within seconds If the link is valid, then the link is considered valid.

[0201] 3.5.2 Browsing Experience Rewards and Penalties:

[0202] If a user spends more than the set time browsing a knowledge card, or exits before the set time has elapsed after clicking, a reward or penalty will be assigned according to preset rules, and the weights of the relationship edges of that knowledge card will be updated.

[0203] The weight update mechanism is as follows:

[0204]

[0205] in, This is the updated user interaction feedback factor; This refers to current (accumulated historical) user interaction feedback factors; The reward or penalty score for the current browsing behavior; The learning rate is determined based on the system's preset evolution rate.

[0206] If a user clicks and immediately exits the page, then A negative value (penalizing the link); a positive value if a longer period of "deep reading" occurs.

[0207] 3.5.3 Linking Pruning and New Growth:

[0208] The knowledge graph is periodically checked, and if the weight of a relation edge in a hidden link in the topology is lower than a set threshold, the connection is broken.

[0209] Link pruning: If a hidden link... Below the threshold If the connection is lost, the connection will be closed (forgetting mechanism).

[0210] Linking new life: If a large number of users are searching for knowledge cards Then search for knowledge cards (Even knowledge cards) If there is no initial link between them, a new relationship edge will be automatically generated. .

[0211] S4. Knowledge Graph Storage and Generative Retrieval

[0212] This invention stores the knowledge card library as a collection of nodes in the knowledge graph in the background and establishes a connection between the knowledge graph and the front-end display / interactive interface.

[0213] 4.1 Graph Database Storage: Knowledge cards are stored as nodes and links are stored as edges in the Neo4j knowledge graph.

[0214] The knowledge graph in the backend can run on a local or cloud server, while the display / interactive interface runs as a front-end application on fixed multimedia terminal devices set up for users in the exhibition hall or museum. These can be fixed devices or mobile terminals (such as smart tablets). In this way, users can easily perform queries or searches and obtain the displayed content on their terminal devices.

[0215] 4.2 Subgraph Retrieval:

[0216] If the target content is associated with a certain knowledge card, in addition to returning that knowledge card, multiple neighboring cards with linking edge weights higher than a set value are also returned through random walk or breadth-first search, thus forming a recommendation list.

[0217] 4.3 User-defined topic-based retrieval and dynamic subgraph generation:

[0218] Based on the topic description entered by the user, matching knowledge cards are selected from the knowledge card library, and a special exhibition page is generated by calculating the topological relationship between the cards in real time.

[0219] This invention can respond to unstructured topic descriptions input by users (such as "the evolution of women's makeup in the Tang Dynasty" or "the tragic aesthetics of war"), automatically select objects from a massive card library, and calculate the topological structure between them in real time to generate a temporary thematic exhibition page.

[0220] 4.3.1 Virtual Theme Anchor Injection

[0221] Based on the text input by the user, topic anchors are inserted into the existing knowledge graph by vectorizing intent and injecting virtual nodes.

[0222] When the user enters query text At this time, the system does not directly perform keyword matching, but instead performs the following operations:

[0223] Intent vectorization: Utilizing the multimodal encoder in S3.1, the intent is vectorized. Mapped to a high-dimensional vector ;

[0224] Injecting virtual nodes: Inserting a "topic anchor" into the existing knowledge graph space. Its position coordinates are determined by Confirmed. This node is temporary and invisible, used only for calculating "gravity".

[0225] 4.3.2 Card Selection Based on Maximum Boundary Correlation (MMR)

[0226] Based on the maximum boundary relevance, knowledge cards that meet the similarity requirements between the input topic and the selected set are selected.

[0227] For example, to avoid overly simplistic search results (e.g., inputting "porcelain" only returns "blue and white porcelain bowls"), the MMR (Maximum Marginal Relevance) strategy is used for selection:

[0228]

[0229] in, The score for the maximum marginal relevance; For cards Similarity to the topic entered by the user (ensuring relevance); For cards With the selected set Similarity of cards (to ensure diversity); These are dynamic weighting coefficients.

[0230] 4.3.3 Construction of Dynamic Contextual Links

[0231] Links are built between the selected knowledge cards to form a story logic.

[0232] After selecting the Top-K isolated cards, it is necessary to "build links" to connect them into a logical story. This involves generating temporary "context edges".

[0233] The text of the knowledge cards is analyzed using a large language model, and the exhibition text is automatically generated in combination with the input text.

[0234] For example, using large language models to analyze knowledge cards and The text, combined with user input The system automatically generates an explanatory text for this edge. The user searches for "tragic aesthetics." Card A is "Farewell My Concubine (Peking Opera)," and card B is "Van Gogh's Self-Portrait." The original graph has no connections; the generated edge attribute is: "Artistic Resonance Across Time and Space: Self-Destruction and Reconstruction."

[0235] 4.3.4 Adaptive Topology Display

[0236] Based on the temporal and spatial distribution characteristics of the knowledge cards, the topology structure presented on the front-end display and interactive interface is determined.

[0237] The topology of the front-end display is automatically determined based on the temporal and spatial distribution characteristics of the selected cards.

[0238] In linear narrative mode: the trigger condition is the temporal variance of the card set. Large, and spatial variance Small. The presentation is arranged chronologically and connected by a "narrative chain".

[0239] In map roaming mode: the trigger condition is that the temporal variance of the card set is small and the spatial variance is large (e.g., "Shops around Kaifeng Prefecture in the Song Dynasty"). The display involves projecting the cards onto a GIS map and connecting them according to their geographical proximity.

[0240] In the galaxy divergence mode: the trigger condition is belonging to a conceptual theme (such as "the introduction of Buddhism"). The display places the "virtual theme anchor" in the center, with cards arranged around it, and the lines representing the relevance weights.

[0241] Example: User inputs: "Food culture exchange along the ancient Silk Road." Search: The system selects "Grapes (Han Dynasty)," "Hu Bing (Tang Dynasty)," and "Blue and White Porcelain Painting of Foreigners Eating (Yuan Dynasty)." Bridging: The system discovers that "Zhang Qian's mission to the Western Regions" is a related event to "Grapes," so it automatically adds the "Zhang Qian" card as the source. Dynamic Linking: Although "Hu Bing" and "Blue and White Porcelain Painting" are originally unrelated, the algorithm calculates that they are both highly correlated with "Silk Road," thus establishing a connection and generating the explanation: "The influence of foreign influences from staple foods to table manners." Display: An automatic knowledge topic is generated, arranged along the map path of "Chang'an -> Dunhuang -> Western Regions."

[0242] Explanation of some terms related to the implementation of this invention:

[0243] 1. Named Entity Recognition (NER)

[0244] NER is a fundamental task in natural language processing, which aims to identify entities with specific meanings from unstructured text and classify them into predefined categories (such as names of people, places, organizations, and times).

[0245] Application in this invention: It is used to automatically extract "core elements of cultural tourism" from text in museum introductions, historical documents, or encyclopedia web pages. For example, it can extract "Wang Ximeng" (person), "Northern Song Dynasty" (era), "A Panorama of Rivers and Mountains" (cultural relic / artwork), and "Palace Museum" (organization) from a description. This is a fundamental step in constructing knowledge card attributes.

[0246] 2. BERT-CRF model

[0247] BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained language model that uses a bidirectional Transformer structure to deeply understand the semantics of words based on context (e.g., distinguishing whether "apple" refers to a fruit or a mobile phone).

[0248] CRF (Conditional Random Field): A discriminative probabilistic model, often attached to the BERT output layer. It can use constraints between labels (such as "B-PER" entities cannot be directly followed by "I-LOC" locations) to correct prediction results and improve the accuracy of sequence labeling.

[0249] Application in this invention: Texts in the cultural and tourism field often contain classical Chinese, proper nouns, or rare characters, resulting in low accuracy with traditional methods. This invention utilizes BERT's powerful semantic understanding capabilities to capture contextual features, and combines this with CRF to ensure the legitimacy of the label sequence, thereby accurately identifying special entities such as cultural relic names and dynasty reign titles.

[0250] 3. Relation Extraction (RE)

[0251] Building upon entity recognition, this process determines whether a semantic relationship exists between two entities and identifies the type of relationship. It is typically expressed in the form of <Entity 1, Relationship, Entity 2>.

[0252] Application in this invention: It is used to construct "explicit links" between knowledge cards. For example, upon recognizing the sentence "Along the River During the Qingming Festival was painted by Zhang Zeduan" in the text, the algorithm extracts and establishes the relationship: <Knowledge Card: Along the River During the Qingming Festival> --[Painter]--> <Knowledge Card: Zhang Zeduan>. This is a key technology for constructing the framework of a cultural tourism knowledge graph.

[0253] 4. ResNet (Residual Network)

[0254] It is a classic convolutional neural network (CNN) architecture. By introducing "skip connections", it solves the gradient vanishing problem that is prone to occur during the training of deep neural networks, which allows the network to be built very deep (such as 50 layers or 101 layers), thereby extracting more abstract and higher-level image features.

[0255] Application in this invention: As a visual feature extractor. When the system captures images of cultural relics, ResNet is used to extract features such as texture, shape, and color distribution, which are then used for subsequent image classification (e.g., identifying whether it is a "bronze" or "porcelain") or for calculating the similarity between images.

[0256] 5. ViT Model (Vision Transformer)

[0257] ViT directly applies the Transformer architecture from the NLP field to image processing. It segments images into small patches and processes these patches like word sequences. Unlike CNNs (such as ResNet) that focus on local features, ViT utilizes a "self-attention mechanism" to better capture the global contextual information of the image.

[0258] Application in this invention: In cultural and tourism scenarios, many artworks (such as landscape paintings and murals) emphasize overall composition and artistic conception. ViT can understand the overall style and layout of paintings better than traditional CNNs, thus more accurately associating images with similar styles (such as those that are both "grand and majestic") when "self-organizing by theme".

[0259] 6. Multimodal

[0260] This refers to the technology of integrating and analyzing multiple data modalities (such as text, images, audio, video, and 3D models). Its core lies in breaking down the barriers between different data modalities to achieve semantic alignment.

[0261] Application in this invention: The "knowledge card" of this invention is a multimodal container. The algorithm analyzes not only text descriptions, but also image content and video narrations. Through "multimodal unified embedding" technology, it enables the computer to understand that "an audio recording describing a horse's neighing," "a picture of a galloping horse," and "the word 'horse'" refer to the same concept.

[0262] 7. CLIP (Contrastive Language-Image Pre-training)

[0263] This is a multimodal pre-trained model proposed by OpenAI. It maps images and text to the same high-dimensional vector space through contrastive learning. If an image and a piece of text are semantically similar, they are closer in the vector space; otherwise, they are farther apart.

[0264] Application in this invention: This is key to achieving step S3, "self-organizing links by topic." It allows the system to directly calculate the similarity between text cards and image cards, thereby enabling cross-modal retrieval and association (for example, if a user searches for "auspiciousness," the system can automatically recommend image cards of "dragon-patterned jade pendant" through semantic vectors, even if the image title does not contain the words "auspiciousness").

[0265] 8. Vector Embedding: Semantic Vectors

[0266] Discrete objects (such as a word, an image, or a knowledge card) are transformed into continuous numerical vectors (arrays of numbers) that can be computed by a computer. The geometric distance between the vectors represents the degree of semantic similarity between the objects.

[0267] Application in this invention: Each "knowledge card" is ultimately compressed into a unique vector. The algorithm determines which cards should be automatically linked together (i.e., a "self-organizing" process) by calculating the cosine similarity between vectors.

[0268] 9. Self-Organization

[0269] This refers to the process by which a system automatically forms an ordered structure through the interaction between its internal elements without any specific external instructions or human intervention.

[0270] Application in this invention: The knowledge card network eliminates the need for manual configuration of each connection line. The algorithm automatically generates new links, breaks weak links, and adjusts link weights based on the similarity of card content (content clustering) and user browsing behavior (interactive feedback), allowing the knowledge graph to dynamically grow and optimize like a biological neural network.

[0271] 10. Sphere2Vec Algorithm

[0272] Sphere2Vec is a geospatial location encoding algorithm specifically designed for spherical surface data. Unlike traditional methods that treat latitude and longitude as planar coordinates, Sphere2Vec uses multi-scale sine and cosine functions (similar to location encoding in Transformers) as kernel functions to map the latitude and longitude coordinates of the sphere into high-dimensional dense vectors. Its core advantage lies in its ability to directly preserve the geodesic distance and spatial topological relationships of the Earth's surface in vector space, effectively solving the distance distortion and pole singularity problems caused by traditional planar projections when processing global-scale geographic data.

[0273] Application in this invention: Sphere2Vec is used to construct geospatial feature vectors for knowledge cards. Specifically, the system inputs the latitude and longitude of the unearthed sites of cultural relics, the locations of historical events, or the locations of museums into the model, generating vector representations that preserve spatial proximity. This allows the algorithm to go beyond simple textual semantic matching when calculating relationships between cards, calculating similarity from a "geographical" dimension (such as automatically associating cultural relics with similar styles from different countries along the "Silk Road," or aggregating all bronzes from the "Yangtze River Basin"), thereby achieving automatic knowledge clustering and implicit association recommendations based on geographical location.

[0274] 11. Euclidean distance

[0275] In a multidimensional vector space, the linear physical distance between two points (data objects) is used. In algorithms, it is used to quantify the degree of difference between two vectors; the smaller the distance value, the more similar the two objects are in terms of features, and vice versa.

[0276] Application in this invention: It is used to calculate the comprehensive difference between any two "knowledge cards" after integrating visual, textual, temporal, and spatial features. In the "topic gravity field" model, Euclidean distance exists as the denominator—that is, the closer the feature distance between two cards (e.g., the more similar the style, the closer the era), the greater the "semantic attraction" between them, thus being judged by the algorithm as having a strong correlation and automatically attracting them together to form a link.

[0277] 12. MMR Strategy

[0278] MMR is a ranking algorithm that balances maximizing the relevance between the query and the result with minimizing the similarity between the selected results using a linear weighting formula, thereby reducing information redundancy and improving result diversity.

[0279] Application in this invention: When responding to user-defined topics, this strategy ensures that the selected set of knowledge cards is not only closely related to the topic (e.g., all about "porcelain"), but also covers different eras, materials, or perspectives (avoiding all being "blue and white porcelain bowls"), thereby constructing a dynamic knowledge subgraph that is rich in content and unique to each other.

[0280] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A method for constructing multimodal intelligent knowledge cards and self-organizing links by topic, characterized in that, include: Data in text, image, and video formats was collected from multiple sources, and the raw data was preprocessed. Multimodal information is extracted from the preprocessed data, and several standard structured data objects are constructed as knowledge cards based on this information. The data objects are then encapsulated in a unified format to build a knowledge card database. Based on the attributes of data in the knowledge card database, multidimensional hybrid embedding vectors are constructed using different knowledge cards; a multimodal semantic adaptive topology network under spatiotemporal constraints is established to achieve topic-based self-organizing links. Knowledge cards are stored as nodes in the knowledge graph in the backend, and a connection is established between the knowledge graph and the frontend display / interactive interface.

2. The method according to claim 1, characterized in that, The data collection from multiple sources includes at least: data extraction from local collections of text, images, and videos of cultural and tourism, cultural relics, historical, or art collection institutions; or, obtaining text, image, and video data from specified website page links or databases through web crawling technology. The data preprocessing includes: binding the text extracted from the raw data to the images and / or videos related to its source in terms of location, and establishing the original correspondence between text fragments and media files.

3. The method according to claim 1, characterized in that, The multimodal information extraction includes: Identify basic information from the text, including: entities and related people, events, dates, and organizations; Multiple basic triplet relationships are formed using the basic information of entity objects; Use ResNet or ViT models to extract features from image / video frames.

4. The method according to claim 1, characterized in that, In the data objects that serve as knowledge cards, unique identifiers, entity types, and names are encapsulated with structured attributes, and relation edges, spatiotemporal profiles, multimodal content data, and vector embeddings are encapsulated with standardized formats. The attributes of the relation edges include: the target knowledge card ID to which it points, the type of relation, the weight of the relation, the contextual description of the relation, and the data source or origin information; the attributes of the spatiotemporal profile include: normalized time, time span, geographic coordinates, and semantic location; the attributes of the multimodal content data include: text summary, image list, video stream, audio guide, and 3D model; the attributes of the vector embedding include: unified vector representation, semantic vector, temporal vector, and spatial vector.

5. The method according to claim 1, characterized in that, The construction of the multidimensional hybrid embedding vector specifically includes: (1) Semantic and visual feature extraction: The semantic and visual features in the knowledge card are extracted using a pre-trained multimodal model, and a weighted fusion high-dimensional vector is output; (2) Time continuity encoding: The time tags in the knowledge cards are continuously encoded and mapped to continuous vectors to calculate the time distance; (3) Geospatial coding: Map latitude and longitude or location names to spherical coordinate vectors so that the proximity of geographic space is preserved in the vector space; (4) Assign learnable hyperparameters to the vectors obtained by the above operations, and then perform vector synthesis.

6. The method according to claim 1, characterized in that, Using the constructed vectors, construct a topological network with linked relationships by performing the following operations: (1) To address the issue of multi-source heterogeneity of data, fuzzy ontology alignment logic is introduced; specifically including: Based on the relationships extracted from multimodal information, deterministic hard links are established between different knowledge cards; a thesaurus is constructed to determine whether different knowledge cards point to the same entity through name similarity and attribute overlap; hierarchical reasoning is performed based on the constructed vectors to determine propagation links or complete explicit links; (2) A dynamic subgraph generation method based on the gravity model to explore unmarked connections between knowledge cards; Specifically, this includes: treating each knowledge card as a point mass in a vector space and defining the semantic attraction between different knowledge cards; setting a critical value for semantic attraction and forming local clusters through dynamic clustering; introducing a time direction vector into the local clusters to detect the evolution trend of the vector over time; and constructing a narrative chain with cause and effect based on the joint assumption of entity continuity and time decay. (3) After completing the link relationship construction, calculate the weight of each knowledge card relationship edge in the topology network.

7. The method according to claim 1, characterized in that, Reinforcement learning is performed based on user behavior to achieve adaptive evolution of the topological network; specifically including: (1) When a user browses a knowledge card in the knowledge graph through the front-end display and interactive interface, if the user continues to browse another knowledge card according to the guidance within the specified time, the link between the two is considered to be valid. (2) If a user browses a knowledge card for more than the set time, or exits before the set time is reached after clicking, a reward or penalty will be given according to the preset rules, and the weight of the relationship edge of the knowledge card will be updated. (2) Periodically check the knowledge graph. If the weight of a relation edge of a certain implicit link in the topology network is lower than the set threshold, then disconnect the connection. (3) If more than a set number of users continue to search for another knowledge card after browsing a certain knowledge card, the relationship edge attributes in the two knowledge cards will be automatically updated to establish a new link relationship between them, even if there is no initial link between them.

8. The method according to claim 1, characterized in that, When a user queries or searches the knowledge graph through the front-end display and interactive interface: (1) If the target content is associated with a knowledge card, while returning the knowledge card, multiple neighbor cards with a weight higher than the set value of the relation edge linked to it are also returned through random walk or breadth-first search, thus forming a recommendation list; (2) Based on the topic description input by the user, select matching knowledge cards from the knowledge card library and generate a special exhibition page by calculating the topological relationship between the cards in real time.

9. The method according to claim 8, characterized in that, The special exhibition page is generated using the following methods: (1) Based on the text input by the user, insert topic anchors into the existing knowledge graph by means of intent vectorization and injection of virtual nodes; (2) Based on the maximum boundary relevance, select knowledge cards that meet the similarity requirements between the input topic and the selected set; (3) Build links between the selected knowledge cards to form a story logic; (4) Use a large language model to analyze the text of the knowledge cards and automatically generate exhibition copy based on the input text; (5) Determine the topology structure presented on the front-end display and interactive interface based on the temporal and spatial distribution characteristics of the knowledge cards.

10. An electronic device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a method for constructing multimodal intelligent knowledge cards and self-organizing links by topic as described in any one of claims 1 to 9.

Images