Method and apparatus for processing oral data on cultural heritage

Abstract

The application provides a method and device for processing oral data about cultural heritage, which can be applied to the fields of artificial intelligence and data processing technology. The method comprises: performing semantic structuralization processing on the oral data according to object attribute information to obtain structured knowledge text; performing data statistics on meta-information related to text features in the oral data to obtain statistical results; performing subject modeling analysis on the oral data to obtain subject evolution analysis results; extracting entities and entity relationships from the oral data to construct a knowledge graph based on the extraction results; analyzing the structured knowledge text, the statistical results, the subject evolution analysis results and semantic features determined based on the knowledge graph based on a preset report template using a large language model to obtain a visual display report; storing the visual display report in a database, and searching in the database based on a target prompt word using a retrieval enhancement generation algorithm to obtain a question and answer report related to the target prompt word.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and data processing technology, and specifically to a method and device for processing oral data related to cultural heritage. Background Technology

[0002] Oral history, as an important medium for recording intangible cultural heritage, historical memory, and social change, holds significant value in contemporary cultural heritage preservation and historical research. However, the processing of oral histories relies heavily on manual transcription and coding, resulting in technical problems such as low processing efficiency and significant subjective bias. Summary of the Invention

[0003] In view of the above problems, this application provides a method and device for processing oral data on cultural heritage.

[0004] According to the first aspect of this application, a method for processing oral data on cultural heritage is provided, comprising: acquiring object attribute information and oral data from multiple target objects regarding a target cultural heritage item, wherein the oral data describes the change process of the target cultural heritage item, and the object attribute information represents the degree of association between the target object and the target cultural heritage; performing semantic structuring processing on the oral data based on the object attribute information to obtain structured knowledge text; performing data statistics on meta-information related to text features in the oral data to obtain statistical results; performing theme modeling analysis on the oral data to obtain theme evolution analysis results; extracting entities and entity relationships from the oral data to construct a knowledge graph based on the extraction results, wherein entities represent narrative elements related to the target cultural heritage, and entity relationships represent semantic associations between narrative elements; analyzing the structured knowledge text, statistical results, theme evolution analysis results, and semantic features determined based on the knowledge graph using a large language model based on a preset report template to obtain a visualization report; storing the visualization report in a database, and using a retrieval enhancement generation algorithm based on target prompts to retrieve questions and answers related to the target prompts from the database.

[0005] According to an embodiment of this application, based on target prompt words, a retrieval enhancement generation algorithm is used to search the database to obtain a question-and-answer report related to the target prompt words. This includes: classifying the target prompt words to obtain prompt categories; processing the target prompt words according to the retrieval enhancement generation algorithm corresponding to the prompt categories to obtain a target retrieval strategy; performing a multi-dimensional retrieval in the database based on the target retrieval strategy to obtain initial question-and-answer results; performing factual verification on the initial question-and-answer results to obtain verification results; and obtaining a question-and-answer report related to the target prompt words based on the verification results and the initial question-and-answer results.

[0006] According to embodiments of this application, target prompt words are processed using a retrieval enhancement generation algorithm corresponding to the prompt category to obtain a target retrieval strategy. This includes: when the prompt category is a global question category, calling a word analysis algorithm to process the target prompt words to obtain a first sub-strategy, which includes prompt questions for retrieval based on statistical results; calling a knowledge graph mining algorithm to process the target prompt words to obtain a second sub-strategy, which includes prompt questions for retrieval based on the knowledge graph; calling a topic modeling analysis algorithm to process the target prompt words to obtain a third sub-strategy, which includes prompt questions for retrieval based on topic evolution analysis results; calling a filtering algorithm to process the target prompt words to obtain a fourth sub-strategy, which includes prompt questions related to object attribute information; and obtaining the target retrieval strategy based on the first, second, third, and fourth sub-strategies.

[0007] According to embodiments of this application, multi-dimensional retrieval is performed in the database based on a target retrieval strategy to obtain initial question-and-answer results, including: keyword retrieval in the database based on the target retrieval strategy to obtain keyword retrieval results; semantic retrieval in the database based on the target retrieval strategy to obtain semantic retrieval results; knowledge graph structured retrieval in the database based on the target retrieval strategy to obtain knowledge graph retrieval results; and initial question-and-answer results are obtained based on the keyword retrieval results, semantic retrieval results, and knowledge graph retrieval results.

[0008] According to embodiments of this application, semantic structuring processing is performed on spoken data based on object attribute information to obtain structured knowledge text. This includes: structuring the object attribute information to obtain structural meta-information, which includes a collection timestamp, collection location information, object association category, and object association weight; performing speech conversion processing on the spoken data to obtain text data; extracting semantic features from the text data using an embedding model to obtain text semantic features; analyzing the spoken data using a large language model based on multiple research dimensions as prompt word prefixes to obtain content summaries corresponding to each of the multiple research dimensions; and encapsulating the structural meta-information, text semantic features, and content summaries to obtain structured knowledge text.

[0009] According to embodiments of this application, statistical analysis is performed on meta-information related to text features in spoken data to obtain statistical results, including: performing word feature statistics on spoken data to obtain word statistical features; performing sentence feature statistics on spoken data to obtain sentence statistical features; performing sliding window statistics on words in spoken data based on a preset sliding window size to obtain a word co-occurrence matrix, which represents the semantic association network topology between words in the text; performing sentiment recognition on spoken data to obtain sentiment classification results; performing statistical analysis on structural meta-information to obtain meta-information cross-statistical results; and encapsulating the word statistical features, sentence statistical features, word co-occurrence matrix, sentiment classification results, and meta-information cross-statistical results to obtain statistical results.

[0010] According to embodiments of this application, topic modeling analysis is performed on spoken data to obtain topic evolution analysis results, including: analyzing spoken data using a clustering algorithm to obtain multiple clusters; extracting keywords from each of the multiple clusters to obtain target words corresponding to each cluster; processing the target words and representative texts corresponding to each cluster using a large language model based on prompt word templates to obtain topic categories corresponding to each cluster, where the representative texts are determined based on spoken data; and performing temporal evolution analysis on topic categories based on the collection timestamps and timestamp weights associated with the topic categories to obtain topic evolution analysis results.

[0011] According to an embodiment of this application, entity and entity relationship extraction is performed on spoken data to construct a knowledge graph based on the extraction results, including: extracting entities and entity relationships from spoken data to obtain multiple triples; constructing an initial knowledge graph based on the multiple triples; weighting the multiple triples with confidence based on the object association weights corresponding to the entities to obtain the extraction results; and updating the initial knowledge graph based on the extraction results to obtain the knowledge graph.

[0012] According to embodiments of this application, the method further includes: constructing a social relationship network based on person entities in the knowledge graph as nodes and the co-occurrence frequency or interaction description intensity of person entities as edge weights; constructing a temporal network by performing temporal evolution analysis on key event entities based on multiple timestamps extracted from the knowledge graph; and analyzing the structured knowledge text, statistical results, topic evolution analysis results, semantic features determined based on the knowledge graph, social relationship network, and temporal network using a large language model to obtain a visual report.

[0013] A second aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0014] According to the oral history data processing method for cultural heritage provided in this application, by acquiring object attribute information representing the degree of association between the target object and the target cultural heritage item, and assigning differentiated weights to the oral history data of different participants, inheritors, and other target objects, the problem of multi-source data conflict and noise interference in traditional oral history processing is effectively resolved. Furthermore, through semantic structuring processing, unstructured natural language narratives are transformed into machine-readable structured knowledge text, achieving a leap from the initial stage of "speech-to-text + keyword retrieval," giving oral history data a computational foundation that allows for retrieval, association, and reasoning. On this basis, by statistically analyzing the meta-information related to textual features in the oral history data, replacing manual coding with machine quantification, objective measurements of richness, frequency, and co-occurrence relationships are achieved at the lexical level, significantly improving processing efficiency and reducing researcher subjectivity bias. Furthermore, by automatically identifying core issues through topic modeling analysis, and... By tracing the historical changes and evolution of its weights, static text is transformed into a dynamic historical context, solving the problem of insufficient depth in the form of the results. Furthermore, by extracting entities and their relationships to construct a knowledge graph, narrative elements such as people, skills, and places, and their semantic relationships are explicitly solidified in a graph structure, deeply integrating knowledge in the field of cultural heritage to form a navigable and verifiable semantic network. Then, based on a pre-set report template, a large language model is used to perform fusion analysis and visualization of structured knowledge text, statistical results, theme evolution, and semantic features of the knowledge graph, achieving efficient transformation from fragmented materials to a systematic research report with a sense of cultural context. Finally, by storing the visualized report in a database and providing interactive question-and-answer services based on a retrieval enhancement generation algorithm, an end-to-end retrieval path is established from the original narrative to an interpretable, verifiable, and interactive knowledge system, enabling users to accurately obtain personalized question-and-answer reports through target prompts. Attached Figure Description

[0015] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings.

[0016] Figure 1 The illustration shows an application scenario of the oral data processing method and device for cultural heritage according to an embodiment of this application.

[0017] Figure 2 A flowchart of an oral data processing method for cultural heritage according to an embodiment of this application is shown.

[0018] Figure 3 A flowchart illustrating the process of obtaining a question-and-answer report according to an embodiment of this application is shown.

[0019] Figure 4 A flowchart illustrating the target retrieval strategy according to an embodiment of this application is shown.

[0020] Figure 5 A block diagram of an electronic device suitable for implementing an oral data processing method for cultural heritage, according to an embodiment of this application, is shown. Detailed Implementation

[0021] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] The processing of oral history relies heavily on manual transcription, coding, and thematic summarization, resulting in low efficiency, significant subjective bias, a lack of structured and quantitative analysis, conflicts in the fusion of multi-source oral data, and a lack of depth in the integrated output. This has led to oral history research remaining at a rudimentary stage of "speech-to-text + keyword retrieval," failing to deeply integrate knowledge from the cultural heritage field. Furthermore, general natural language processing tools lack a deep understanding of cultural heritage terminology, historical events, and relationships between figures, making it difficult to accurately extract key information. Therefore, an end-to-end intelligent retrieval path from original oral narratives to an interpretable, verifiable, and interactive knowledge system has not been established, particularly in generating culturally contextualized analytical reports and personalized search queries; no systematic and feasible technical solutions exist.

[0026] In view of this, embodiments of this application provide a method for processing oral data related to cultural heritage. The method includes: semantically structuring the oral data based on object attribute information to obtain structured knowledge text; statistically analyzing the meta-information related to text features in the oral data to obtain statistical results; performing topic modeling analysis on the oral data to obtain topic evolution analysis results; extracting entities and entity relationships from the oral data to construct a knowledge graph based on the extraction results; analyzing the structured knowledge text, statistical results, topic evolution analysis results, and semantic features determined based on the knowledge graph using a large language model based on a preset report template to obtain a visualization report; storing the visualization report in a database, and then using a retrieval enhancement generation algorithm based on target prompts to retrieve a question-and-answer report related to the target prompts from the database.

[0027] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0028] Figure 1 The illustration shows an application scenario of the oral data processing method and device for cultural heritage according to an embodiment of this application.

[0029] like Figure 1 As shown, the application scenario 100 of the oral data processing method and device for cultural heritage according to this application embodiment includes a terminal device 101, a server 102, and a network 103. The network 103 serves as a medium for providing a communication link between the terminal device 101 and the server 102. The network 103 may include various connection types, such as wired, wireless communication links, or fiber optic cables. The test server 102 may be a server providing testing services. The terminal device 101 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0030] Server 102 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal device 101 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0031] It should be noted that the oral data processing method for cultural heritage provided in this application embodiment can generally be executed by server 102.

[0032] It should be understood that Figure 1 The number of terminal devices shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices and servers.

[0033] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0034] Figure 2 A flowchart of an oral data processing method for cultural heritage according to an embodiment of this application is shown.

[0035] like Figure 2 As shown, the method includes operations S210 to S270.

[0036] In operation S210, object attribute information and oral data of multiple target objects for the target cultural heritage item are obtained. The oral data describes the change process of the target cultural heritage item, and the object attribute information represents the degree of association between the target object and the target cultural heritage.

[0037] The target audience is the interviewees who receive the interview request.

[0038] Acquire multimodal oral data, including audio or video data from the target individuals regarding the evolution of the target cultural heritage item. Interviews with the target individuals can be conducted using a web-scale supervised pretraining for speech recognition (Whisper) model enhanced with a cultural heritage lexicon to improve the accuracy of recognizing dialects, local accents, and historical proper nouns (such as "blue calico" and "Nuoxi").

[0039] During the recognition process, errors in sentence attribution caused by the target subject switching speech or speech overlap are corrected. If there are segments in the recognition results with a confidence level lower than the preset threshold, for example, the output confidence level is less than 0.6, they are marked as items requiring manual correction, and the user is prompted to intervene in subsequent processes. The preset threshold can be set according to the actual collection environment and equipment performance, for example, it can be 0.6.

[0040] During the identification process, the object attribute information is collected synchronously. The object attribute information includes identity tags, object age range, collection time, collection location and association weight. Identity tags and association weights together reflect the degree of association between the target object and the target cultural heritage. For example, the identity tags are "person who experienced it" or "intangible cultural heritage inheritor". The association weight of a person who experienced it is 1.0 and the association weight of an intangible cultural heritage inheritor is 0.9.

[0041] Time information is jointly identified by rule templates (such as "XX year", "1950s") and large language models, and uniformly converted into the ISO 8601 standard format; geographic information is aligned with geographic information databases through place name entity recognition models to achieve standardized coding; association weights are used for subsequent conflict detection and confidence assessment, and their values ​​can be configured according to academic norms or project needs.

[0042] In operation S220, the spoken data is semantically structured based on the object attribute information to obtain structured knowledge text.

[0043] The audio or video data in the oral data is transcribed into text data. The large language model (Qwen3) is used to perform cleaning and semantic standardization on the initially transcribed text data, automatically removing colloquial filler words, repairing semantic breaks, and integrating identity awareness mechanism and cultural common sense constraints to automatically delete colloquial filler words such as "um", "ah", and "that", and merge paused and broken sentences.

[0044] The text data is segmented into Chinese using dictionaries and algorithms specifically designed for the cultural heritage field. The segmentation results are then input into an embedding model (Canon-embedding-V1) to generate dense vector representations. During vector generation, the target object identifier and timestamp are retained as additional dimensions to support subsequent multi-condition semantic retrieval. The generated dense vector representations are stored in a vector database, and an index mapping is established between the vector representations and the original oral data.

[0045] By using a large language model, the text data is summarized from multiple research dimensions. Based on unique event identifiers, the summary, transcribed text data, word segmentation results, dense vector representations, etc. are encapsulated into structured knowledge text and stored as "text state" in a polymorphic knowledge base.

[0046] If any sub-operation fails (e.g., confidence level is too low), the system records the error type and location and highlights it in the human-computer interaction interface, waiting for user confirmation or correction; the event identifier item generates an auto-incrementing sequence to ensure global uniqueness and traceability.

[0047] In operation S230, data statistics are performed on the metadata related to text features in the oral data to obtain statistical results.

[0048] Multi-level semantic analysis is performed on the transcribed text data. Text features represent quantitative indicators of the linguistic attributes of the spoken data; these quantitative indicators include at least lexical richness, frequency, and length indicators, as well as sentence length, information density, and co-occurrence relationships at the sentence level.

[0049] For example, at the word level, the Type-Token Ratio (TTR), word frequency distribution, and average word length are calculated; at the sentence level, the average sentence length and information density are statistically analyzed; the word co-occurrence matrix of two words appearing simultaneously within a window is statistically analyzed, and the word co-occurrence matrix represents the degree of semantic association; fine-grained sentiment analysis is performed using a large language model, such as for words like "nostalgia," "pride," and "regret"; at the same time, cross-statistics are performed on meta-information such as the target object's identity and age, and a structured language and meta-information analysis report is generated based on the statistical results of multi-level semantic analysis.

[0050] In operation S240, topic modeling analysis is performed on the oral data to obtain topic evolution analysis results.

[0051] The vector content and methods for thematic model analysis are designed based on the average text length of the transcribed text data and the planned research direction and dimensions. If the average text length exceeds 3000 words, it is recommended to use summary text and abstracts generated from each research dimension for vectorization; if the average text length does not exceed 3000 words, the text data can be directly vectorized, and the selected model is the Canon-embedding-V1 embedded model.

[0052] A nonlinear dimensionality reduction algorithm (Uniform Manifold Approximation and Projection, UMAP) is used to reduce high-dimensional vectors to a low-dimensional space. An embedding-based unsupervised clustering method is then used to cluster the low-dimensional vectors, automatically identifying semantic topic clusters.

[0053] Keywords and semantic tags are generated for each topic using a large language model; further, by combining time data, the topics are mapped to preset time intervals, their time-series weights are calculated and evolution curves are plotted, and the large language model provides attribution explanations for trend changes, forming structured topic evolution analysis results.

[0054] In operation S250, entities and entity relationships are extracted from the oral data to construct a knowledge graph based on the extraction results. Entities represent narrative elements related to the target cultural heritage, and entity relationships represent the semantic associations between narrative elements.

[0055] Based on large language models and prompting engineering, triples that conform to subject-relationship-object are extracted from text data to obtain entities and entity relations. Unique identifiers are assigned to entities and attempts are made to link them to the domain knowledge base. Entities with low confidence or newly appearing are marked as "awaiting manual verification".

[0056] For example, entities include representative figures, locations, skill names, festivals, etc.; entity relationships include teaching relationships, participation relationships, origin relationships, etc.

[0057] If there are low-confidence triples or unresolved entity ambiguities during the knowledge graph construction process, the system will perform confidence weighting based on the association weights of the target objects for unresolved entity ambiguities. When the conflict exceeds a preset threshold, the system will prompt the user to intervene and generate a "list to be verified" which will be highlighted in the human-computer interaction interface for expert intervention and correction.

[0058] When operating the S260, based on a preset report template, the large language model is used to analyze structured knowledge text, statistical results, topic evolution analysis results, and semantic features determined based on knowledge graphs to obtain a visual report.

[0059] Deep semantic feature mining is performed on the fused knowledge graph: In the social network dimension, with human entities as nodes and co-occurrence frequency or interaction intensity as edge weights, indicators such as degree centrality and betweenness centrality are calculated to identify core target objects and community structures; in the temporal dimension, a three-dimensional network of "entity-event-time" is constructed to generate the evolution path of key skills or events.

[0060] By using a unified project identifier, the semantic features of structured knowledge text, statistical results, topic evolution analysis results, and knowledge graphs are aligned.

[0061] The data is organized into structured data objects according to a preset academic report template. The preset academic report template includes six core chapters: "Project Overview", "Linguistic Features", "Emotional Stance", "Thematic Evolution", "Personal Relationships", and "Cultural Value". Among them, the unified project identifier can be an auto-incrementing sequence within the project to ensure the consistency and traceability of data across stages.

[0062] Under the Retrieval-Augmented Generation (RAG) mechanism, the large language model, combined with prompt word templates (such as "historian's perspective" and "intangible cultural heritage inheritor's perspective"), transforms structured data into coherent, traceable, and culturally contextualized natural language narratives. Interactive visualization components are embedded to generate visual reports that support long-term archiving and exploration. The report output is in Markdown+CSV format.

[0063] In operation S270, the visualization report is stored in the database. Based on the target prompt, the database is searched using a search enhancement generation algorithm to obtain a question and answer report related to the target prompt.

[0064] Store the visualization report in the database.

[0065] Target prompts can be obtained by combining structured knowledge text, statistical results, topic evolution analysis results, and the patterns of semantic feature analysis, in order to improve retrieval accuracy and efficiency.

[0066] For example, first load the corresponding perspective template (such as "historian's perspective" or "intangible cultural heritage inheritor's perspective") according to the purpose of the report (such as "academic archiving", "public communication", or "policy recommendation"); then combine the patterns in the structured knowledge (such as "88% of those over 60 years old mentioned 'ancestral teachings'"), the key analysis information in the charts in the statistical results (such as "the weight of the theme 'skill innovation' increased by 45% after 2010"), and the high-confidence triples (such as "Chen XX - [teaching] -> paper-cutting skills, confidence level 0.92") into enhanced target prompts.

[0067] By using retrieval-enhanced generation algorithms, text data fragments are associated with knowledge graph entities under the constraint of target prompt words, generating coherent, traceable, and culturally contextualized question-and-answer reports.

[0068] For example, the Q&A report stated: "In a village in xx province, 88% of elderly paper-cutting artists still have a glimmer in their eyes when they mention the words 'ancestral precepts.' These are the paper-cutting techniques passed down by the third-generation inheritor of Chen XX's craft, and the bottom line he has guarded for thirty years. However, a quiet change is taking place after 2010: when the frequency of 'technical innovation' mentioned in oral accounts increases by 45%, Chen XX begins to hand his scissors to the young people in the village. From 'ancestral precepts' to 'innovation,' a pair of scissors has cut open the boundary between tradition and modernity, and also cut out a new path for the living inheritance of intangible cultural heritage..."

[0069] According to the embodiments of this application, by acquiring object attribute information representing the degree of association between the target object and the target cultural heritage item, differentiated weights are assigned to the oral history data of different participants, inheritors, and other target objects, effectively resolving the multi-source data conflict and noise interference problems in traditional oral history processing. Furthermore, through semantic structuring processing, unstructured natural language narratives are transformed into machine-readable structured knowledge text, achieving a leap from the initial stage of "speech-to-text + keyword retrieval," giving oral history data a computational foundation that allows it to be searchable, associative, and inferential. Based on this, by performing data statistics on meta-information related to textual features in the oral history data, replacing manual coding with machine quantification, objective measurements of richness, frequency, co-occurrence relationships, etc., are achieved at the lexical level, significantly improving processing efficiency and reducing researcher subjectivity bias. Through topic modeling analysis, core issues are automatically identified and their diachronic weights are tracked. By focusing on changes and evolutionary trajectories, static text is transformed into a dynamic historical context, addressing the lack of depth in the presentation of research findings. Furthermore, by extracting entities and their relationships to construct a knowledge graph, narrative elements such as figures, techniques, and locations, along with their semantic relationships, are explicitly solidified in a graph structure. This deeply integrates knowledge from the cultural heritage field, forming a navigable and verifiable semantic network. Then, based on a pre-defined report template, a large language model is used to fuse and analyze structured knowledge text, statistical results, thematic evolution, and the semantic features of the knowledge graph, achieving efficient transformation from fragmented materials to a systematic research report with cultural context. Finally, by storing the visualized report in a database and providing interactive question-and-answer services based on a retrieval-enhanced generation algorithm, an end-to-end retrieval path is established from the original narrative to an interpretable, verifiable, and interactive knowledge system, enabling users to accurately obtain personalized question-and-answer reports using target keywords.

[0070] According to embodiments of this application, semantic structuring processing is performed on spoken data based on object attribute information to obtain structured knowledge text. This includes: structuring the object attribute information to obtain structural meta-information, which includes a collection timestamp, collection location information, object association category, and object association weight; performing speech conversion processing on the spoken data to obtain text data; extracting semantic features from the text data using an embedding model to obtain text semantic features; analyzing the spoken data using a large language model based on multiple research dimensions as prompt word prefixes to obtain content summaries corresponding to each of the multiple research dimensions; and encapsulating the structural meta-information, text semantic features, and content summaries to obtain structured knowledge text.

[0071] The object attribute information consists of the original attribute data of the respondents who described the oral data, such as name, age, occupation, and interview background.

[0072] The object attribute information is cleaned, mapped, and formatted to be transformed into standardized structured metadata that is machine-readable, searchable, and associative. The structured metadata consists of standardized descriptive fields after normalization, including fields such as collection timestamp, collection location information, object association category, and object association weight.

[0073] The collection timestamp is used to accurately record the date and time of the interview; the collection location information is the geographical coordinates or administrative division of the interview location; the subject association category is the identity label, such as "person who experienced it", "intangible cultural heritage inheritor", "researcher", "ordinary villager" or "local cultural and historical worker", etc.

[0074] The object association weight is a quantitative value reflecting the degree of association between the target object and the target cultural heritage in the overall study. For example, the object association weight of the witness is 1.0, the object association weight of the intangible cultural heritage inheritor is 0.9, the object association weight of the researcher is 0.8, and the object association weight of the ordinary villager is 0.6.

[0075] Use speech recognition models to transcribe audio or video data from spoken data into text data.

[0076] We use dictionaries and algorithms for the cultural heritage field to perform Chinese word segmentation on text data, and input the segmentation results into the embedding model (Canon-embedding-V1) to generate dense vector representations.

[0077] The word segmentation process uses a dictionary and algorithm optimized for the cultural heritage field, which can accurately segment historical terms (such as "tie-dye" and "shehuo"), compound proper nouns (such as "blue calico making technique") and dialect words (such as "stove painting" and "walking book"), avoiding semantic fragmentation caused by general word segmenters. The word segmentation results are output in the form of word sequences, with each word accompanied by a part-of-speech tag and an identifier indicating whether it is a domain term.

[0078] The embedding model is a pre-trained language model such as Canon-embedding-V1, which encodes the results of each semantically complete sentence segmentation into a 768-dimensional dense vector. The generated vectors are stored in a vector database (Facebook AI SimilaritySearch, FAISS) and an index mapping with the spoken data is established. The vector dimension can be set according to the model architecture and retrieval accuracy requirements, and can be 768-dimensional, 768-dimensional, or 512-dimensional.

[0079] By encoding text data into dense vectors using an embedding model, and using research dimensions as cue word prefixes to guide a large language model, the full text is read in a targeted manner and a summary is generated, outputting a content summary that corresponds one-to-one with each research dimension.

[0080] Based on each preset research dimension, a large language model and cue word engineering are used to traverse the text data, generating a structured content summary for each research dimension. Then, the content summaries from each research dimension are combined to generate an overall summary text. The content summary is a concise summary text generated for each research dimension.

[0081] The preset threshold for the content summary can be set according to the actual density of the spoken content, for example, it can be 300 words, 500 words or 800 words.

[0082] Pre-defined research dimensions may include individual memory, cultural genes, and contemporary expression.

[0083] Structured knowledge text includes text data, structured meta-information, word segmentation results, dense vector representation, and content summary; structured knowledge text is stored as "text-state" data in a polymorphic knowledge base.

[0084] According to embodiments of this application, statistical analysis is performed on meta-information related to text features in spoken data to obtain statistical results, including: performing word feature statistics on spoken data to obtain word statistical features; performing sentence feature statistics on spoken data to obtain sentence statistical features; performing sliding window statistics on words in spoken data based on a preset sliding window size to obtain a word co-occurrence matrix, which represents the semantic association network topology between words in the text; performing sentiment recognition on spoken data to obtain sentiment classification results; performing statistical analysis on structural meta-information to obtain meta-information cross-statistical results; and encapsulating the word statistical features, sentence statistical features, word co-occurrence matrix, sentiment classification results, and meta-information cross-statistical results to obtain statistical results.

[0085] The oral data was transcribed to obtain text data. Then, linguistic feature statistics were performed on the text data at the word level to obtain word statistical features. Word statistical features include word richness, word frequency, and average word length.

[0086] Word richness is obtained by calculating the Type-Token Ratio (TR), which is the ratio of the number of different word types to the total number of words. Word frequency is obtained by traversing the word sequence after word segmentation, counting the number of occurrences of each word, and generating a word frequency table in descending order. The average word length is the sum of the number of characters in all words divided by the total number of words. Among them, the TTR threshold, the number of high-frequency words to be selected, and the unit of average length calculation can be set according to the analysis granularity requirements. For example, TTR can be set to 0.45, the top 100 high-frequency words can be selected, and the average length unit is the number of Chinese characters.

[0087] Sentence-level linguistic feature analysis is performed on text data to obtain sentence statistical features, which include sentence length and information density.

[0088] Sentence length is the number of words contained in each sentence. The system calculates the average sentence length, maximum sentence length, and standard deviation of sentence length for all sentences. Information density is defined as the ratio of the number of content words (such as nouns, verbs, and adjectives) in a sentence to the total number of words. It is used to measure the amount of effective semantic information carried per unit length. Among them, the content word categories can be specifically defined based on the content of relevant dictionaries in the field of cultural heritage and the research content.

[0089] A word co-occurrence matrix was constructed from the text data of a single interview to reveal the semantic association structure. A fixed sliding window was used to slide across the text data, word by word, to count the co-occurrence frequency of word pairs within the window. This data was then aggregated into a word co-occurrence matrix, which was further mapped to the network topology.

[0090] For example, the sliding window size can be set to 5 words. If two words appear simultaneously in the window, the co-occurrence count is incremented by 1. The matrix is ​​normalized and then used for subsequent co-occurrence matrix construction. The sliding window size can be specifically set according to the text semantic coherence requirements, such as 3 words, 5 words, or 8 words.

[0091] Sentiment analysis is performed on text data to obtain sentiment classification results, in order to identify the sentiment tendencies and stance distribution in the narrative.

[0092] Sentiment analysis employs a large language model to categorize each piece of text data into positive, negative, or neutral sentiment categories and outputs confidence scores. Sentiment tags are further associated with the identity and events of the target audience, forming an analysis view of "identity-event-sentiment (type)-confidence score-sentiment fine-grainedness". The number of sentiment categories and classification thresholds can be set according to the specific research objectives. For example, it can be expanded to fine-grained tags such as "nostalgia", "pride", and "regret", and the classification confidence threshold can be set to 0.8.

[0093] For example, House Repair - Neutral - Nostalgic - 0.85.

[0094] The system performs statistical analysis on the structural metadata, aggregates the structural metadata, and generates pie charts of target object distribution ratios, heat maps of geographical distribution, and scatter plots of age versus collection time. It also calculates the differences in other statistical dimensions among different identity groups. The geographical coding standard can adopt the administrative division code GB / T 2260, and the age range can be divided into "over 60 years old", "40–60 years old", or "under 40 years old", etc., thus obtaining the cross-statistical results of the metadata.

[0095] The outputs of each operation are integrated to form structured statistical results. These results include: word frequency statistics table, TTR and average word length indicators, sentence length distribution histogram, co-occurrence matrix, sentiment classification results, and meta-information cross-statistics table. All results are linked through a unified project identifier. If any sub-operation fails to complete due to data sparsity or formatting abnormalities (such as a single spoken record being less than 10 words), the system records the abnormality type and skips the record, ensuring the robustness of the overall analysis process.

[0096] According to the embodiments of this application, multi-dimensional semantic analysis is performed sequentially on the text data after oral history data conversion. The word co-occurrence matrix explicitly presents the implicit semantic relationships between words in the text through a network topology structure, making up for the shortcomings of traditional manual coding in capturing deep semantic connections. The sentiment classification results add a quantitative representation of the emotional dimension to the oral history data. Meta-information cross-statistics analyzes the correlation between the background attributes of the interviewee, such as collection time, region, and identity, and text features, effectively revealing the distribution patterns and differences of oral history data under different groups or spatiotemporal conditions. Finally, the heterogeneous statistical results of the lexical layer, syntactic layer, semantic network layer, sentiment layer, and meta-information layer are uniformly encapsulated, forming a reusable, searchable, and visualized standardized statistical asset. This significantly improves the globality, objectivity, and analytical depth of oral history data processing and provides a high-quality data foundation for the subsequent generation of multi-perspective intelligent reports based on retrieval enhancement. This greatly reduces the time cost and subjective bias of manual reading, coding, and summarizing of each article in traditional oral history research.

[0097] According to embodiments of this application, topic modeling analysis is performed on spoken data to obtain topic evolution analysis results, including: analyzing spoken data using a clustering algorithm to obtain multiple clusters; extracting keywords from each of the multiple clusters to obtain target words corresponding to each cluster; processing the target words and representative texts corresponding to each cluster using a large language model based on prompt word templates to obtain topic categories corresponding to each cluster, where the representative texts are determined based on spoken data; and performing temporal evolution analysis on topic categories based on the collection timestamps and timestamp weights associated with the topic categories to obtain topic evolution analysis results.

[0098] The transcribed text data undergoes vectorization and dimensionality reduction, followed by clustering using the density-based clustering algorithm (HDBSCAN). Clusters are automatically created based on semantic similarity. Each cluster consists of a key vector center and adjacent sentence vectors. A large language model is used to generate topic semantic labels, such as "rules for skill inheritance," "family memory construction," and "festival ritual procedures." The minimum cluster size (min_cluster_size) can be set according to the data scale and topic granularity requirements; for example, the embedding dimension can be 768, and min_cluster_size can be set to 20.

[0099] Keyword extraction and semantic enhancement analysis were performed for each cluster. First, the class-level term frequency-inverse document frequency (c-TF-IDF) value of each word within the cluster was calculated, and the Top-K high-frequency words were selected as target words. The representative text corresponding to the topic and the target words were input into the large language model, and enhanced topic semantic tags were generated through prompt word templates. The target words in the enhanced topic semantic tags were used as topic categories, and the topic categories combined statistical significance and semantic representativeness.

[0100] The representative text can be a summary of the content most relevant to the topic.

[0101] The value of Top-K can be flexibly set according to the complexity of the topic, for example, 5, 10 or 15.

[0102] For example, the prompt template could be "Please select the 5 target words that best represent the theme of 'skills inheritance' from the following text".

[0103] We performed a time-series evolution analysis on each theme to reveal its rise and fall trends at different historical stages.

[0104] First, the text data corresponding to each cluster is mapped to a preset time interval according to its associated timestamp (e.g., "1940–1960", "1960–1980", "1980–2000", "2000 to present"). Second, the weight of the topic within each time interval is calculated, and the weight is defined as the proportion of the number of texts belonging to the topic within the interval to the total number of texts in the interval. Finally, a line graph of the topic-time weight evolution is plotted, and the weight changes are attributed and explained using a large language model. For example, "the weight of the topic 'technological innovation' increased by 45% after 2000, mainly due to a significant increase in the frequency of mention by younger inheritors."

[0105] The granularity of the time interval division can be set according to the specific research objectives. For example, it can be divided according to ten years, twenty years, or major historical events.

[0106] The topic evolution analysis report includes: a list of clusters (including topic semantic tags, core text, and target words), a topic-time weight matrix (line graph), a topic evolution (line graph) curve, and attribution explanation text; all results are indexed bidirectionally by unified project identifiers and oral data and knowledge graph entities, and stored as a sub-module of "Topic Evolution Analysis" in the text state of the polymorphic knowledge base.

[0107] According to the embodiments of this application, the report supports interactive filtering and selection by users according to dimensions such as topic, year, and interviewee identity, and provides high-level semantic support for the subsequent overall visualization of the report and user search response; if a topic cannot be analyzed in time series due to data sparsity, such as appearing only in a single time period, the system marks it as a "static topic" and classifies it separately, and provides user reminders to ensure the completeness and interpretability of the analysis results.

[0108] According to an embodiment of this application, entity and entity relationship extraction is performed on spoken data to construct a knowledge graph based on the extraction results, including: extracting entities and entity relationships from spoken data to obtain multiple triples; constructing an initial knowledge graph based on the multiple triples; weighting the multiple triples with confidence based on the object association weights corresponding to the entities to obtain the extraction results; and updating the initial knowledge graph based on the extraction results to obtain the knowledge graph.

[0109] The text data is segmented into segments based on sentence integrity, phrases, and word count. Then, through prompt engineering, the output format is defined as (subject, relation, object) triples. The model is guided to identify cultural heritage-related entities (such as people, places, skill names, festival activities) and their interrelationships (such as "transmission", "participation", "origin") implicit in the text, resulting in multiple triples.

[0110] In cases where the relationship type is ambiguous, such as "teaching" in "Old Zhang teaches Little Wang to cut paper" (which could mean "master-apprentice inheritance" or "skill transmission"), the system automatically generates a list of candidate relationships and triggers a second confirmation of the model.

[0111] The set of relationship types can be specifically defined according to the preset data model schema of the knowledge graph in the field of cultural heritage. For example, it can include categories such as "lineage of heritage", "skill association", "event participation", and "spatial affiliation".

[0112] An initial knowledge graph is constructed based on the extracted triples. The initial knowledge graph is organized in a resource description framework format, with each triple represented as (subject-verb-object). The system first assigns a unique identifier to each newly identified entity, such as PERSON_00123 and SKILL_00456, and aligns it with standard entities in the pre-built cultural heritage domain knowledge base using an entity linking algorithm. If the entity is a newly emerging cultural concept (such as the local craft "stove painting"), a new node is created and marked as "awaiting manual verification". During the initial graph construction process, the source text identifier, target object, and confidence score of each triple are recorded synchronously to support subsequent source tracing and conflict detection.

[0113] The confidence score is calculated by comparing entity matching and relationship reasonableness scores with labeled data in the same field. The scoring threshold can be set according to the specific reliability requirements of extraction, for example, it can be 0.8.

[0114] Perform entity disambiguation and cross-source fusion on the initial knowledge graph.

[0115] Entity disambiguation includes synonym merging and normalization. The system uses a large language model to clean the content of text data by interpreting pronouns or phrases with directional characteristics, such as "she" and "that old house". In addition, for different expressions of the same entity, the large language model is used to judge and merge them through context engineering. For example, X University, X1 University, and the first modern university in China are the same entity. At the same time, by combining context and meta-information, such as region, time, and target object, ambiguous entities (such as multiple "Master Chen") are distinguished, and a unique target object identifier is generated through clustering or rule filtering.

[0116] Cross-source fusion refers to the logical alignment and content integration of triples from different target objects that point to the same event or relationship. When multiple target objects have different descriptions of the same fact (such as "who taught paper-cutting skills"), the system first maps each triple to a unified event template and extracts key elements (subject, relationship, object). Then, it weights each triple with confidence based on pre-set object association weights. If the high-weighted narrative conflicts with other narratives on key elements (such as "Master Chen" vs. "Master Li"), and the difference exceeds a preset conflict threshold, the system generates a conflict prompt, listing the content of each narrative, source identity, association weight, and original quoted fragment, and provides three options: "adopt the high-weighted narrative," "adopt the user-specified narrative," or "retain multiple versions." After receiving the user's confirmation instruction, the system updates the graph entities and entity relationships according to the user's selection, completing the fusion and obtaining the knowledge graph.

[0117] According to embodiments of this application, the method further includes: constructing a social relationship network based on person entities in the knowledge graph as nodes and the co-occurrence frequency or interaction description intensity of person entities as edge weights; constructing a temporal network by performing temporal evolution analysis on key event entities based on multiple timestamps extracted from the knowledge graph; and analyzing the structured knowledge text, statistical results, topic evolution analysis results, semantic features determined based on the knowledge graph, social relationship network, and temporal network using a large language model to obtain a visual report.

[0118] The system performs in-depth mining and multi-dimensional analysis on the merged knowledge graph. In the social network analysis dimension, the system constructs a social relationship network using the person entities in the knowledge graph as nodes and "co-occurrence frequency" or "interaction description intensity" as edge weights.

[0119] By calculating semantic features such as node centrality, betweenness centrality, and cohesive subgroup coefficient, the core inheritors and community structure in the network are accurately identified, clearly presenting the degree of connection between individuals and the characteristics of group distribution.

[0120] In the temporal evolution analysis dimension, the system combines the timestamps recorded in the knowledge graph to perform timeline alignment processing on the core event entities, and constructs a three-dimensional temporal network of "entity-event-time". Through this network, the evolution path of the event can be intuitively generated, and the changing patterns of the event's development process in the time dimension can be clearly seen.

[0121] The outputs of the aforementioned operations are integrated to form a structured cultural heritage knowledge graph database. The integrated outputs include: standardized triplet files, entity disambiguation maps, social relationship networks, and temporal networks. All outputs are indexed bidirectionally using unified project identifiers and text data, thematic evolution analysis results, and dense vector representations, and stored as "graph states" in a polymorphic knowledge base. This knowledge graph database supports recursive abbreviation queries (SPARQL Protocol and RDF Query Language, SPARQL), subgraph export, entity tracing, and relationship path exploration, and provides structured knowledge support for visual reports and user-interactive Q&A. If low-confidence triples or unresolved entity ambiguities exist during the graph construction process, the system generates a "list to be verified" and highlights it in the human-computer interaction interface for expert intervention and correction.

[0122] By using a unified project identifier, the semantic features of structured knowledge text, statistical results, topic evolution analysis results, and knowledge graphs are aligned.

[0123] The data is organized into structured data objects according to a preset academic report template, which includes six core chapters: "Project Overview", "Linguistic Features", "Emotional Stance", "Theme Evolution", "Personal Relationships", and "Cultural Value". Under the retrieval enhancement generation mechanism, the large language model, combined with the prompt word template, transforms the structured data into a coherent, traceable, and culturally contextualized natural language narrative, embedding interactive visualization components to generate a visual presentation report that supports long-term archiving and exploration.

[0124] By integrating a domain-wide language model, a conflict resolution mechanism driven by object association weights, a text-graph-vector three-state knowledge representation, and a generative question-answering system with reflective capabilities, this approach not only significantly improves the automation and objectivity of oral history processing but also realizes a human-machine collaborative analysis paradigm that is "machine-computable, human-understandable, and expert-intervention-enabled." This provides an intelligent solution that combines efficiency, accuracy, and cultural sensitivity for the digital archiving, academic research, and public dissemination of intangible cultural heritage.

[0125] Figure 3 A flowchart illustrating the process of obtaining a question-and-answer report according to an embodiment of this application is shown.

[0126] like Figure 3 As shown, the process includes operations S310 to S350.

[0127] In operation S310, the target prompt words are classified to obtain the prompt category.

[0128] In operation S320, the target prompt words are processed according to the retrieval enhancement generation algorithm corresponding to the prompt category to obtain the target retrieval strategy.

[0129] When operating S330, a multi-dimensional search is performed in the database based on the target retrieval strategy to obtain initial question and answer results.

[0130] In operation S340, the initial question and answer results are verified for factual accuracy, and the verification results are obtained.

[0131] During operation of S350, a question-and-answer report related to the target prompt word is obtained based on the verification results and the initial question-and-answer results.

[0132] The target prompts are categorized based on preset criteria to obtain prompt categories. Prompt categories can be simple question categories, difficult question categories, global question categories, or irrelevant question categories.

[0133] For example, the preset criteria include: questions with a single entity in the target prompt or explicit interrogative words (who / when / whether) can be classified as simple questions; questions with multiple intentions or explicit logical relationships (comparison / analysis / causation) can be classified as difficult questions; open-ended exploratory queries (comprehensive analysis / organization) can be classified as global questions; and questions unrelated to cultural heritage can be classified as irrelevant questions.

[0134] The target retrieval strategy is a retrieval execution plan customized for a specific prompt category, which includes parameters such as database selection, retrieval method, retrieval path, reordering rules and generation constraints.

[0135] Multi-dimensional retrieval refers to a joint retrieval operation across multiple data sources, such as structured knowledge text databases, statistical result databases, topic databases, and knowledge graph databases.

[0136] For searches of irrelevant question categories, queries that are determined by the classifier to be unrelated to the field of oral history of cultural heritage will trigger a rejection mechanism and return a question and answer report with the preset message "This question is outside the scope of this system's services".

[0137] For simple question categories, the target retrieval strategy employs a four-level progressive rewriting approach: Level 1 (colloquial standardization): colloquial and fragmented expressions (e.g., "What did Master Chen say?") are transformed into grammatically correct and semantically complete structured questions (e.g., "What does Master Chen say about the origin of paper-cutting techniques?"); Level 2 (derivative question generation): based on the entity relationship network of the cultural heritage knowledge graph, 3-5 semantically related but complementary derivative questions are automatically generated (e.g., extending from "inheritance methods" to dimensions such as inheritance rules, historical changes, regional comparisons, generational differences, and future challenges); Level 3 (compound intent decomposition): for queries with multiple implicit intents (e.g., "technique characteristics and their historical origins"), rule-based decomposition is performed into a sequence of atomic-level sub-questions; Level 4 (synonym expansion): semantic expansion of target prompts is performed using a pre-built cultural heritage thesaurus, with the number of expansions dynamically set to 3-5 variants based on the degree of entity ambiguity.

[0138] The target retrieval strategy also includes constructing multi-path heterogeneous retrieval. For each question / sub-question, a multi-path heterogeneous retrieval approach is adopted, based on semantic retrieval, keyword retrieval, and knowledge graph structured retrieval using multi-vector fusion.

[0139] The target retrieval strategy also includes a two-stage re-ranking mechanism: the first-stage re-ranking is based on the fine-grained semantic matching model Qwen3-Reranker-8B, which calculates the relevance score between the query and candidate paragraphs; the second-stage re-ranking introduces authority weighting, which adjusts the relevance score according to the object association weight of the source target object (eyewitness 1.0 > non-heritage inheritor 0.9 > researcher 0.8) (content retrieved from the high-value memory bank is directly included in the Top-5 and does not participate in the retrieval and screening); the re-ranked Top-N evidence fragments (such as Top-10) and 5 data from the high-value memory bank are used as context input; among them, the N value, weight coefficient and model type can be specifically configured according to the answer quality requirements, for example, N can be set to 10 and the eyewitness weight can be set to 1.0.

[0140] For retrieval of difficult question categories, the target retrieval strategy is as follows: Qwen3-8B-Instruct, finely tuned from a cultural heritage corpus, is invoked. The original query question and target prompts are input, generating a structured sequence of sub-questions. Each sub-question satisfies the following: it contains only a single retrieval intent; it can be independently retrieved and answered through subsequent operations; when logical dependencies exist between sub-questions, dependency numbers are explicitly marked; for each sub-question, a retrieval strategy targeting simple question categories is invoked. The evidence returned by each retrieval is merged and deduplicated to form a dedicated evidence set for the sub-question (including 30-60 evidence fragments).

[0141] For each sub-question and its evidence set, a two-level reordering is performed, and the Top-20 evidence fragments are selected and input into the Qwen3-8B main answer model; atomic-level sub-answers are generated, and the list of evidence fragment sources and authority weights are recorded; the system aggregates the initial question-and-answer results and sub-answers of all sub-questions to form an enhanced context package.

[0142] The initial question-and-answer results are generated by the system based on the retrieved evidence and are not verified.

[0143] The initial question-and-answer results undergo factual reflection, comparing their consistency with evidence paragraphs in the database and marking those that do not cite high-weight evidence or contradictory statements. If the factual verification passes, the process proceeds to the expert judgment mechanism and the construction of the memory knowledge base. If it fails, the initial question-and-answer results, reflected errors, and supplementary evidence are combined into enhanced prompts, and the answer is retrieved again, with a maximum of 3 iterations.

[0144] The initial question-and-answer results that pass the verification are solidified in a structured form into the memory bank and trigger the expert judgment process; the question-and-answer pairs that pass the judgment are injected into the memory knowledge base for direct retrieval and reuse in subsequent simple questions.

[0145] Factual verification refers to the process of verifying the consistency, accuracy, and traceability between the initial question-and-answer results and the evidence paragraphs.

[0146] Based on the validation results and initial question-and-answer results, a question-and-answer report related to the target prompt words is generated. The report includes: the large model's response; dimensional analysis cards (visualization of word / graph / topic dimension tool calls); self-supervised enhancement explanations (annotated with enhancement evidence such as "enhancement based on topic evolution analysis and expert consensus"); and reflective validation results (showing the factual validation pass status). The report is output in interactive hypertext markup language (HTML) format, supporting click-through to trace back to the original audio and video, forming a complete interactive loop.

[0147] According to the embodiments of this application, the target prompts are classified into simple, difficult, global, and irrelevant question difficulty modes, which effectively achieves dynamic adaptation of computing resources and processing depth, significantly reducing unnecessary reasoning overhead while ensuring analysis quality. The initial question-and-answer results obtained from multi-way retrieval are subject to factual verification, which effectively filters retrieval noise and generated illusions, ensuring the reliability of the analysis conclusions. The verified results are integrated with the initial question-and-answer results into a complete question-and-answer report, enabling the retrieval enhancement generation system to have the ability to systematically analyze open-ended questions, rather than just returning fragmented facts, which significantly improves the analytical depth, evidence completeness, and conclusion credibility of oral history intelligent question answering in professional research scenarios.

[0148] Figure 4A flowchart illustrating the target retrieval strategy according to an embodiment of this application is shown.

[0149] like Figure 4 As shown, the process includes operations S410 to S450.

[0150] When operating S410, if the prompt category is the global question category, the word analysis algorithm is invoked to process the target prompt words and obtain the first sub-strategy. The first sub-strategy includes the prompt question retrieved from the statistical results.

[0151] When operating S420, the knowledge graph mining algorithm is invoked to process the target prompt words and obtain the second sub-strategy. The second sub-strategy includes prompt questions for retrieval in the knowledge graph.

[0152] In operation S430, the topic modeling analysis algorithm is invoked to process the target prompt words and obtain the third sub-strategy. The third sub-strategy includes prompt questions for retrieval based on the topic evolution analysis results.

[0153] In operation S440, the filtering algorithm is invoked to process the target prompt words and obtain the fourth sub-strategy, which includes prompt questions related to object attribute information.

[0154] In operation S450, the target retrieval strategy is obtained based on the first sub-strategy, the second sub-strategy, the third sub-strategy, and the fourth sub-strategy.

[0155] For example, when the query is an open-ended, exploratory query (such as "a comprehensive analysis of the inheritance system of paper-cutting techniques" or "a review of the evolution of techniques and regional differences over the past century"), the suggested category is the global question category.

[0156] When the prompt category is a global problem category, the large language model calls five types of analysis tools in parallel to generate structured analysis clues, which serve as the guiding basis for sub-problem decomposition.

[0157] The intent of the target prompt words is analyzed. When the word-level analysis requirement is identified, the preset word analysis tool is called to extract and calculate the statistical results in a targeted manner to obtain the first sub-strategy. The word analysis tool includes functions such as word frequency statistics, type-lexical ratio and average word length calculation, sentence length distribution statistics, word co-occurrence matrix construction, sentiment classification and meta-information cross-statistics, so that the large language model can generate question and answer content related to the target prompt words based on the extraction results.

[0158] Knowledge graph mining algorithms are knowledge graph mining tools that are condensed from underlying data resources such as standardized triples, entity disambiguation maps, social visualization networks, and time-series networks, as well as graph analysis algorithms such as node centrality calculation, betweenness centrality analysis, and agglomerative subgroup coefficient detection. They are used by large models in deep mining scenarios of relational networks.

[0159] The second sub-strategy is a query strategy generated after intent parsing of the target prompt words, used to guide knowledge graph retrieval. It takes the form of a prompt question or query template adapted to the query syntax of the graph database. This operation identifies the core entities and relational intents in the target prompt words through a semantic parsing module, maps them to node and edge types in the knowledge graph, and then generates a prompt question containing entity anchors, relational paths, and filtering conditions, thus obtaining the second sub-strategy.

[0160] The topic modeling and analysis algorithm is a topic modeling and analysis tool based on underlying data resources and visualization assets such as topic cluster list (including semantic tags, core text, keywords, etc.), topic-time weight matrix (line chart), topic evolution (line chart), and attribution explanation text. It is used by large models in topic discovery and evolution tracking scenarios.

[0161] The third sub-strategy is a query strategy generated after intent parsing of target prompts, used to guide retrieval in the results of topic evolution analysis. It is represented by prompt questions that include time windows, topic tags, and evolution trend constraints.

[0162] For example, the time range and topic keywords of the target prompt words are parsed, and the natural language query is mapped to coordinate intervals and semantic labels in the topic-time weight matrix; the topic clustering and evolution tracking algorithms in the tool are called to locate the core text and keywords in the topic cluster list, and the cross-time trend changes are analyzed based on the topic-time weight matrix and topic evolution line graph; at the same time, semantic association is performed on the driving factors of topic changes by combining the attribution explanation text; finally, based on the above topic analysis results, prompt questions for retrieving topic evolution trajectory, trend changes and attribution explanations of changes are generated as the third sub-strategy output.

[0163] The filtering algorithm is an intelligent filtering tool that is based on metadata indexes (identity category, age, region, collection time) and knowledge graph related node type indexes. It performs Boolean logic filtering (such as "1980–2000 AND North China AND eyewitness identity" combined filtering) and outputs a set of historical material subset identifiers that meet the conditions. It is a functional module that can be called by large models in scenarios where object attribute information is precisely filtered.

[0164] The fourth sub-strategy is a query strategy generated after the target prompt words are analyzed for intent, which is used to guide precise filtering based on object attribute information. It is expressed as a combination of prompt questions that include attribute constraints such as identity category, age range, region, collection time, and knowledge graph node type constraints.

[0165] For example, attribute qualifiers are extracted and Boolean logic is parsed from the target prompt words to map the descriptions such as time range, region, and identity category in the natural language query to constraints on the metadata index fields, and the target node type in the knowledge graph is identified. Then, the Boolean logic filtering algorithm in the tool is called to perform intersection, union, and difference operations on the metadata index and the knowledge graph node type index to quickly locate the historical material subset that meets multiple constraints. Finally, based on the above filtering results, a prompt question is generated to perform compound condition filtering in the database and output a set of unique identifiers for the historical material subset. This is output as the fourth sub-strategy to guide subsequent multi-dimensional filtering and retrieval based on object attribute information.

[0166] The four types of tools are required by the large language model. Depending on the actual needs, one or more models can be called. Among them, the filtering tool plays more of an auxiliary role to the other three tools, making it easier to achieve more accurate analysis.

[0167] In one embodiment, multi-dimensional retrieval is as shown in formula (1):

[0168] (1).

[0169] in, The representation hint category is a global problem category. This is a global problem indicator function. For pre-built toolsets, For filtering tools, As a word analysis tool, For knowledge graph mining tools, To model and analyze algorithms for the topic.

[0170] The system receives query text and analysis clues. The target retrieval strategy is to call four tools to generate a logically hierarchical sequence of atomic sub-problems, perform heterogeneous retrieval in parallel and independently for each sub-problem (sharing the same retrieval engine with simple / hard problems), and perform two-level reordering for each sub-problem and the evidence returned by each retrieval tool to finally obtain an enhanced context package.

[0171] According to embodiments of this application, the first sub-strategy is responsible for linguistic statistical retrieval at the word level, the second sub-strategy is responsible for relational network retrieval in the knowledge graph, the third sub-strategy is responsible for temporal semantic retrieval at the topic evolution level, and the fourth sub-strategy is responsible for metadata filtering of object attributes. By decomposing a single target prompt word into four complementary sub-strategies, multi-dimensional retrieval coverage is achieved, ensuring that subsequent multi-path retrieval can accurately match relevant content in different data formats, significantly improving the comprehensiveness and targeting of the retrieval.

[0172] According to embodiments of this application, multi-dimensional retrieval is performed in the database based on a target retrieval strategy to obtain initial question-and-answer results, including: keyword retrieval in the database based on the target retrieval strategy to obtain keyword retrieval results; semantic retrieval in the database based on the target retrieval strategy to obtain semantic retrieval results; knowledge graph structured retrieval in the database based on the target retrieval strategy to obtain knowledge graph retrieval results; and initial question-and-answer results are obtained based on the keyword retrieval results, semantic retrieval results, and knowledge graph retrieval results.

[0173] The target retrieval strategy is a dynamically configured set of retrieval parameters based on the prompt category, including a keyword thesaurus, matching patterns, and weight thresholds. Based on the preset keyword rules in the target retrieval strategy, the system performs word segmentation and stemming on the target prompt words, and then performs Boolean or TF-IDF weighted retrieval in the database's text index to filter out document fragments containing the target keywords, thus obtaining the keyword retrieval results.

[0174] The system uses an embedding model to encode target prompts into query vectors. Based on the similarity measurement method and recall Top-K parameters set by the target retrieval strategy, it performs an approximate nearest neighbor search in the vector database (Facebook AI Similarity Search, FAISS) to recall a batch of text chunks that are closest to the query vector, thereby obtaining semantic retrieval results.

[0175] Based on the graph query template in the target retrieval strategy, the system extracts entities from the target prompts as query anchors, performs multi-hop relationship traversal or subgraph matching in the knowledge graph, retrieves the set of triples associated with the entities and the associated paths, and thus obtains the knowledge graph retrieval results.

[0176] The system performs confidence weighting and deduplication on keyword retrieval results, semantic retrieval results, and knowledge graph retrieval results. Based on the fusion rules set by the target retrieval strategy (such as linear weighting), the mixed evidence is reordered. The selected high-quality evidence fragments are concatenated with target prompts to form enhanced context, which is then input into a large language model for generation, thus obtaining the initial question-answering results.

[0177] The three features—knowledge text, statistical results, and topic distribution—are forcibly unified into a 768-dimensional space for weighted fusion to eliminate semantic drift caused by dimensional mismatch; the query and document undergo a completely isomorphic feature extraction process to ensure retrieval alignment.

[0178] According to the embodiments of this application, by fusing three heterogeneous retrieval results, the initial question-and-answer results combine the precision of keyword retrieval, the generalization of semantic retrieval, and the structured relevance of knowledge graphs, significantly improving the comprehensiveness and diversity of generated content and reducing the information blind spots caused by a single retrieval path; the three retrieval methods complement each other, enabling the system to simultaneously handle factual, analytical, and relational queries, significantly improving the adaptability and recall rate of complex oral history question-and-answer scenarios.

[0179] Figure 5 A block diagram of an electronic device suitable for implementing an oral data processing method for cultural heritage, according to an embodiment of this application, is shown.

[0180] Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0181] like Figure 5 As shown, an electronic device 500 according to an embodiment of this application includes a processor 501, which can perform various appropriate actions and processes according to a program stored in ROM 502 (Read-Only Memory) or a program loaded from storage portion 508 into RAM 503 (Random Access Memory). The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0182] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in one or more memories.

[0183] In embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0184] The embodiments of this application, and the method flow according to the embodiments of this application, can be implemented as a computer software program. For example, an embodiment of this application includes a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of the embodiments of this application. In the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0185] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The aforementioned computer-readable storage medium carries one or more programs, which, when executed, implement the oral data processing method for cultural heritage according to the embodiments of this application.

[0186] In embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] For example, in embodiments of this application, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0188] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the oral data processing method for cultural heritage provided in the embodiments of this application.

[0189] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this application embodiment. In the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0190] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0191] In embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0193] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A method for processing oral history data on cultural heritage, characterized in that, The method includes: Obtain object attribute information and oral data from multiple target objects regarding a target cultural heritage item. The oral data describes the change process of the target cultural heritage item, and the object attribute information characterizes the degree of association between the target object and the target cultural heritage. The spoken data is semantically structured based on the object attribute information to obtain structured knowledge text; Data statistics were performed on the meta-information related to text features in the oral data to obtain statistical results; The oral data was subjected to topic modeling analysis to obtain topic evolution analysis results; The oral data is used to extract entities and entity relationships to construct a knowledge graph based on the extraction results. The entities represent narrative elements related to the target cultural heritage, and the entity relationships represent the semantic associations between the narrative elements. Based on a preset report template, a large language model is used to analyze the structured knowledge text, the statistical results, the topic evolution analysis results, and the semantic features determined based on the knowledge graph to obtain a visual report. The visualization report is stored in a database, and a search enhancement generation algorithm is used to retrieve the report related to the target prompt based on the target prompt.

2. The method according to claim 1, characterized in that, The process involves using a retrieval enhancement generation algorithm based on target prompts to search the database and obtain a question-and-answer report related to the target prompts, including: The target prompt words are classified to obtain prompt categories; The target prompt words are processed according to the retrieval enhancement generation algorithm corresponding to the prompt category to obtain the target retrieval strategy; Based on the target retrieval strategy, a multi-dimensional retrieval is performed in the database to obtain initial question-and-answer results; The initial question-and-answer results are then subjected to factual verification to obtain the verification results. Based on the verification results and the initial question-and-answer results, a question-and-answer report related to the target prompt word is obtained.

3. The method according to claim 2, characterized in that, The step of processing the target prompt words according to the retrieval enhancement generation algorithm corresponding to the prompt category to obtain the target retrieval strategy includes: If the prompt category is a global question category, a word analysis algorithm is invoked to process the target prompt words to obtain a first sub-strategy. The first sub-strategy includes the prompt questions retrieved from the statistical results. A knowledge graph mining algorithm is invoked to process the target prompt words to obtain a second sub-strategy, which includes prompt questions retrieved from the knowledge graph. The topic modeling and analysis algorithm is invoked to process the target prompt words to obtain a third sub-strategy, which includes prompt questions for retrieval based on the topic evolution analysis results; The filtering algorithm is invoked to process the target prompt words to obtain a fourth sub-strategy, which includes prompt questions related to object attribute information; The target retrieval strategy is obtained based on the first sub-strategy, the second sub-strategy, the third sub-strategy, and the fourth sub-strategy.

4. The method according to claim 2, characterized in that, The initial question-and-answer results are obtained by performing a multi-dimensional search in the database based on the target retrieval strategy, including: Based on the target retrieval strategy, keyword retrieval is performed in the database to obtain keyword retrieval results; Based on the target retrieval strategy, a semantic retrieval is performed in the database to obtain semantic retrieval results; Based on the target retrieval strategy, a knowledge graph structure retrieval is performed in the database to obtain knowledge graph retrieval results; Based on the keyword search results, the semantic search results, and the knowledge graph search results, the initial question-and-answer results are obtained.

5. The method according to claim 1, characterized in that, The step of semantically structuring the spoken data based on the object attribute information to obtain structured knowledge text includes: The object attribute information is processed in a structured manner to obtain structural meta-information, which includes collection timestamp, collection location information, object association category, and object association weight. The spoken data is processed by speech conversion to obtain text data; The semantic features of the text data are extracted using an embedding model to obtain the text semantic features; Based on multiple research dimensions as prompt word prefixes, the oral data is analyzed using a large language model to obtain content summaries corresponding to each of the multiple research dimensions. The structured knowledge text is obtained by encapsulating the structural meta-information, the text semantic features, and the content summary.

6. The method according to claim 5, characterized in that, The statistical analysis of meta-information related to text features in the spoken data to obtain statistical results includes: The spoken data is subjected to word feature statistics to obtain word statistical features; Sentence feature statistics are performed on the spoken data to obtain sentence statistical features; Based on a preset sliding window size, the words in the spoken data are statistically analyzed using a sliding window to obtain a word co-occurrence matrix. The word co-occurrence matrix represents the semantic association network topology between words in the text. Emotion recognition is performed on the spoken data to obtain emotion classification results; Statistical analysis is performed on the structural meta-information to obtain the cross-statistical results of the meta-information; The statistical results are obtained by encapsulating the word statistical features, the sentence statistical features, the word co-occurrence matrix, the sentiment classification results, and the meta-information cross-statistics results.

7. The method according to claim 5, characterized in that, The topic modeling analysis of the oral data to obtain topic evolution analysis results includes: The oral data was analyzed using a clustering algorithm to obtain multiple clusters; Keyword extraction is performed on each of the multiple clusters to obtain the target words corresponding to each cluster. Based on the prompt word template, the target words and representative texts corresponding to each of the multiple clusters are processed using a large language model to obtain the topic categories corresponding to each of the multiple clusters. The representative texts are determined based on the spoken data. Based on the collection timestamps and timestamp weights associated with the topic category, a time-series evolution analysis is performed on the topic category to obtain the topic evolution analysis results.

8. The method according to claim 5, characterized in that, The step of extracting entities and entity relationships from the spoken data to construct a knowledge graph based on the extraction results includes: Entities and entity relationships are extracted from the spoken data to obtain multiple triples; An initial knowledge graph is constructed based on multiple triples; Based on the object association weights corresponding to the entities, multiple triples are weighted by confidence to obtain the extraction result; The initial knowledge graph is updated based on the extraction results to obtain the knowledge graph.

9. The method according to claim 1, characterized in that, The method further includes: A social relationship network is constructed based on the personal entities in the knowledge graph as nodes and the co-occurrence frequency or interaction description strength of the personal entities as edge weights. Based on multiple timestamps extracted from the knowledge graph, a temporal evolution analysis is performed on key event entities to construct a temporal network; The structured knowledge text, statistical results, topic evolution analysis results, semantic features determined based on the knowledge graph, social relationship network, and temporal network are analyzed using a large language model to obtain a visualization report.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

Images