A method and system for intelligent interaction of digital tourism people adapted to the cultural context of ethnic minorities

Abstract

The application relates to a digital tourism person intelligent interaction method and system suitable for a minority cultural context. The method comprises the following steps: original byte streams are obtained by cutting multi-modal original data, cultural object identification is generated by combining collected information hash, and the original byte streams are encapsulated as atomic data units; an initial knowledge graph is constructed by performing entity recognition and relationship extraction, an embedding space is obtained by graph representation learning, multi-modal semantic features are extracted to generate candidate triplets; the structural consistency score of the candidate triplets is calculated and compared with a threshold value, and an entity cultural knowledge graph is constructed; a graph query and a vector retrieval request are generated by rewriting a tourist question, an enhanced context is generated by fusing and rearranging the results, and then a domain enhanced answer text is obtained; and a synthesized output answer voice and a phoneme sequence are output. By adopting the method, multi-modal data can be traceably encapsulated, knowledge graph consistency can be verified, and domain enhanced question answering can be realized, so that the accuracy of the answer of the digital person and the cultural adaptability are improved.

Description

Technical Field

[0001] This invention belongs to the field of cultural tourism interaction technology, and in particular relates to a method and system for intelligent interaction of digital human in cultural tourism that is adapted to the cultural context of ethnic minorities. Background Technology

[0002] With the development of artificial intelligence, digital humans, and cultural tourism interaction technologies, virtual digital human guides have emerged for cultural tourism scenarios. These systems, utilizing natural language processing, speech synthesis, and knowledge graph technologies, can provide tourists with human-like cultural information inquiry and explanation services, and are gradually being applied in the dissemination of ethnic minority cultures.

[0003] Traditional techniques typically involve manually organizing and textualizing ethnic minority cultural data before constructing keyword indexes or simple knowledge graphs to support question-and-answer generation. For multimodal content such as music and dance, only metadata is extracted for tagging and display, knowledge extraction relies mainly on general entity recognition and manual proofreading, and question-and-answer generation uses rule-based templates or general retrieval enhancement methods.

[0004] However, current approaches suffer from three main technical problems: First, in the data acquisition stage, multi-source, heterogeneous cultural data lacks unified traceability and encapsulation standards, leading to discontinuous data flow and difficulty in tracing the source. Second, in the knowledge graph construction stage, multimodal knowledge extraction lacks an effective quality verification mechanism, the consistency of candidate knowledge units cannot be quantitatively evaluated, and the reliability of the graph is difficult to guarantee. Third, in the question-answering model construction stage, there is a lack of continuous, process-oriented collaboration between query retrieval and answer generation in the context of minority cultures, and the integration of knowledge retrieval results and generation models is insufficient, making it difficult to control the domain adaptability and quality of the answers. Summary of the Invention

[0005] Based on this, it is necessary to provide a method and system for intelligent interaction of digital human in cultural tourism that is adapted to the cultural context of ethnic minorities, which can realize traceable encapsulation of multimodal cultural data, consistency verification of knowledge graphs, and domain knowledge-enhanced question answering.

[0006] Firstly, this application provides a method for intelligent interaction of digital human in cultural tourism that is adapted to the cultural context of ethnic minorities, including:

[0007] S1. Obtain multimodal raw data for minority cultures, segment the multimodal raw data according to speech segments or image frames to obtain raw byte streams, obtain the acquisition device information and acquisition timestamp, and convert the acquisition device information and acquisition timestamp into cultural object identifiers through hash operation, and combine the raw byte stream and cultural object identifiers to obtain atomic data units;

[0008] S2. Entity recognition and relation extraction are performed on atomic data units to construct an initial knowledge graph. Graph embedding space is obtained through graph representation learning on the initial knowledge graph. Multimodal semantic features are obtained by extracting text, audio and video features from atomic data units to obtain candidate triples.

[0009] S3. Calculate the structural consistency score of the candidate triples in the graph embedding space, compare the structural consistency score with the consistency threshold, take the candidate triples whose structural consistency scores meet the consistency threshold as graph nodes, and take the cultural object identifiers corresponding to the graph nodes as relation edges to obtain a three-dimensional cultural knowledge graph.

[0010] S4. Based on the three-dimensional cultural knowledge graph, train the query rewriting model to obtain the adaptive query rewriting model, obtain tourist questions, input the tourist questions into the adaptive query rewriting model to obtain graph query statements and vector retrieval requests, and perform graph structure query and embedded vector retrieval on the three-dimensional cultural knowledge graph based on the graph query statements and vector retrieval requests to obtain graph query results and vector retrieval results. Rearrange and fuse the graph query results and vector retrieval results to obtain enhanced context, and obtain domain-enhanced answer text based on the enhanced context.

[0011] S5. Perform speech synthesis on the domain-enhanced response text to obtain the response speech and phoneme sequence, and use the response speech and phoneme sequence as the visitor query results.

[0012] In one embodiment, entity recognition and relation extraction are performed on atomic data units to construct an initial knowledge graph. A graph embedding space is obtained through graph representation learning on the initial knowledge graph. Multimodal semantic features are then extracted from the atomic data units using text, audio, and video features, resulting in candidate triples, including:

[0013] S11. Context-encode the atomic data units to obtain a context-aware text embedding vector sequence;

[0014] S12. Perform named entity recognition on the text embedding vector sequence to identify culturally specific type entities in the text and obtain a set of entities with cultural semantic type annotations.

[0015] S13. Using each entity in the entity set as the initial node and the entity relationships extracted from the text embedding vector sequence as the initial edges, construct the initial knowledge graph.

[0016] S14. Perform graph representation learning training on the initial knowledge graph, learn to recognize the neighborhood structure information of the initial nodes in the initial knowledge graph, extract features from the neighborhood structure information to obtain embedding vectors, and summarize the embedding vectors of all the initial nodes to form the graph embedding space.

[0017] S15. Perform multi-part structure separation on the atomic data unit, extract the fundamental frequency corresponding to the sustained bass and the melody of the high-pitched part and integrate them to obtain the melody embedding vector. Extract the spatiotemporal features of the skeletal point motion trajectory of the ethnic dance action video in the atomic data unit to obtain the action primitive embedding vector.

[0018] S16. Semantically align the melody embedding vector and action primitive embedding vector with the text embedding vector sequence to obtain multimodal semantic features. Integrate the entity set and multimodal semantic features to obtain candidate triples.

[0019] In one embodiment, the domain-enhanced response text is obtained based on the enhanced context, including:

[0020] S21. Perform word-by-word autoregressive decoding on the enhanced context and visitor questions to obtain the currently generated text sequence;

[0021] S22. Calculate the semantic implication score between the text sequence and the enhanced context;

[0022] S23. Input the semantic entailment score into the probability adjustment function to calculate the adjusted vocabulary probability distribution; wherein, the expression for the vocabulary probability distribution function can be:

[0023]

[0024] in, To adjust the probability distribution of the vocabulary, The probability distribution of the original vocabulary. The target word for the current decoding step. All context words preceding the current word, To control the adjustable hyperparameters of cultural constraint strength, The semantic entailment score. For time steps, The words in the vocabulary list;

[0025] S24. Select the word with the highest probability in the adjusted vocabulary probability distribution as the current word to be generated. Add the current word to be generated to the generated text sequence to obtain the updated generated text sequence. Use the updated generated text sequence as the generated text sequence for the next decoding step.

[0026] S25. Repeat S22 to S24 until decoding ends, and use the final generated text sequence at the end of decoding as the domain-enhanced response text.

[0027] In one embodiment of the present invention, after performing speech synthesis on the domain-enhanced response text to obtain the response speech and phoneme sequence, and using the response speech and phoneme sequence as the visitor query result, the method further includes:

[0028] S31. Obtain the interaction log of tourists to the tourist query results, and extract feedback signals from the interaction log;

[0029] S32. Perform state analysis on the feedback signal, extract multi-dimensional interaction state information, and perform feature mapping on the interaction state information to obtain the interaction state vector.

[0030] S33. Perform probabilistic reasoning on the interaction state vector, calculate the posterior probability distribution of each preset root cause node, and take the root cause node whose posterior probability in the posterior probability distribution exceeds the attribution threshold as the root cause type that causes the interaction problem.

[0031] S34. When the root cause type is knowledge deficiency, a knowledge supplementation instruction is generated and fed back to S2, so that S2 supplements the three-dimensional cultural knowledge graph according to the knowledge supplementation instruction to obtain an updated three-dimensional cultural knowledge graph. When the root cause type is incorrect answer, an answer correction instruction is generated and fed back to S4, so that S4 adjusts according to the answer correction instruction to obtain an updated domain-enhanced answer text generation process. The updated three-dimensional cultural knowledge graph and the updated domain-enhanced answer text generation process are used to process subsequent visitor questions to obtain optimized visitor query results.

[0032] In one embodiment of the present invention, the structural consistency score of candidate triples in the graph embedding space is calculated, and the structural consistency score is compared with a consistency threshold. Candidate triples whose structural consistency scores satisfy the consistency threshold are taken as graph nodes, and the cultural object identifiers corresponding to the graph nodes are taken as relation edges, thus obtaining a three-dimensional cultural knowledge graph, including:

[0033] S41. Obtain the head entity embedding vector, relation embedding vector, and tail entity embedding vector of the candidate triple from the graph embedding space;

[0034] S42. Calculate the sum of the head entity embedding vector and the relation embedding vector to obtain the first vector. Calculate the Euclidean distance between the first vector and the tail entity embedding vector. Use the Euclidean distance as the structural consistency score of the candidate triple in the graph embedding space.

[0035] S43. Based on the mean and standard deviation of the structural consistency scores of existing triplets in the current graph, obtain the consistency threshold. Compare the structural consistency score with the consistency threshold. When the structural consistency score is less than or equal to the consistency threshold, the candidate triplet is determined to pass the verification and the passed triplet is obtained.

[0036] S44. Using the head and tail entities of the passed triples as graph nodes, and the relation types of the passed triples and the cultural object identifiers corresponding to the graph nodes as relation edges, a three-dimensional cultural knowledge graph is obtained.

[0037] Secondly, this application also provides a digital human intelligent interaction system for cultural tourism adapted to the cultural context of ethnic minorities, used to implement the method described in the first aspect, including:

[0038] The data encapsulation module is used to acquire multimodal raw data for ethnic minority cultures. It segments the multimodal raw data according to speech segments or image frames to obtain raw byte streams, acquires the acquisition device information and acquisition timestamp, and converts the acquisition device information and acquisition timestamp into cultural object identifiers through hash operation. It combines the raw byte stream and cultural object identifiers to obtain atomic data units.

[0039] The graph construction module is used to perform entity recognition and relation extraction on atomic data units to construct an initial knowledge graph. On the initial knowledge graph, the graph embedding space is obtained through graph representation learning. Multimodal semantic features are obtained by extracting text, audio and video features from atomic data units, and candidate triples are obtained.

[0040] The knowledge verification module is used to calculate the structural consistency score of candidate triples in the graph embedding space, compare the structural consistency score with the consistency threshold, take the candidate triples whose structural consistency scores meet the consistency threshold as graph nodes, and take the cultural object identifiers corresponding to the graph nodes as relation edges to obtain a three-dimensional cultural knowledge graph.

[0041] The answer generation module is used to train a query rewriting model based on a three-dimensional cultural knowledge graph to obtain an adaptive query rewriting model, acquire tourist questions, input tourist questions into the adaptive query rewriting model to obtain graph query statements and vector retrieval requests, and perform graph structure queries and embedded vector retrieval on the three-dimensional cultural knowledge graph based on the graph query statements and vector retrieval requests to obtain graph query results and vector retrieval results. The graph query results and vector retrieval results are rearranged and fused to obtain enhanced context, and domain-enhanced answer text is obtained based on the enhanced context.

[0042] The results output module is used to perform speech synthesis on the domain-enhanced response text to obtain the response speech and phoneme sequence, and to use the response speech and phoneme sequence as the visitor query results.

[0043] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods in the first aspect of this application.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods in the first aspect of this application.

[0045] The aforementioned intelligent interactive method and system for digital human culture and tourism adapted to the cultural context of ethnic minorities generates cultural object identifiers by hashing multimodal raw data with data collection device information and collection timestamps, and constructs atomic data units based on these identifiers. This achieves traceable encapsulation of multi-source heterogeneous cultural data, solving the problems of difficult traceability of data sources and discontinuous processes in the data collection stage. By calculating the structural consistency score of candidate triples in the graph embedding space and quantitatively comparing it with the dynamic consistency threshold obtained based on the current graph statistical characteristics, only those that pass the verification can be included in the three-dimensional cultural knowledge graph. This enables the quality evaluation of the knowledge graph construction process and ensures the credibility of the graph. By first generating graph query statements and vector retrieval requests through a query rewriting model for tourist questions, and then rearranging and fusing the graph query results and vector retrieval results to obtain enhanced context, and generating domain-enhanced answer text based on the enhanced context, a process-oriented collaboration between multi-path retrieval and answer generation is achieved. This compensates for the shortcomings of domain adaptability and difficulty in quality control, thereby improving the accuracy and cultural adaptability of digital human answers in the context of ethnic minorities. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of a digital human intelligent interaction method for cultural tourism adapted to the cultural context of ethnic minorities, as described in one embodiment of the present invention.

[0048] Figure 2 This is a flowchart illustrating the extraction of candidate triples in one embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the structure of a cultural tourism digital human intelligent interaction system adapted to the cultural context of ethnic minorities, according to one embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In one embodiment, such as Figure 1As shown, a digital human intelligent interaction method for cultural tourism adapted to the cultural context of ethnic minorities is provided. This embodiment illustrates the application of this method to an intelligent interactive terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0052] S1. Obtain multimodal raw data for ethnic minority cultures, segment the multimodal raw data according to speech segments or image frames to obtain raw byte streams, obtain the acquisition device information and acquisition timestamp, and convert the acquisition device information and acquisition timestamp into cultural object identifiers through hash operation, and combine the raw byte stream and cultural object identifiers to obtain atomic data units.

[0053] Specifically, multimodal raw data can refer to raw cultural data oriented towards ethnic minority cultural scenarios, including various types such as text, audio, video, and images; raw byte streams can be the multimodal raw data that has been segmented and converted into standard binary data format for easy subsequent storage and processing; acquisition device information can refer to the attribute information of the device used to acquire multimodal raw data, and acquisition timestamps can refer to the time record of data acquisition; cultural object identifiers can be unique identifiers converted from acquisition device information and acquisition timestamps through hash operations, used to distinguish different cultural data units; atomic data units can be basic data units formed by standardizing and encapsulating raw byte streams and cultural object identifiers.

[0054] For example, the intelligent interactive terminal can acquire multimodal raw data related to ethnic minority cultures, segment the multimodal raw data according to voice segments or image frames to obtain raw byte streams; the intelligent interactive terminal can simultaneously acquire acquisition device information and acquisition timestamps, and convert the acquisition device information and acquisition timestamps into cultural object identifiers through hash operations, which can ensure the uniqueness and immutability of cultural object identifiers; the intelligent interactive terminal can combine the raw byte streams and cultural object identifiers, and obtain atomic data units after standardized encapsulation; the intelligent interactive terminal can store the atomic data units in a distributed file system and manage them by partition according to cultural object identifiers.

[0055] S2. Entity recognition and relation extraction are performed on atomic data units to construct an initial knowledge graph. The graph embedding space is obtained through graph representation learning on the initial knowledge graph. Multimodal semantic features are obtained by extracting text, audio and video features from atomic data units, and candidate triples are obtained.

[0056] Specifically, entity recognition refers to identifying entities related to ethnic minority culture from atomic data units. These entities cover types such as ethnic costumes, traditional festivals, intangible cultural heritage skills, ethnic musical instruments, and folk customs. Relation extraction refers to extracting the relationships between identified entities from atomic data units, clarifying the semantic connections between entities. The initial knowledge graph can be a structured knowledge network built based on the identified entities and extracted relationships, used to initially organize knowledge of ethnic minority culture. The graph embedding space can be a space formed by mapping entities and relationships in the initial knowledge graph to a low-dimensional vector space through graph representation learning algorithms, facilitating subsequent similarity calculations and structural analysis. Multimodal semantic features can be semantic representations obtained after extracting features from different types of data such as text, audio, and video in atomic data units, reflecting the semantic information of the data. Candidate triples are knowledge units formed by combining relationships between entities, multimodal semantic features of entities, and graph embedding vectors. Candidate triples can be candidate objects for constructing the knowledge graph.

[0057] For example, the intelligent interactive terminal can preprocess atomic data units, employing corresponding preprocessing methods for different types of atomic data units to remove redundant information and reduce noise interference. The intelligent interactive terminal can then perform entity recognition and relation extraction on the preprocessed atomic data units, constructing an initial knowledge graph based on the identified entities and extracted relations, and using a graph representation learning algorithm to obtain the graph embedding space on the initial knowledge graph. The intelligent interactive terminal can also use the TransE algorithm to map the entities and relations in the knowledge graph to a low-dimensional vector space, obtaining the vector relationship between the embedding vectors of tail entities, head entities, and relations. The expression for the vector relationship can be:

[0058] in, The embedding vector of the head entity. For the embedding vector of the relation, The embedding vector of the tail entity is used; the intelligent interactive terminal can extract text, audio and video features from atomic data units to obtain multimodal semantic features. The multimodal semantic features of the head entity and tail entity are fused with the embedding vectors of the head entity and tail entity in the graph embedding space, respectively. Combined with the relationship between entities, candidate triples are obtained.

[0059] S3. Calculate the structural consistency score of the candidate triples in the graph embedding space, compare the structural consistency score with the consistency threshold, take the candidate triples whose structural consistency scores meet the consistency threshold as graph nodes, and take the cultural object identifiers corresponding to the graph nodes as relation edges to obtain a three-dimensional cultural knowledge graph.

[0060] Specifically, the structural consistency score can be used as an indicator to quantify the structural rationality of candidate triples in the graph embedding space. It is calculated by combining the vector similarity and the similarity of multimodal semantic features in the graph embedding space, and can reflect the structural integrity and semantic consistency of candidate triples. The consistency threshold can be used as a criterion for screening high-quality candidate triples. It is calibrated through a large amount of experimental data to ensure that the screened triples can meet the credibility requirements of the knowledge graph. The three-dimensional cultural knowledge graph can be a knowledge graph constructed by using candidate triples that meet the threshold requirements after structural consistency verification as nodes and the cultural object identifiers corresponding to the nodes as relation edges. It has the characteristics of being structured and traceable, and can comprehensively and accurately organize the knowledge of ethnic minority cultures.

[0061] For example, the intelligent interactive terminal can calculate the structural consistency score of candidate triples in the graph embedding space; wherein, the expression for calculating the structural consistency score can be:

[0062]

[0063] In the formula, The structural consistency score. For structural similarity weights, Structural similarity of candidate triples in the graph embedding space. Let be the multimodal semantic feature similarity between the head and tail entities in the candidate triplet; where the structural similarity of the candidate triplet in the graph embedding space can be calculated as follows:

[0064]

[0065] in, The embedding vector of the head entity. For the embedding vector of the relation, The embedding vector of the tail entity. The L2 norm is used; the expression for calculating the multimodal semantic feature similarity between the head and tail entities in a candidate triplet can be:

[0066]

[0067] in, The multimodal fusion feature vector of the head entity is the feature vector obtained by fusing the multimodal semantic features of the head entity with the head entity embedding vector. The multimodal fusion feature vector of the tail entity is the feature vector obtained by fusing the multimodal semantic features of the tail entity with the tail entity embedding vector. The intelligent interactive terminal can normalize the calculated structural consistency score to obtain the normalized structural consistency score. The intelligent interactive terminal can compare the structural consistency score with the consistency threshold, take the candidate triples that meet the consistency threshold as graph nodes, and take the cultural object identifiers corresponding to the graph nodes as relation edges to construct a three-dimensional cultural knowledge graph, and at the same time perform quality verification on the three-dimensional cultural knowledge graph.

[0068] S4. Based on the three-dimensional cultural knowledge graph, train the query rewriting model to obtain the adaptive query rewriting model, obtain tourist questions, input the tourist questions into the adaptive query rewriting model to obtain graph query statements and vector retrieval requests, and perform graph structure query and embedded vector retrieval on the three-dimensional cultural knowledge graph based on the graph query statements and vector retrieval requests to obtain graph query results and vector retrieval results. Rearrange and fuse the graph query results and vector retrieval results to obtain enhanced context, and obtain domain-enhanced answer text based on the enhanced context.

[0069] Specifically, the query rewriting model can be a model trained on a three-dimensional cultural knowledge graph. Its function is to transform tourists' natural language questions into graph query statements and vector retrieval requests that can be used to query the three-dimensional cultural knowledge graph, addressing issues such as non-standard question expression and semantic ambiguity. The graph query statement is a structured query statement adapted to the graph database, used for graph structure traversal and node querying of the three-dimensional cultural knowledge graph. The vector retrieval request can be a request for multimodal feature index retrieval, achieving rapid retrieval of relevant nodes based on semantic similarity. The enhanced context is structured information obtained by rearranging and fusing the graph query results and vector retrieval results. The domain-enhanced answer text can be a standardized answer text generated based on the enhanced context and conforming to the cultural context of ethnic minorities, ensuring the accuracy and domain adaptability of the answer.

[0070] For example, the intelligent interactive terminal can construct a query corpus of ethnic minority cultural tourism based on a three-dimensional cultural knowledge graph, train it using a basic model, and obtain a query rewriting model. The intelligent interactive terminal can acquire tourist questions, standardize them, remove redundant information, and obtain standardized text questions. The intelligent interactive terminal can input the standardized text questions into the query rewriting model to obtain graph query statements and vector retrieval requests. Based on the graph query statements and vector retrieval requests, the intelligent interactive terminal can perform graph structure queries and embedded vector retrievals on the three-dimensional cultural knowledge graph, obtaining graph query results and vector retrieval results respectively. The intelligent interactive terminal can rearrange and merge the graph query results and vector retrieval results, calculate the comprehensive score of each node using a weighted sorting method, assign reasonable weights to the graph query results and vector retrieval results, sort them from high to low according to the comprehensive score, select the information of the top few nodes as enhanced context, and obtain preliminary answer text using a generation model fine-tuned from the ethnic minority cultural corpus based on the enhanced context. After performing grammatical error correction and semantic verification on the preliminary answer text, a domain-enhanced answer text is obtained.

[0071] S5. Perform speech synthesis on the domain-enhanced response text to obtain the response speech and phoneme sequence, and use the response speech and phoneme sequence as the visitor query results.

[0072] Specifically, speech synthesis can be the process of converting domain-enhanced response text into playable speech. The generated speech needs to be natural and fluent, suitable for cultural and tourism interaction scenarios, and accurately adapted to the pronunciation of minority vocabulary. The response speech can be playable audio obtained after speech synthesis, used to provide tourists with interactive feedback in the form of speech. The phoneme sequence can be the sequence of the smallest unit of speech pronunciation, marked with the International Phonetic Alphabet. The phoneme sequence can include the pronunciation and tone of each syllable, used to provide tourists with pronunciation references. The tourist query results can be multimodal interactive results obtained by standardizing and encapsulating the response speech and phoneme sequence, which can meet the multi-dimensional interactive needs of tourists.

[0073] For example, the intelligent interactive terminal can perform text preprocessing on the domain-enhanced response text, completing word segmentation, punctuation processing, and pronunciation annotation to prepare for speech synthesis. The intelligent interactive terminal can use a speech synthesis model finely tuned with minority language speech corpora to synthesize the preprocessed text, obtaining the response speech and phoneme sequence. During speech synthesis, the intelligent interactive terminal can convert the text into a phoneme sequence using the speech synthesis model, then predict the Mel spectrum of the speech, and combine it with a vocoder to convert the Mel spectrum into a speech waveform to obtain the response speech. The intelligent interactive terminal can standardize and encapsulate the response speech and phoneme sequence into a specified format to ensure rapid data transmission and playback. The intelligent interactive terminal can use the encapsulated response speech and phoneme sequence as the tourist query result, feeding it back to the tourist through the cultural tourism digital human interactive interface. Simultaneously, it can perform quality verification on the synthesized response speech and phoneme sequence to ensure it meets preset quality standards; if the standards are not met, speech synthesis is re-performed.

[0074] This application provides a method for intelligent interaction of digital human in cultural tourism that adapts to the cultural context of ethnic minorities. It generates cultural object identifiers by hashing multimodal raw data with data acquisition device information and acquisition timestamps, and constructs atomic data units based on these identifiers. This achieves traceable encapsulation of multi-source heterogeneous cultural data, solving the problems of difficult traceability and discontinuous processes in the data acquisition stage. By calculating the structural consistency score of candidate triples in the graph embedding space and quantitatively comparing it with a dynamic consistency threshold obtained based on the current graph statistical characteristics, only those that pass the verification can be included in the three-dimensional cultural knowledge graph. This ensures the quality evaluation of the knowledge graph construction process and guarantees the graph's credibility. Furthermore, by first generating graph query statements and vector retrieval requests through a query rewriting model for tourist questions, and then rearranging and fusing the graph query results and vector retrieval results to obtain enhanced context, and generating domain-enhanced answer text based on this enhanced context, a streamlined collaborative process for multi-path retrieval and answer generation is achieved. This compensates for the shortcomings of domain adaptability and difficulty in quality control, thereby improving the overall accuracy and cultural adaptability of the digital human's responses in ethnic minority cultural contexts.

[0075] In one embodiment of the present invention, such as Figure 2 As shown, entity recognition and relation extraction are performed on atomic data units to construct an initial knowledge graph. A graph embedding space is obtained through graph representation learning on the initial knowledge graph. Multimodal semantic features are then extracted from the atomic data units using text, audio, and video data, resulting in candidate triples, including:

[0076] S11. Context-encode the atomic data units to obtain a context-aware text embedding vector sequence.

[0077] Specifically, context encoding can be a process of semantically encoding the text content in atomic data units. Context encoding can capture the correlation information between text contexts. The text embedding vector sequence can be a set of vectors obtained after encoding. Each vector corresponds to a character or word in the text and carries contextual semantic information to support subsequent entity recognition.

[0078] For example, the intelligent interactive terminal can acquire the text content in the preprocessed atomic data units, and use a semantic coding model to perform contextual coding on the text content. During the coding process, the semantic associations and contextual dependencies between words in the text are fully captured, resulting in a text embedding vector sequence; wherein, the expression for the text embedding vector sequence can be:

[0079]

[0080] in, Embed a sequence of vectors for context-aware text. The original text content within the atomic data unit. This is the embedding weight matrix of the semantic encoding model. This is the positional encoding weight matrix.

[0081] S12. Perform named entity recognition on the text embedding vector sequence to identify culturally specific type entities in the text and obtain a set of entities with cultural semantic type annotations.

[0082] Specifically, named entity recognition can be a process of entity mining from text embedding vector sequences, with a focus on entities related to the cultural domain; culturally unique entities can refer to entities closely related to ethnic minority cultures, covering categories such as ethnic costumes, traditional festivals, and intangible cultural heritage skills; the set of entities carrying cultural semantic type annotations can be the set of identified entities and their corresponding cultural semantic types, clarifying the cultural attributes of each entity.

[0083] For example, the intelligent interactive terminal can perform named entity recognition on the obtained context-aware text embedding vector sequence, adopt a recognition model adapted to the cultural domain, focus on identifying entity types in the text that are unique to ethnic minority cultures, label each identified entity with the corresponding cultural semantic type, remove entities that are unrelated to ethnic minority cultures, and obtain a set of entities labeled with cultural semantic types.

[0084] S13. Using each entity in the entity set as the initial node and the entity relationships extracted from the text embedding vector sequence as the initial edges, construct the initial knowledge graph.

[0085] Specifically, initial nodes can be the basic units for constructing an initial knowledge graph, directly served by each entity in the entity set; the initial knowledge graph can be a structured knowledge network built based on initial nodes and initial edges, used to initially organize cultural entities and the relationships between them.

[0086] For example, the intelligent interactive terminal can use each entity in the entity set carrying cultural semantic type annotation as the initial node of the initial knowledge graph, extract the relationship between each entity from the context-aware text embedding vector sequence, use these relationships as the initial edges connecting the corresponding initial nodes, and integrate the initial nodes and initial edges according to the construction rules of the structured knowledge network to construct the initial knowledge graph.

[0087] S14. Perform graph representation learning training on the initial knowledge graph. Learn to identify the neighborhood structure information of the initial nodes in the initial knowledge graph, and extract features from the neighborhood structure information to obtain embedding vectors. Summarize the embedding vectors of all the initial nodes to form the graph embedding space.

[0088] Specifically, graph representation learning training can be a process of vector mapping of the initial knowledge graph to mine the structural features of the initial nodes; neighborhood structure information can be the relevant information of the initial node's neighboring nodes and edges in the initial knowledge graph, which can reflect the structural relationships between nodes. The embedding vector can be a low-dimensional vector obtained after feature extraction of the initial node's neighborhood structure information, and the graph embedding space is the set of all initial node embedding vectors.

[0089] For example, the intelligent interactive terminal can train the constructed initial knowledge graph to learn graph representation. During the training process, it focuses on learning to identify the neighborhood structure information of each initial node, capturing the association features between the initial node and its neighboring nodes and edges, performing feature extraction and dimensionality reduction on the extracted neighborhood structure information, obtaining the embedding vector corresponding to each initial node, and summing up the embedding vectors of all initial nodes to form the graph embedding space.

[0090] S15. Perform multi-part structure separation on the atomic data unit, extract the fundamental frequency corresponding to the sustained bass and the melody of the high-pitched part and integrate them to obtain the melody embedding vector. Extract the spatiotemporal features of the skeletal motion trajectory of the ethnic dance action video in the atomic data unit to obtain the action primitive embedding vector.

[0091] Specifically, multi-voice structure separation can be the separation and processing of audio content in atomic data units to distinguish between sustained bass and treble voices; skeletal point motion trajectory can be the human skeletal motion path extracted from ethnic dance movement videos; spatiotemporal feature extraction can be the capture of the temporal and spatial features of the trajectory; and motion primitive embedding vector is the vector obtained after feature extraction, representing the dance movement features.

[0092] For example, the intelligent interactive terminal can perform multi-part structure separation on the audio content in the atomic data unit, separating the sustained bass and treble parts, extracting the fundamental frequency and melody of the treble part corresponding to the sustained bass, and integrating the features of the fundamental frequency and treble part melody to obtain a melody embedding vector; wherein, the calculation expression of the melody embedding vector can be:

[0093]

[0094] in, Embed vectors for melodies. This is the feature vector corresponding to the sustained low fundamental frequency. This is the feature vector corresponding to the treble melody. These are the weighting coefficients of the eigenvector corresponding to the sustained low-frequency fundamental frequency.

[0095] S16. Semantically align the melody embedding vector and action primitive embedding vector with the text embedding vector sequence to obtain multimodal semantic features. Integrate the entity set and multimodal semantic features to obtain candidate triples.

[0096] Specifically, the intelligent interactive terminal can semantically align the obtained melody embedding vector and action primitive embedding vector with the context-aware text embedding vector sequence using a semantic alignment function, thereby obtaining initial multimodal semantic features; whereby the expression for the semantic alignment function can be:

[0097]

[0098] in, For initial multimodal semantic features, The modal vector to be aligned, The text is embedded as a vector sequence; the intelligent interactive terminal can fuse the initial multimodal semantic features and the text embedded vector sequence and adjust the dimensions to obtain multimodal semantic features that integrate audio, video and text semantics; the intelligent interactive terminal can associate the entity set carrying cultural semantic type annotations with the multimodal semantic features, and combine the extracted inter-entity relationships to combine the entities, relationships and corresponding multimodal semantic features to obtain candidate triples.

[0099] This invention provides a digital human intelligent interaction method for cultural tourism that adapts to the cultural context of ethnic minorities. By performing context encoding on atomic data units, identifying culturally specific entities, and constructing an initial knowledge graph, and combining graph representation learning to obtain a graph embedding space, the method simultaneously extracts multimodal features of audio melodies and dance movements and aligns them with text semantics. Finally, it generates candidate triples that are semantically rich, structurally sound, and fit the cultural context of ethnic minorities. This effectively improves the accuracy and relevance of multimodal knowledge extraction, strengthens the correlation between entities and multimodal features, and provides a high-quality foundation for the reliable construction of a three-dimensional cultural knowledge graph.

[0100] In one embodiment of the present invention, the domain-enhanced response text obtained based on enhanced context includes:

[0101] S21. Perform word-by-word autoregressive decoding on the enhanced context and visitor questions to obtain the currently generated text sequence.

[0102] Specifically, word-by-word autoregressive decoding can refer to a decoding method that generates text words one by one in sequence. Each step of the decoding process depends on the words generated in the previous step. The currently generated text sequence can be the set of words generated after a certain decoding step in the word-by-word autoregressive decoding process, which is used as the input basis for the next decoding step.

[0103] For example, the intelligent interactive terminal can obtain the enhanced context and standardized tourist questions. The intelligent interactive terminal can use the enhanced context and standardized tourist questions as input to the generative model finely tuned by the minority cultural corpus, and start the word-by-word autoregressive decoding process. In the decoding process, only one word is generated in each step, and the generated word depends on all the words generated in the previous step to ensure the semantic coherence of the text sequence. After the initial decoding step, the currently generated text sequence is obtained.

[0104] S22. Calculate the semantic implication score between the text sequence and the enhanced context.

[0105] Specifically, the semantic implication score can be used as an indicator to quantify the degree of semantic association between the generated text sequence and the enhanced context. The higher the score, the higher the semantic fit between the generated text and the enhanced context, and the more it conforms to the cultural context of ethnic minorities and query needs.

[0106] For example, the intelligent interactive terminal can extract the semantic features of the currently generated text sequence and the enhanced context, and use a semantic matching algorithm to calculate the semantic implication relationship score between the two. The focus is on measuring whether the generated text accurately matches the minority cultural knowledge contained in the enhanced context, ensuring that the score can truly reflect the degree of semantic association between the two.

[0107] S23. Input the semantic implication relation score into the probability adjustment function to calculate the adjusted vocabulary probability distribution.

[0108] Specifically, the expression for calculating the vocabulary probability distribution function can be:

[0109]

[0110] in, To adjust the probability distribution of the vocabulary, The probability distribution of the original vocabulary. The target word for the current decoding step. All context words preceding the current word, To control the adjustable hyperparameters of cultural constraint strength, The score represents the semantic entailment relation. For time steps, These are words from the vocabulary list.

[0111] Specifically, the probability adjustment function can be a function used to correct the probability distribution of the original vocabulary of the generative model. Its core function is to combine the semantic implication relation score to strengthen the probability of words that conform to the semantic enhancement context and suppress the probability of irrelevant words. The vocabulary probability distribution can be the set of generation probabilities of each candidate word output by the generative model, which is used to determine the target word of the current decoding step.

[0112] S24. Select the word with the highest probability in the adjusted vocabulary probability distribution as the current word to be generated. Add the current word to be generated to the generated text sequence to obtain the updated generated text sequence. Use the updated generated text sequence as the generated text sequence for the next decoding step.

[0113] Specifically, the word to be generated can be the word with the highest generation probability selected from the adjusted vocabulary probability distribution, and is the word that needs to be added to the text sequence in the current decoding step; the updated generated text sequence can be a new sequence formed by adding the word to be generated to the original generated text sequence, and is used as the input for the next decoding step.

[0114] For example, the intelligent interactive terminal can sort the probability distribution of the adjusted vocabulary and select the word with the highest probability as the current word to be generated; the intelligent interactive terminal can add the current word to be generated to the generated text sequence to obtain the updated generated text sequence, and use the updated sequence as the generated text sequence for the next decoding step to provide input for subsequent decoding.

[0115] S25. Repeat S22 to S24 until decoding ends, and use the final generated text sequence at the end of decoding as the domain-enhanced response text.

[0116] Specifically, the final generated text sequence can be the complete text obtained after decoding, which is the domain-enhanced answer text that meets the requirements.

[0117] For example, the intelligent interactive terminal can repeatedly execute the operations from S22 to S24. In each round of decoding, the semantic implication relationship score between the currently generated text sequence and the enhanced context is calculated, the word probability distribution is adjusted, the highest probability word is selected and the text sequence is updated, until the decoding stops when the preset conditions are met. The final generated text sequence at the end of the decoding is used as the domain-enhanced answer text to ensure that the text is semantically coherent, fits the cultural context of ethnic minorities, and accurately responds to tourists' questions.

[0118] Specifically, the preset conditions can include two categories. One is that the length of the text sequence meets the standard, that is, the length of the currently generated text sequence reaches a preset reasonable threshold. The other is to generate a preset end marker. The end marker can be a special character or mark set in advance. When the generation model outputs the marker, it indicates that the answer text has been completely generated and there is no need to continue decoding.

[0119] This application provides a digital human intelligent interaction method for cultural tourism that adapts to the cultural context of ethnic minorities. By combining word-by-word autoregressive decoding with semantic implication relationship scores to adjust the probability distribution of the word list, iteratively optimizes the text sequence to ensure that the generated domain-enhanced answer text is semantically coherent, accurately fits the enhanced context of ethnic minority cultural knowledge, adapts to tourists' questioning needs, strengthens cultural constraints, improves the pertinence and standardization of the answers, and ensures that the answers conform to the cultural context of ethnic minorities.

[0120] In one embodiment of the present invention, after performing speech synthesis on the domain-enhanced response text to obtain the response speech and phoneme sequence, and using the response speech and phoneme sequence as the visitor query result, the method further includes:

[0121] S31. Obtain the interaction log of tourists to tourist query results, and extract feedback signals from the interaction log.

[0122] Specifically, the interaction log can be the operation record generated when tourists use the cultural and tourism digital human interactive function, covering various interactive behaviors such as playing, pausing, repeating, and providing feedback on the query results; the feedback signal can be the signal extracted from the interaction log that can reflect the tourist's satisfaction with the query results, including behavioral characteristics related to positive feedback, negative feedback, and neutral feedback.

[0123] For example, the intelligent interactive terminal can obtain the interaction logs of tourists' queries; the intelligent interactive terminal filters and extracts the interaction logs, removes invalid and redundant records, and focuses on extracting feedback signals that can reflect the tourist's feedback intentions, including but not limited to signals such as tourists repeating voice messages, closing the interactive interface, and clicking negative feedback buttons, to provide a basis for subsequent status analysis.

[0124] S32. Perform state analysis on the feedback signal, extract multi-dimensional interaction state information, and perform feature mapping on the interaction state information to obtain the interaction state vector.

[0125] Specifically, state analysis can be the process of classifying and parsing feedback signals to uncover the interaction issues behind them; multi-dimensional interaction state information can be information extracted from feedback signals that covers multiple dimensions such as interaction behavior, feedback tendency, and operation frequency; and interaction state vectors can be low-dimensional vectors obtained by feature mapping of interaction state information, used to quantify interaction states and support subsequent probabilistic reasoning.

[0126] For example, the intelligent interactive terminal can perform state analysis on the extracted feedback signals, extract multi-dimensional interactive state information according to dimensions such as interactive behavior type, feedback tendency, and operation frequency, and identify potential problems in the tourist's interaction process; the intelligent interactive terminal can use a feature mapping algorithm to map the multi-dimensional interactive state information into a fixed-dimensional interactive state vector, unify the vector format, and ensure that the vector can accurately represent the core features of the interactive state.

[0127] S33. Perform probabilistic reasoning on the interaction state vector, calculate the posterior probability distribution of each preset root cause node, and take the root cause node whose posterior probability in the posterior probability distribution exceeds the attribution threshold as the root cause type causing the interaction problem.

[0128] Specifically, probabilistic reasoning can be a process of calculating the probability of occurrence of each preset root cause node based on the interaction state vector, used to locate the core cause of the interaction problem; the root cause node can be a preset type of cause that may lead to the interaction problem, including knowledge gaps, incorrect answers, etc.; the posterior probability distribution is the probability distribution of each root cause node under the given interaction state vector, and the attribution threshold can be a probability standard used to determine the root cause type.

[0129] For example, the intelligent interactive terminal can perform probabilistic inference on the interaction state vector, and use a Bayesian inference algorithm to calculate the posterior probability distribution of each preset root cause node; wherein, the expression for calculating the posterior probability distribution can be:

[0130]

[0131] in, For the interaction state vector, For the first One root cause node, For in the given hour, The posterior probability, for When it occurs, it appears The conditional probability, for The prior probability, for The prior probability.

[0132] S34. When the root cause type is knowledge deficiency, a knowledge supplementation instruction is generated and fed back to S2, so that S2 supplements the three-dimensional cultural knowledge graph according to the knowledge supplementation instruction to obtain an updated three-dimensional cultural knowledge graph. When the root cause type is incorrect answer, an answer correction instruction is generated and fed back to S4, so that S4 adjusts according to the answer correction instruction to obtain an updated domain-enhanced answer text generation process. The updated three-dimensional cultural knowledge graph and the updated domain-enhanced answer text generation process are used to process subsequent visitor questions to obtain optimized visitor query results.

[0133] Specifically, the knowledge supplementation instruction can be an instruction generated when the root cause type is knowledge deficiency, used to instruct step S2 to supplement the missing knowledge in the three-dimensional cultural knowledge graph; the answer correction instruction can be an instruction generated when the root cause type is answer error, used to instruct step S4 to adjust the generation process of the domain-enhanced answer text.

[0134] For example, the intelligent interactive terminal can generate corresponding adjustment instructions based on the determined root cause type. When the root cause type is knowledge deficiency, the intelligent interactive terminal can generate a knowledge supplementation instruction and feed it back to step S2, so that step S2 can supplement the missing knowledge in the three-dimensional cultural knowledge graph according to the knowledge supplementation instruction, and obtain an updated three-dimensional cultural knowledge graph. When the root cause type is incorrect answer, the intelligent interactive terminal can generate an answer correction instruction and feed it back to step S4, so that step S4 can adjust the generation parameters and process of the domain-enhanced answer text according to the answer correction instruction, and obtain an updated domain-enhanced answer text generation process. When processing visitor questions, the intelligent interactive terminal can use the updated three-dimensional cultural knowledge graph and the updated domain-enhanced answer text generation process to process visitor questions and obtain optimized visitor query results, continuously improving the accuracy of interaction and visitor experience.

[0135] This application provides a digital human intelligent interaction method for cultural tourism that adapts to the cultural context of ethnic minorities. By extracting feedback signals from tourist interaction logs, analyzing the interaction status, and locating the root cause of interaction problems through probabilistic reasoning, corresponding adjustment instructions are generated for different root causes such as knowledge gaps and incorrect answers. These instructions are fed back to the corresponding steps to optimize the knowledge graph and answer generation process, thereby achieving dynamic iteration of the interaction effect and continuously improving the accuracy, adaptability, and tourist experience of the digital human interaction for cultural tourism.

[0136] In one embodiment of the present invention, the structural consistency score of candidate triples in the graph embedding space is calculated, and the structural consistency score is compared with a consistency threshold. Candidate triples whose structural consistency scores satisfy the consistency threshold are taken as graph nodes, and the cultural object identifiers corresponding to the graph nodes are taken as relation edges, thus obtaining a three-dimensional cultural knowledge graph, including:

[0137] S41. Obtain the head entity embedding vector, relation embedding vector, and tail entity embedding vector of the candidate triples from the graph embedding space.

[0138] Specifically, the head entity embedding vector, relation embedding vector, and tail entity embedding vector are all low-dimensional vector representations of candidate triples in the graph embedding space, corresponding to the head entity, relation, and tail entity in the candidate triples, respectively. They are the core foundation for calculating the structural consistency score and correspond to the embedding vectors of each initial node in the graph embedding space.

[0139] For example, the intelligent interactive terminal can obtain the head entity embedding vector, relation embedding vector, and tail entity embedding vector corresponding to the candidate triplet to be verified from the constructed graph embedding space, and perform matching retrieval through the entity identifier and relation identifier of the candidate triplet to ensure that the three obtained vectors correspond to the candidate triplet and avoid vector confusion.

[0140] S42. Calculate the sum of the head entity embedding vector and the relation embedding vector to obtain the first vector. Calculate the Euclidean distance between the first vector and the tail entity embedding vector. Use the Euclidean distance as the structural consistency score of the candidate triple in the graph embedding space.

[0141] Specifically, the first vector can be a fused vector obtained by adding the head entity embedding vector and the relation embedding vector, which is used to represent the semantic features after the head entity and relation are combined; the structural consistency score here specifically refers to the value calculated by Euclidean distance, which is used to quantify the structural rationality of the candidate triples.

[0142] For example, the intelligent interactive terminal can perform vector addition on the acquired head entity embedding vector and relation embedding vector, adding the corresponding dimensions of the head entity embedding vector and relation embedding vector respectively to obtain a first vector, thus completing the semantic fusion of head entities and relations; the intelligent interactive terminal can calculate the Euclidean distance between the first vector and the tail entity embedding vector; wherein, the formula for calculating the Euclidean distance is:

[0143] in, For Euclidean distance, For the first vector, the first... Each dimension value The tail entity embedding vector of the first Each dimension value Given the dimension of the vector, the intelligent interactive terminal can use the Euclidean distance as the structural consistency score of the current candidate triple in the graph embedding space, providing a quantitative basis for subsequent verification.

[0144] S43. Based on the mean and standard deviation of the structural consistency scores of existing triplets in the current graph, obtain the consistency threshold. Compare the structural consistency score with the consistency threshold. When the structural consistency score is less than or equal to the consistency threshold, the candidate triplet is determined to pass the verification and the passed triplet is obtained.

[0145] Specifically, the mean can be the average of the structural consistency scores of existing triplets in the current graph, and the standard deviation is an indicator that reflects the dispersion of the structural consistency scores of existing triplets. The combination of the mean and standard deviation can be used to dynamically determine the consistency threshold to fit the current graph. The passed triplet can be a candidate triplet whose structural consistency score meets the consistency threshold requirement and passes the verification. It is the core unit for constructing a three-dimensional cultural knowledge graph.

[0146] For example, the intelligent interactive terminal can statistically analyze the structural consistency scores of all existing triples in the current graph, calculate the mean and standard deviation of these scores, construct threshold calculation rules based on the mean and standard deviation, and obtain a consistency threshold that is suitable for the current graph, ensuring that the threshold can dynamically adapt to the overall quality of the graph. The intelligent interactive terminal can compare the structural consistency score of the current candidate triple with the consistency threshold. When the structural consistency score is less than or equal to the consistency threshold, the candidate triple is determined to have passed the verification and is obtained as an approved triple. If it is greater than the consistency threshold, it is determined to have failed the verification and is removed.

[0147] S44. Using the head and tail entities of the passed triples as graph nodes, and the relation types of the passed triples and the cultural object identifiers corresponding to the graph nodes as relation edges, a three-dimensional cultural knowledge graph is obtained.

[0148] Specifically, graph nodes can be the basic units of a three-dimensional cultural knowledge graph, directly serving as the head and tail entities of the checked triples, carrying the multimodal semantic information and cultural attributes of the entities. Relationship edges can be the links connecting graph nodes, jointly composed of the relationship type of the checked triples and the cultural object identifier corresponding to the graph node, clarifying the association between nodes and data traceability information.

[0149] For example, the intelligent interactive terminal can organize all the verified triples, and treat the head entity and tail entity in each verified triple as independent graph nodes of the three-dimensional cultural knowledge graph, retaining the multimodal semantic features and cultural semantic type annotations of the entities; the intelligent interactive terminal can integrate the relation type of the verified triple with the cultural object identifier of the corresponding graph node, as a relation edge connecting the corresponding head entity node and tail entity node, clarifying the association relationship and data traceability information between nodes, and associating and integrating all graph nodes and relation edges according to structured rules to obtain the three-dimensional cultural knowledge graph.

[0150] This application provides a method for intelligent interaction of cultural tourism digital humans that adapts to the cultural context of ethnic minorities. By accurately acquiring the embedding vectors of candidate triples, quantifying their structural consistency using Euclidean distance, and dynamically determining the threshold to screen qualified triples by combining the mean and standard deviation of the current graph triple scores, a three-dimensional cultural knowledge graph with reasonable structure, semantic coherence, and traceability is finally constructed, providing reliable knowledge support for the subsequent precise interaction of cultural tourism digital humans.

[0151] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0152] Based on the same inventive concept, this application also provides a system for implementing the above-mentioned intelligent interactive method for cultural tourism digital humans adapted to the cultural context of ethnic minorities. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the intelligent interactive system for cultural tourism digital humans adapted to the cultural context of ethnic minorities provided below can be found in the limitations of the intelligent interactive method for cultural tourism digital humans adapted to the cultural context of ethnic minorities described above, and will not be repeated here.

[0153] In one exemplary embodiment, such as Figure 3 As shown, a digital human intelligent interaction system 500 for cultural tourism, adapted to the cultural context of ethnic minorities, is provided to implement the methods in the above-described method embodiments, including:

[0154] The data encapsulation module 501 is used to acquire multimodal raw data for ethnic minority culture, segment the multimodal raw data according to speech segments or image frames to obtain raw byte streams, acquire acquisition device information and acquisition timestamps, and convert the acquisition device information and acquisition timestamps into cultural object identifiers through hash operation, and combine the raw byte streams and cultural object identifiers to obtain atomic data units;

[0155] The graph construction module 502 is used to perform entity recognition and relation extraction on atomic data units to construct an initial knowledge graph. On the initial knowledge graph, the graph embedding space is obtained through graph representation learning. Multimodal semantic features are obtained by extracting text, audio and video features from atomic data units to obtain candidate triples.

[0156] The knowledge verification module 503 is used to calculate the structural consistency score of candidate triples in the graph embedding space, compare the structural consistency score with the consistency threshold, take the candidate triples whose structural consistency scores meet the consistency threshold as graph nodes, and take the cultural object identifiers corresponding to the graph nodes as relation edges to obtain a three-dimensional cultural knowledge graph.

[0157] The answer generation module 504 is used to train a query rewriting model based on a three-dimensional cultural knowledge graph to obtain an adaptive query rewriting model, acquire tourist questions, input tourist questions into the adaptive query rewriting model to obtain graph query statements and vector retrieval requests, and perform graph structure query and embedded vector retrieval on the three-dimensional cultural knowledge graph based on the graph query statements and vector retrieval requests to obtain graph query results and vector retrieval results. The graph query results and vector retrieval results are rearranged and fused to obtain enhanced context, and domain-enhanced answer text is obtained based on the enhanced context.

[0158] The result output module 505 is used to perform speech synthesis on the domain-enhanced answer text to obtain the answer speech and phoneme sequence, and to use the answer speech and phoneme sequence as the visitor query result.

[0159] In one embodiment of the present invention, the map construction module can also be used for:

[0160] S11. Context-encode the atomic data units to obtain a context-aware text embedding vector sequence;

[0161] S12. Perform named entity recognition on the text embedding vector sequence to identify culturally specific type entities in the text and obtain a set of entities with cultural semantic type annotations.

[0162] S13. Using each entity in the entity set as the initial node and the entity relationships extracted from the text embedding vector sequence as the initial edges, construct the initial knowledge graph.

[0163] S14. Perform graph representation learning training on the initial knowledge graph, learn to recognize the neighborhood structure information of the initial nodes in the initial knowledge graph, extract features from the neighborhood structure information to obtain embedding vectors, and summarize the embedding vectors of all the initial nodes to form the graph embedding space.

[0164] S15. Perform multi-part structure separation on the atomic data unit, extract the fundamental frequency corresponding to the sustained bass and the melody of the high-pitched part and integrate them to obtain the melody embedding vector. Extract the spatiotemporal features of the skeletal point motion trajectory of the ethnic dance action video in the atomic data unit to obtain the action primitive embedding vector.

[0165] S16. Semantically align the melody embedding vector and action primitive embedding vector with the text embedding vector sequence to obtain multimodal semantic features. Integrate the entity set and multimodal semantic features to obtain candidate triples.

[0166] In one embodiment, the answer generation module 504 can also be used for:

[0167] S21. Perform word-by-word autoregressive decoding on the enhanced context and visitor questions to obtain the currently generated text sequence;

[0168] S22. Calculate the semantic implication score between the text sequence and the enhanced context;

[0169] S23. Input the semantic entailment relation scores into the probability adjustment function to calculate the adjusted vocabulary probability distribution; wherein, the expression for the vocabulary probability distribution function can be:

[0170]

[0171] in, To adjust the probability distribution of the vocabulary, The probability distribution of the original vocabulary. The target word for the current decoding step. All context words preceding the current word, To control the adjustable hyperparameters of cultural constraint strength, The score represents the semantic implication relation.

[0172] S24. Select the word with the highest probability in the adjusted vocabulary probability distribution as the current word to be generated. Add the current word to be generated to the generated text sequence to obtain the updated generated text sequence. Use the updated generated text sequence as the generated text sequence for the next decoding step.

[0173] S25. Repeat S22 to S24 until decoding ends, and use the final generated text sequence at the end of decoding as the domain-enhanced response text.

[0174] In one embodiment of the present invention, after performing speech synthesis on the domain-enhanced response text to obtain the response speech and phoneme sequence, and using the response speech and phoneme sequence as the visitor query result, the method further includes:

[0175] S31. Obtain the interaction log of tourists to the tourist query results, and extract feedback signals from the interaction log;

[0176] S32. Perform state analysis on the feedback signal, extract multi-dimensional interaction state information, and perform feature mapping on the interaction state information to obtain the interaction state vector.

[0177] S33. Perform probabilistic reasoning on the interaction state vector, calculate the posterior probability distribution of each preset root cause node, and take the root cause node whose posterior probability exceeds the attribution threshold as the root cause type that causes the interaction problem.

[0178] S34. When the root cause type is knowledge deficiency, a knowledge supplementation instruction is generated and fed back to S2, so that S2 supplements the three-dimensional cultural knowledge graph according to the knowledge supplementation instruction to obtain an updated three-dimensional cultural knowledge graph. When the root cause type is incorrect answer, an answer correction instruction is generated and fed back to S4, so that S4 adjusts according to the answer correction instruction to obtain an updated domain-enhanced answer text generation process. The updated three-dimensional cultural knowledge graph and the updated domain-enhanced answer text generation process are used to process subsequent visitor questions to obtain optimized visitor query results.

[0179] In one embodiment of the present invention, the knowledge verification module 503 can also be used for:

[0180] S41. Obtain the head entity embedding vector, relation embedding vector, and tail entity embedding vector of the candidate triple from the graph embedding space;

[0181] S42. Calculate the sum of the head entity embedding vector and the relation embedding vector to obtain the first vector. Calculate the Euclidean distance between the first vector and the tail entity embedding vector. Use the Euclidean distance as the structural consistency score of the candidate triple in the graph embedding space.

[0182] S43. Based on the mean and standard deviation of the structural consistency scores of existing triplets in the current graph, obtain the consistency threshold. Compare the structural consistency score with the consistency threshold. When the structural consistency score is less than or equal to the consistency threshold, the candidate triplet is determined to pass the verification and the passed triplet is obtained.

[0183] S44. Using the head and tail entities of the passed triples as graph nodes, and the relation types of the passed triples and the cultural object identifiers corresponding to the graph nodes as relation edges, a three-dimensional cultural knowledge graph is obtained.

[0184] In one embodiment, this application provides a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0186] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0187] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for intelligent interaction of digital human in cultural tourism that adapts to the cultural context of ethnic minorities, characterized in that, The method includes: S1. Obtain multimodal raw data for minority cultures, segment the multimodal raw data according to speech segments or image frames to obtain raw byte streams, obtain acquisition device information and acquisition timestamps, and convert the acquisition device information and acquisition timestamps into cultural object identifiers through hash operation, and combine the raw byte streams and cultural object identifiers to obtain atomic data units; S2. Entity recognition and relation extraction are performed on the atomic data units to construct an initial knowledge graph. Graph embedding space is obtained on the initial knowledge graph through graph representation learning. Multimodal semantic features are obtained by extracting text, audio, and video features from the atomic data units to obtain candidate triples. S3. Calculate the structural consistency score of the candidate triple in the graph embedding space, compare the structural consistency score with the consistency threshold, take the candidate triple corresponding to the structural consistency score satisfying the consistency threshold as a graph node, and take the cultural object identifier corresponding to the graph node as a relation edge to obtain a three-dimensional cultural knowledge graph; S4. Based on the three-dimensional cultural knowledge graph, train the query rewriting model to obtain the adaptive query rewriting model, obtain the tourist question, input the tourist question into the adaptive query rewriting model to obtain the graph query statement and the vector retrieval request, and perform graph structure query and embedded vector retrieval on the three-dimensional cultural knowledge graph based on the graph query statement and the vector retrieval request to obtain the graph query result and the vector retrieval result. Rearrange and fuse the graph query result and the vector retrieval result to obtain the enhanced context, and obtain the domain-enhanced answer text based on the enhanced context. S5. Perform speech synthesis on the enhanced response text to obtain the response speech and phoneme sequence, and use the response speech and the phoneme sequence as the visitor query result.

2. The method according to claim 1, characterized in that, The process involves entity recognition and relation extraction of the atomic data units to construct an initial knowledge graph. A graph embedding space is then obtained through graph representation learning on this initial knowledge graph. Furthermore, multimodal semantic features are extracted from the atomic data units using text, audio, and video data, resulting in candidate triples, including: S11. Context-encode the atomic data units to obtain a context-aware text embedding vector sequence; S12. Perform named entity recognition on the text embedding vector sequence to identify culturally specific type entities in the text and obtain a set of entities carrying cultural semantic type annotations. S13. Using each entity in the entity set as an initial node and the entity relationships extracted from the text embedding vector sequence as initial edges, construct the initial knowledge graph; S14. Perform graph representation learning training on the initial knowledge graph, learn to recognize the neighborhood structure information of the initial node in the initial knowledge graph, extract features from the neighborhood structure information to obtain embedding vectors, and summarize the embedding vectors of all the initial nodes to form the graph embedding space. S15. Perform multi-part structure separation on the atomic data unit, extract the fundamental frequency corresponding to the continuous bass and the melody of the high-pitched part and integrate them to obtain the melody embedding vector. Extract the spatiotemporal features of the skeletal point motion trajectory of the ethnic dance action video in the atomic data unit to obtain the action primitive embedding vector. S16. Semantically align the melody embedding vector and the action primitive embedding vector with the text embedding vector sequence to obtain multimodal semantic features, and integrate the entity set and the multimodal semantic features to obtain the candidate triplet.

3. The method according to claim 1, characterized in that, The domain-enhanced response text obtained based on the enhanced context includes: S21. Perform word-by-word autoregressive decoding on the enhanced context and the tourist question to obtain the currently generated text sequence; S22. Calculate the semantic implication score between the text sequence and the enhanced context; S23. Input the semantic implication relation score into the probability adjustment function to calculate the adjusted vocabulary probability distribution; wherein, the calculation expression of the vocabulary probability distribution function is: in, To adjust the probability distribution of the vocabulary, The probability distribution of the original vocabulary. The target word for the current decoding step. All context words preceding the current word, To control the adjustable hyperparameters of cultural constraint strength, The semantic entailment score. For time steps, The words in the vocabulary list; S24. Select the word with the highest probability in the adjusted vocabulary probability distribution as the current word to be generated, add the current word to be generated to the generated text sequence to obtain the updated generated text sequence, and use the updated generated text sequence as the generated text sequence for the next decoding step. S25. Repeat S22 to S24 until decoding ends, and use the final generated text sequence at the end of decoding as the domain-enhanced response text.

4. The method according to claim 1, characterized in that, The process of synthesizing the enhanced response text to obtain the response speech and phoneme sequence, and then using the response speech and phoneme sequence as the visitor query results, further includes: S31. Obtain the interaction log of the tourist's query results, and extract the feedback signal from the interaction log; S32. Perform state analysis on the feedback signal, extract multi-dimensional interaction state information, and perform feature mapping on the interaction state information to obtain an interaction state vector; S33. Perform probabilistic reasoning on the interaction state vector, calculate the posterior probability distribution of each preset root cause node, and take the root cause node whose posterior probability in the posterior probability distribution exceeds the attribution threshold as the root cause type causing the interaction problem. S34. When the root cause type is knowledge deficiency, a knowledge supplementation instruction is generated and fed back to S2, so that S2 supplements the three-dimensional cultural knowledge graph according to the knowledge supplementation instruction to obtain an updated three-dimensional cultural knowledge graph. When the root cause type is incorrect answer, an answer correction instruction is generated and fed back to S4, so that S4 adjusts according to the answer correction instruction to obtain an updated domain-enhanced answer text generation process. The updated three-dimensional cultural knowledge graph and the updated domain-enhanced answer text generation process are used to process subsequent visitor questions to obtain optimized visitor query results.

5. The method according to claim 1, characterized in that, The process involves calculating the structural consistency score of the candidate triples in the graph embedding space, comparing the structural consistency score with a consistency threshold, designating the candidate triples whose structural consistency scores satisfy the consistency threshold as graph nodes, and using the cultural object identifiers corresponding to the graph nodes as relation edges to obtain a three-dimensional cultural knowledge graph, including: S41. Obtain the head entity embedding vector, relation embedding vector, and tail entity embedding vector of the candidate triple from the graph embedding space; S42. Calculate the sum of the head entity embedding vector and the relation embedding vector to obtain a first vector. Calculate the Euclidean distance between the first vector and the tail entity embedding vector. Use the Euclidean distance as the structural consistency score of the candidate triple in the graph embedding space. S43. Based on the mean and standard deviation of the structural consistency scores of existing triplet structures in the current graph, a consistency threshold is obtained. The structural consistency score is compared with the consistency threshold. When the structural consistency score is less than or equal to the consistency threshold, the candidate triplet is determined to pass the verification and a passed triplet is obtained. S44. The head entity and tail entity of the passed triple are used as the graph node, and the relation type of the passed triple and the cultural object identifier corresponding to the graph node are used as relation edges to obtain the three-dimensional cultural knowledge graph.

6. A digital human intelligent interactive system for cultural tourism adapted to the cultural context of ethnic minorities, used to implement the method according to any one of claims 1 to 5, characterized in that, The system includes: The data encapsulation module is used to acquire multimodal raw data for ethnic minority cultures, segment the multimodal raw data according to speech segments or image frames to obtain raw byte streams, acquire acquisition device information and acquisition timestamps, and convert the acquisition device information and acquisition timestamps into cultural object identifiers through hash operations, and combine the raw byte streams and cultural object identifiers to obtain atomic data units; The graph construction module is used to perform entity recognition and relation extraction on the atomic data units to construct an initial knowledge graph. On the initial knowledge graph, a graph embedding space is obtained through graph representation learning. The atomic data units are used to extract text, audio and video features to obtain multimodal semantic features and obtain candidate triples. The knowledge verification module is used to calculate the structural consistency score of the candidate triple in the graph embedding space, compare the structural consistency score with the consistency threshold, take the candidate triple corresponding to the structural consistency score satisfying the consistency threshold as a graph node, and take the cultural object identifier corresponding to the graph node as a relation edge to obtain a three-dimensional cultural knowledge graph. The answer generation module is used to train a query rewriting model based on the three-dimensional cultural knowledge graph to obtain an adaptive query rewriting model, acquire tourist questions, input the tourist questions into the adaptive query rewriting model to obtain graph query statements and vector retrieval requests, and perform graph structure query and embedded vector retrieval on the three-dimensional cultural knowledge graph based on the graph query statements and vector retrieval requests to obtain graph query results and vector retrieval results. The graph query results and vector retrieval results are rearranged and fused to obtain enhanced context, and domain-enhanced answer text is obtained based on the enhanced context. The result output module is used to perform speech synthesis on the domain-enhanced answer text to obtain the answer speech and phoneme sequence, and to use the answer speech and the phoneme sequence as the visitor query result.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images