Cultural knowledge fusion dialogue method and system for mixed context

Abstract

The invention discloses a cultural knowledge fusion dialogue method and system for a mixed context, and the method comprises the steps: obtaining multi-modal input information, and constructing mixed context data; retrieving the mixed context data in the cross-language mapping model, and determining cultural background information; determining cultural knowledge entries based on the cultural context information; and generating a target text based on the mixed context data and the corresponding cultural knowledge entries. Therefore, during dialogue translation, cultural knowledge backgrounds such as languages, regions, voice features and the like of the multi-modal input information are recognized, the multi-modal input information is placed into the high-dimensional semantic space for eliminating language differences, and then coherent contents conforming to current dialogue scenes and user preferences are output according to the cultural knowledge backgrounds and contexts. Under the condition of multilingual and cross-culture backgrounds, the accuracy of translation can be ensured, the real intention of dialogue can be conveyed, and the information transmission efficiency can be effectively ensured.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for cultural knowledge fusion dialogue in mixed contexts. Background Technology

[0002] With increasing global exchanges across various fields, beyond traditional political and business areas, cross-border tourism and e-commerce are also experiencing a surge in translation needs. Typically, in formal political and business exchanges, both parties use standardized terminology. However, in on-the-spot conversations during cross-border tourism or ad-hoc inquiries during e-commerce transactions, the speakers are not professional translators and may not use standard grammar, potentially leading to a mix of languages. Furthermore, even when the same language is used, geographical and cultural differences can result in variations such as cultural customs, slang, allusions, and taboos.

[0003] As can be seen, the dialogue scenarios listed above not only cross language barriers but also involve cross-cultural backgrounds, and may even involve multilingual communication. Many expressions have deeper meanings or are closely related to specific cultural contexts. Existing translation tools lack corresponding cultural knowledge bases, are unable to effectively understand and translate beyond the literal meaning, and cannot accurately identify the linguistic focus of the dialogue.

[0004] Furthermore, existing translation tools are mostly based on sentence-level content and perform breakpoint translation, which cannot handle the dissolution of references and cultural implications in multi-turn dialogues. They lack a deep understanding of broader contexts such as multi-turn dialogues or non-linguistic cues. Especially when dealing with mixed language inputs such as Chinese-English intermingling and code-switching, they often fail to accurately capture the true intentions of the speakers, resulting in inaccurate translations or even semantic deviations.

[0005] In summary, existing translation tools fail to consider the cultural and knowledge background of the dialogue during translation and lack the ability to understand the whole text. In mixed contexts, they suffer from incoherent translations, poor personalized experience, and low information transmission efficiency. Summary of the Invention

[0006] The first aspect of this invention discloses a method for cultural knowledge fusion dialogue in mixed contexts, specifically including: Acquire multimodal input information and construct mixed context data; Retrieve the mixed-context data in the cross-language mapping model to determine cultural background information; Cultural knowledge items are determined based on the aforementioned cultural background information; Based on the mixed context data and the corresponding cultural knowledge entries, the target text is generated.

[0007] As an optional implementation, the multimodal input information includes text information, voice information, language information, geolocation information, and voice feature information; The mixed contextual data includes language tags, semantic vectors, and cultural trigger points generated based on the multimodal input information.

[0008] As an optional implementation, before obtaining the original text, the method further includes: Collect training data and establish mixed-context representations; A high-dimensional semantic space is established based on the aforementioned hybrid context representation; The cross-language mapping model is established by associating the high-dimensional semantic space with a cultural knowledge base; The cross-language mapping model is updated based on the multimodal input information and the collected iterative data.

[0009] As an optional implementation, the method further includes: A dynamic contextual representation model based on a Transformer encoder is constructed, which includes encoding historical dialogue information of fixed length using temporal information; The system fuses current multimodal input information with historical dialogue turn information to dynamically capture semantic associations and referential relationships in cross-linguistic contexts, outputting a fixed-dimensional context vector. ; Where PR(t) is the position code for time step t.

[0010] As an optional implementation, the step of establishing a high-dimensional semantic space based on the hybrid context representation includes: A multi-head attention adjustment mechanism with cultural gating is adopted to capture the correlation between different language segments in the high-dimensional semantic space. Based on language similarity and cultural relevance signals, the attention weight of specific language segments in the high-dimensional semantic space is dynamically adjusted. Each attention head is characterized as ; Calculate the language similarity bias for each language segment: ; in, , used to represent the pre-computed language similarity vector, with a value range of [0, 1]; α is the learnable scaling factor, with an initial value of 0.5; Modulate the cultural relevance of each language segment: ; in, , used to characterize the cultural knowledge embedding matrix; , used to characterize the current cultural feature vector; Multi-head aggregation of attention points: ; in, , is used to characterize the output projection matrix.

[0011] As an optional implementation, the method further includes: An N-gram probability model is set up to perform language preference analysis on continuous sequences of text in the multimodal input information. The N-gram probability model is as follows: ; in, This represents the i-th N-gram of text D; This represents the N-gram conditional probability obtained statistically from the training data; Furthermore, the Laplace smoothing method is used to eliminate zero probability.

[0012] A second aspect of this invention discloses a cultural knowledge fusion dialogue system for mixed contexts, comprising: The context understanding module is used to acquire multimodal input information and construct mixed context data; A cross-language attention mechanism module is used to retrieve the mixed context data in the cross-language mapping model through a culturally gated multi-head attention mechanism to determine cultural background information; The cultural knowledge integration module is used to determine cultural knowledge items based on the cultural background information. The natural language generation module is used to generate target text based on the mixed context data and corresponding cultural knowledge entries.

[0013] As an optional implementation, the system further includes: The context understanding module is also used to collect training data through a multilingual parallel corpus and establish a hybrid context representation based on the Transformer architecture; The cross-language attention mechanism module is also used to establish a high-dimensional semantic space based on the hybrid context representation; The model building module is used to associate the high-dimensional semantic space with a cultural knowledge base to establish the cross-language mapping model; The update module is used to update the cross-language mapping model based on the multimodal input information and the collected iterative data.

[0014] As an optional implementation, the cross-language attention mechanism module performs multilingual embedding mapping on the multimodal input information or the training data to establish a high-dimensional semantic space that is independent of language distinctions; In addition, dynamic attention calculation is performed to construct semantic associations and referential relationships within the high-dimensional semantic space; In addition, cultural weight adjustments are made to dynamically adjust the attention weights of specific language segments in the high-dimensional semantic space.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: During dialogue translation, the system identifies cultural background information such as language, region, and speech features of multimodal input information and places it into a high-dimensional semantic space that eliminates language differences. Then, based on cultural background and context, it outputs coherent content that conforms to the current dialogue scenario and user preferences. This ensures the accuracy of translation and conveys the true intent of the dialogue in multilingual and cross-cultural contexts, effectively ensuring the efficiency of information transmission. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the workflow of a cultural knowledge fusion dialogue method for mixed contexts disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the system structure of a cultural knowledge fusion dialogue system for mixed contexts disclosed in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that the terms "first," "second," "third," "fourth," etc., used in the specification and claims of this invention are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0020] Example 1 Please see Figure 1 ,like Figure 1 As shown in the figure, an embodiment of the present invention discloses a method for cultural knowledge fusion dialogue in mixed contexts, which includes the following steps: 101. Obtain multimodal input information and construct mixed context data.

[0021] In this embodiment, the multimodal input information includes text information, voice information, language information, geographic location information, and voice feature information; Mixed-context data includes language labels, semantic vectors, and cultural triggers generated based on multimodal input information.

[0022] Here, text information can be character data exchanged between the two parties online, and language information can be audio files.

[0023] The language information can be preset by the user. If the user has not selected a language, it will be left to step 102 for identification.

[0024] Geolocation information is collected based on the user's personal computer, smartphone, and translator to help identify the user's cultural background. For example, both the United States and the United Kingdom use English, but there are many differences in pronunciation, vocabulary, and grammar. Even within the United States, there are significant differences in grammar, slang, and common words among different states. Therefore, geolocation information can be used to further refine and clarify the user's cultural background to ensure accurate understanding of the inputter's intent.

[0025] Cultural triggers can be specific words or phrases. It is understandable that some words or phrases have deeper meanings in specific regions, and these need to be marked for further retrieval to avoid missing the deeper meanings that users want to express.

[0026] As an optional implementation, a dynamic contextual representation model based on a Transformer encoder is constructed, which includes encoding historical dialogue information of a fixed length using temporal information; The system fuses current multimodal input information with historical dialogue turn information to dynamically capture semantic associations and referential relationships in cross-linguistic contexts, outputting a fixed-dimensional context vector. ; Where PR(t) is the position code for time step t.

[0027] Here, multimodal input information is converted into context vectors in the Transformer encoder, which facilitates input into the cross-language mapping model.

[0028] In this embodiment, the mixed context data will be placed into the cross-language mapping model for retrieval. The cross-language mapping model is constructed and generated based on the training data and is continuously iterated and updated according to the multimodal input information during use.

[0029] As an optional implementation, before acquiring the original text, training data is collected to establish a hybrid context representation; a high-dimensional semantic space is established based on the hybrid context representation; a cultural knowledge base is used to associate the high-dimensional semantic space to establish a cross-language mapping model; and the cross-language mapping model is updated based on multimodal input information and collected iterative data.

[0030] Here, the cultural knowledge base is built based on Wikipedia or regional corpora, and is regularly updated through user feedback data and historical records.

[0031] Some of the sources for the construction of the cultural knowledge base are as follows:

[0032] A multi-head attention adjustment mechanism with cultural gating is adopted to capture the correlation between different language segments in a high-dimensional semantic space. Based on language similarity and cultural relevance signals, the attention weight of specific language segments in the high-dimensional semantic space is dynamically adjusted. Each attention head is characterized as ; Calculate the language similarity bias for each language segment: ; in, , used to represent the pre-computed language similarity vector, with a value range of [0, 1]; α is the learnable scaling factor, with an initial value of 0.5; Modulate the cultural relevance of each language segment: ; in, This is used to characterize the cultural knowledge embedding matrix; , used to represent the current cultural feature vector; Perform multi-head aggregation on each attention head: ; Among them, , used to represent the output projection matrix.

[0033] Furthermore, set the N-gram probability model to perform language preference analysis on the continuous sequence of text in the multi-modal input information. The N-gram probability model is as follows: ; Among them, represents the i-th N-gram of text D; represents the N-gram conditional probability statistically obtained from the training data; And, use the Laplace smoothing method to eliminate zero probabilities.

[0034] Thus, establish a cross-language mapping model based on the training data. In the model, the text information eliminates language barriers, and texts with similar meanings are mapped to similar positions in the high-dimensional semantic space. For example, the mapping distances of "你好" and "hello" are close.

[0035] In addition, different language components are aligned at the semantic level. For example, in the sentence "我想预订一张去北京的ticket", although there are cross-language and multi-positional relationships among "北京", "ticket" and "预订", they are semantically related in this sentence and will be given key attention in the subsequent translation process.

[0036] In addition, cultural trigger points will also be identified and marked. For example, after identifying "春节", the query instance {entity: "春节", relations: ["习俗", "食物", "禁忌"], context: "与家人团聚"} will be triggered, and the relevant candidate cultural knowledge entries retrieved will be scored for relevance according to the complete context, and the cultural knowledge entry with the highest score will be selected for cultural adaptation processing. The adapted cultural knowledge entry can be converted into a set of feature vectors or a descriptive text for reference when generating the target text later.

[0037] 102. Retrieve mixed context data in the cross-language mapping model to determine cultural background information.

[0038] In this embodiment, the cultural background information is identified based on language information, geographical location information, cultural trigger points, etc. on the one hand, and is determined according to the vector mutual relationship in the cross-language mapping model on the other hand.

[0039] Here, cultural background information can be obtained by matching semantic similarity and knowledge reasoning retrieval within a structured cultural knowledge base.

[0040] 103. Determine cultural knowledge items based on cultural background information.

[0041] In this embodiment, the content retrieved and matched in step 102 can be idiom explanations, sources of allusions, etiquette norms, taboos, regional dialect differences, etc. After scoring and ranking the multiple cultural knowledge items retrieved, the most relevant cultural knowledge items are selected and activated to achieve efficient matching.

[0042] 104. Generate target text based on mixed context data and corresponding cultural knowledge entries.

[0043] In this embodiment, based on the cultural knowledge entries obtained through retrieval, and through the dynamic adaptation of cultural knowledge entries and the contextual and scene information obtained through context awareness, it is possible to further match the user's cultural background, expression habits and current expression context. On the basis of standardized translation, further personalized adjustments and adaptations are made. From the dimensions of content selection, expression mode conversion, sensitive content filtering and multilingual paraphrasing, a target text that conforms to the culture, is coherent and has a consistent expression style is generated.

[0044] As an optional implementation, multiple target texts can be output for selection, such as multiple target texts with different levels of formality, different levels of politeness, and whether they contain humor and emotional color, so that users can choose the target text they like best for conversation.

[0045] As another alternative implementation method, the semantic weight of the target text selected by the user in the higher-dimensional semantic space will be adjusted to highlight the user's personalized needs, and the subsequent translation and conversion results will be more in line with the user's preferences.

[0046] As another alternative implementation method, the multimodal input information input by the two or more parties in the dialogue will also be used as training data to continuously update and iterate the cross-language mapping model, ensuring that its translation performance continues to improve.

[0047] It is evident that in dialogue translation, recognizing cultural background information such as language, region, and speech features of multimodal input information, and placing this information into a high-dimensional semantic space that eliminates language differences, allows for the output of coherent content that aligns with the current dialogue scenario and user preferences, even in multilingual and cross-cultural contexts. This ensures translation accuracy, conveys the true intent of the dialogue, and effectively guarantees information transmission efficiency. Example 2 Please refer to the diagram. (For example...) Figure 2As shown, this cultural knowledge fusion dialogue system for mixed contexts may include: The context understanding module is used to acquire multimodal input information and construct mixed context data; The cross-linguistic attention mechanism module is used to retrieve mixed contextual data in the cross-linguistic mapping model through a culturally gated multi-head attention mechanism to determine cultural background information; The cultural knowledge integration module is used to determine cultural knowledge items based on cultural background information; The natural language generation module is used to generate target text based on mixed context data and corresponding cultural knowledge entries; The context understanding module is also used to collect training data from multilingual parallel corpora and build hybrid context representations based on the Transformer architecture; The cross-linguistic attention mechanism module is also used to build a high-dimensional semantic space based on mixed context representations; The model building module is used to establish a cross-language mapping model by associating a cultural knowledge base with a high-dimensional semantic space. The update module is used to update the cross-language mapping model based on multimodal input information and collected iterative data.

[0048] In this embodiment, the dialogue statements are not directly translated and output. Instead, the cultural background such as language, region, and speech features is first identified and placed into a high-dimensional semantic space that eliminates language differences. Then, based on the cultural background and context, coherent content that conforms to the current dialogue scenario and user preferences is output.

[0049] Therefore, by effectively eliminating the ambiguity and semantic omissions caused by sentence-by-sentence translation of existing translation tools, the accuracy of translation can be ensured in multilingual and cross-cultural contexts, conveying the true intent of the dialogue and effectively ensuring the efficiency of information transmission.

[0050] The foregoing has provided a detailed description of a cultural knowledge fusion dialogue method and system for mixed contexts disclosed in the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for cultural knowledge integration and dialogue in mixed contexts, characterized in that, The method includes: Acquire multimodal input information and construct mixed-context data; Retrieve the mixed-context data in the cross-language mapping model to determine cultural background information; Cultural knowledge items are determined based on the aforementioned cultural background information; Based on the mixed context data and the corresponding cultural knowledge entries, the target text is generated.

2. The cultural knowledge fusion dialogue method for mixed contexts according to claim 1, characterized in that, include: The multimodal input information includes text information, voice information, language information, geographic location information, and voice feature information; The mixed contextual data includes language tags, semantic vectors, and cultural trigger points generated based on the multimodal input information.

3. The cultural knowledge fusion dialogue method for mixed contexts according to claim 1, characterized in that, Prior to obtaining the original text, the method further includes: Collect training data and establish mixed-context representations; A high-dimensional semantic space is established based on the aforementioned hybrid context representation; The cross-language mapping model is established by associating the high-dimensional semantic space with a cultural knowledge base; The cross-language mapping model is updated based on the multimodal input information and the collected iterative data.

4. The cultural knowledge fusion dialogue method for mixed contexts according to claim 3, characterized in that, The method further includes: A dynamic contextual representation model based on a Transformer encoder is constructed, which includes encoding historical dialogue information of fixed length using temporal information; The system fuses current multimodal input information with historical dialogue turn information to dynamically capture semantic associations and referential relationships in cross-linguistic contexts, outputting a fixed-dimensional context vector. ； Where PR(t) is the position code for time step t.

5. The cultural knowledge fusion dialogue method for mixed contexts according to claim 4, characterized in that, The establishment of a high-dimensional semantic space based on the hybrid context representation includes: A multi-head attention adjustment mechanism with cultural gating is adopted to capture the correlation between different language segments in the high-dimensional semantic space. Based on language similarity and cultural relevance signals, the attention weight of specific language segments in the high-dimensional semantic space is dynamically adjusted. Each attention head is characterized as ； Calculate the language similarity bias for each language segment: ； in, , used to represent the pre-computed language similarity vector, with a value range of [0, 1]; α is the learnable scaling factor, with an initial value of 0.5; Modulate the cultural relevance of each language segment: ； in, , used to characterize the cultural knowledge embedding matrix; , used to characterize the current cultural feature vector; Multi-head aggregation of attention points: ； in, , is used to characterize the output projection matrix.

6. The cultural knowledge fusion dialogue method for mixed contexts according to claim 5, characterized in that, The method further includes: An N-gram probability model is set up to perform language preference analysis on continuous sequences of text in the multimodal input information. The N-gram probability model is as follows: ； in, This represents the i-th N-gram of text D; This represents the N-gram conditional probability obtained statistically from the training data; Furthermore, the Laplace smoothing method is used to eliminate zero probability.

7. A cultural knowledge fusion dialogue system for mixed contexts, characterized in that, include: The context understanding module is used to acquire multimodal input information and construct mixed context data; A cross-language attention mechanism module is used to retrieve the mixed context data in the cross-language mapping model through a culturally gated multi-head attention mechanism to determine cultural background information; The cultural knowledge integration module is used to determine cultural knowledge items based on the cultural background information. The natural language generation module is used to generate target text based on the mixed context data and corresponding cultural knowledge entries.

8. A cultural knowledge fusion dialogue system for mixed contexts according to claim 7, characterized in that, The system also includes: The context understanding module is also used to collect training data through a multilingual parallel corpus and establish a hybrid context representation based on the Transformer architecture; The cross-language attention mechanism module is also used to establish a high-dimensional semantic space based on the hybrid context representation; The model building module is used to associate the high-dimensional semantic space with a cultural knowledge base to establish the cross-language mapping model; The update module is used to update the cross-language mapping model based on the multimodal input information and the collected iterative data.

9. A cultural knowledge fusion dialogue system for mixed contexts according to claim 8, characterized in that, include: The cross-language attention mechanism module performs multi-language embedding mapping on the multimodal input information or the training data to establish a high-dimensional semantic space that is independent of language distinctions; In addition, dynamic attention calculation is performed to construct semantic associations and referential relationships within the high-dimensional semantic space; In addition, cultural weight adjustments are made to dynamically adjust the attention weights of specific language segments in the high-dimensional semantic space.

Images