A hierarchical code-switching augmentation method based on cross-cultural adaptation

Abstract

The application discloses a hierarchical code mixing augmentation method based on cross-cultural adaptation. First, common cultural concepts of different target languages are obtained and a cultural concept set is formed, then a cultural concept adaptation graph is constructed by means of the hyponym-hypernym relationship of a multilingual semantic network, and cross-culture and cross-language are unified by a hierarchical code mixing strategy to improve the performance of a pre-trained multilingual multimodal model on a data set with cross-cultural attributes. Compared with the prior art, the application utilizes the hyponym-hypernym relationship of a semantic dictionary to construct a culture concept adaptation method without labeling, and simultaneously utilizes the obtained concepts in combination with the hierarchical code mixing method to effectively improve the performance of the pre-trained multilingual multimodal model on the data set with cross-cultural attributes.

Description

Technical Field

[0001] This invention relates to the application of cultural adaptation and pre-trained models on multimodal and multilingual datasets, and particularly to technical methods for improving model performance under cross-cultural settings. Background Technology

[0002] Human language and imagery are intertwined with their cultures, co-evolving and influencing each other. Different countries often have different cultures. In recent years, while many methods have improved the cross-linguistic and cross-modal capabilities of models, they have often neglected cross-cultural capabilities. Improving the transferability of models across different cultures is important, as it can help models leverage high-resource language and cultural data to understand low-resource languages ​​and cultures.

[0003] In recent years, significant progress has been made in cross-linguistic and cross-modal approaches within the computer vision (CV) and natural language processing (NLP) communities. For example, XLM introduced three language modeling objectives: causal language modeling, masked language modeling, and translational language modeling, and its performance on cross-linguistic tasks significantly outperformed previous state-of-the-art techniques. UC2 proposed masked region-to-token modeling and visual translational language modeling pre-training tasks, learning context-aware representations that combine language and vision. While these methods improve the cross-linguistic and cross-modal capabilities of models, they neglect cross-cultural capabilities. Enhancing the model's perception and transfer capabilities across different cultures is crucial; effective cross-cultural approaches can improve the model's ability to understand low-resource languages ​​and cultures using high-resource languages ​​and cultures. Indeed, several works in the past few years have addressed or focused on cultural topics. For example, the semantics of part-of-day nouns have been studied by leveraging Twitter and specific time greetings (e.g., "good morning") used in different cultures, or cultural differences between the US and India regarding common cross-cultural rituals such as birth, marriage, and funerals have been investigated. Some related works have also expanded existing visual question-answering and inference datasets using data from non-Western cultures. However, most of them focus on the impact of cultural differences or building benchmarks to test the cross-cultural competence of models, rather than improving them. This is mainly because improving cross-cultural competence is difficult: (1) labeling cross-cultural data is costly because few labelers are familiar with two cultures from different countries. (2) The language itself is scarce, and cultural concepts within it are even less frequent. Thus, a novel and important question arises: how to improve the cross-cultural competence of models? This patent adopts a label-free cultural adaptation method and combines it with a code-mixing strategy to simplify the cross-cultural problem into a cross-linguistic problem, thereby effectively improving the model's capabilities. Summary of the Invention

[0004] The purpose of this invention is to solve the problems existing in the prior art and to provide a hierarchical code-mixing augmentation algorithm based on cross-cultural adaptation.

[0005] The specific technical solution adopted in this invention is as follows:

[0006] In a first aspect, the present invention provides a hierarchical code-mixing augmentation method based on cross-cultural adaptation, which is used to augment language sample data under the source language culture. The method steps are as follows:

[0007] S1: For all target languages ​​and cultures to be augmented, obtain the cultural concepts under each language and culture and form a cultural concept set;

[0008] S2: Traverse each cultural concept in the cultural concept set, and for the current cultural concept that has been traversed, construct a cultural adaptation graph of the current cultural concept in the source language culture using the hypernym and hyponym relationship graph in the multilingual semantic network;

[0009] S3. For each current cultural concept in the S2 traversal process, extract all other nodes at the same hierarchical level as the current cultural concept from the corresponding cultural adaptation graph. Take the cultural concepts represented by all the extracted nodes as the adaptation set of the current cultural concept in the source language culture. At the same time, set a confidence level for each cultural concept in the adaptation set. The closer the path distance between each cultural concept in the adaptation set and the current cultural concept in the cultural adaptation graph, the higher the confidence level set.

[0010] S4. Using a hierarchical code-switching strategy, code-switching is performed on language sample data under the source language culture at the cultural concept level, phrase level, and word level in sequence to obtain augmented language samples that integrate concepts from other languages ​​and cultures, which are then used to train a multimodal and multilingual pre-trained model.

[0011] Preferably, in S1, the source language culture is English culture, and the target language culture is other languages ​​and cultures from different countries or regions.

[0012] Preferably, in S1, the language sample data is a visual reasoning task dataset. Each sample in the dataset contains a pair of images, a text statement describing the pair of images, and a label indicating whether the text statement correctly describes the pair of images. When using a hierarchical code-switching strategy for code conversion, the object of conversion is the text statement describing the pair of images in the sample.

[0013] Preferably, in step S2, the specific steps for constructing a cultural adaptation graph for each traversed current cultural concept are as follows:

[0014] S21: For the current cultural concept that has been traversed, based on the target language culture to which the current cultural concept belongs, query its multi-level superordinate words in the superordinate and hypoordinate word relationship graph of the multilingual semantic network.

[0015] S22: For each hypernym found in S21, continue to search for the synonyms of the hypernym in the source language culture in the relation graph, and then search for all the hyponyms of each synonym in the source language culture.

[0016] S23: The current cultural concept that will be traversed, as well as all the hypernyms, hyponyms, and synonyms found in S21 and S22, will be used as nodes of the graph. All nodes will be connected according to their word order relationships to form a cultural adaptation graph of the current cultural concept in the source language culture.

[0017] Preferably, in step S21, the multi-level hypernym is a third-level hypernym.

[0018] Preferably, in S2, the multilingual semantic network uses ConceptNet and WordNet. In ConceptNet, " / r / IsA" is used to query hypernyms and " / r / Synonym" is used to query synonyms, while in WordNet, "hyponyms()" is used to query hyponyms.

[0019] Preferably, in step S4, each step of obtaining an augmented language sample through code conversion using a hierarchical code mixing strategy is as follows:

[0020] S41: Randomly sample one target language culture L from all target language cultures, then sample a cultural concept C1 from the set of cultural concepts in the target language culture L, and then sample a cultural concept C2 from the adaptation set CA of cultural concept C1 in the source language culture. When sampling cultural concept C2 from the adaptation set CA, the probability of each cultural concept in the set being sampled is positively correlated with its confidence level.

[0021] S42: Retrieve language sample S1 containing cultural concept C2 from the language sample data under the source language culture, and then perform code mixing at the cultural concept level on language sample S1, replacing cultural concept C2 in language sample S1 with cultural concept C1 sampled in S41 to form language sample S2.

[0022] S43: Perform phrase-level code mixing on the language sample S2 obtained in S42, sample phrases from the source language culture from the language sample S2, sample a target language culture, and replace each sampled phrase with a synonym of the phrase in the sampled target language culture to form language sample S3.

[0023] S44: Perform word-level code mixing on the language sample S3 obtained in S43, sample words from the source language culture from the language sample S3, sample a target language culture, and replace each sampled word with a synonym of the word in the sampled target language culture to form the final augmented language sample S4.

[0024] Preferably, in step S43, before performing phrase-level code mixing, high-frequency phrases need to be selected from the language sample data of the source language culture to construct a source language phrase set, and each phrase in the source language phrase set is translated into other target language cultures to form synonyms; when performing phrase-level code mixing, it is necessary to determine whether there are any phrases in the language sample S2 that exist in the source language phrase set. If they exist, it is determined whether each phrase in the source language phrase set should be replaced according to a preset replacement probability. If there are phrases that need to be replaced, a target language culture is sampled and synonyms of the phrases that need to be replaced are found in this target language culture, and each phrase that needs to be replaced is replaced with the found synonyms.

[0025] Preferably, in step S44, before performing word-level code mixing, high-frequency words need to be selected from the language sample data of the source language culture to construct a source language word set, and each word in the source language word set is translated into other target language cultures to form synonyms. When performing word-level code mixing, it is necessary to determine whether there are any words in the language sample S2 that exist in the source language word set. If they exist, it is determined whether each word in the source language word set should be replaced according to a preset replacement probability. If there are words that need to be replaced, a target language culture is sampled and synonyms of the words to be replaced are found in this target language culture. Each word to be replaced is replaced with the found synonym.

[0026] Secondly, this invention provides a cross-cultural and cross-linguistic model training method, which trains a multimodal and multilingual pre-trained model on a dataset containing augmented language samples obtained by any of the augmentation methods described in the first aspect, thereby obtaining a model with better cross-cultural and cross-linguistic capabilities.

[0027] Preferably, the multimodal multilingual pre-trained model is a mUNITER, xUNITER, M3P, or UC2 model, and the loss function used for training is preferably cross-entropy loss. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the steps of a hierarchical code-mixing augmentation method based on cross-cultural adaptation.

[0029] Figure 2This is a diagram illustrating the cultural adaptation of the erhu.

[0030] Figure 3 Here are sampling examples from NLVR2 and MaRVL.

[0031] Figure 4 A flowchart representing cultural adaptation and hierarchical code mixing. Detailed Implementation

[0032] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0033] like Figure 1 As shown, in a preferred embodiment of the present invention, a hierarchical code-mixing augmentation method based on cross-cultural adaptation is provided, which is used to augment language sample data under the source language culture. The specific steps of the augmentation method are as follows:

[0034] S1: For all target languages ​​and cultures to be augmented, obtain the cultural concepts under each language and culture and form a cultural concept set.

[0035] The source language culture and target language culture mentioned above can be selected according to actual needs. Since English is currently the most widely used language in the world and has the most language data, in the embodiments of the present invention, English culture can be used as the source language culture, while the target language culture can be other languages ​​and cultures of different countries or regions.

[0036] Furthermore, the language sample data can theoretically be any data containing language text. In the embodiments of this invention, the language sample data is a visual reasoning task dataset. Each sample in the dataset contains a pair of images, a text statement describing the pair of images, and a label indicating whether the text statement correctly describes the pair of images. Therefore, when performing code-switching using a hierarchical code-switching strategy, the object of conversion is the text statement describing the pair of images in the sample.

[0037] S2: Traverse each cultural concept in the cultural concept set. For the current cultural concept that has been traversed, construct a cultural adaptation graph of the current cultural concept in the source language culture using the hypernym and hyponym relationship graph in the multilingual semantic network.

[0038] In an embodiment of the present invention, in step S2 above, during the traversal of each cultural concept in the cultural concept set, the currently traversed cultural concept is called the current cultural concept. Each current cultural concept belongs to a specific target language culture. Therefore, the specific steps for constructing a cultural adaptation map for each traversed current cultural concept are as follows:

[0039] S21: For the current cultural concept that has been traversed, based on the target language culture to which the current cultural concept belongs, query its multi-level superordinate words in the superordinate and hypoordinate word relationship graph of the multilingual semantic network.

[0040] S22: For each hypernym found in S21, continue to search for the synonyms of the hypernym in the source language culture in the relation graph, and then search for all the hyponyms of each synonym in the source language culture.

[0041] S23: The current cultural concept that will be traversed, as well as all the hypernyms, hyponyms, and synonyms found in S21 and S22, will be used as nodes of the graph. All nodes will be connected according to their word order relationships to form a cultural adaptation graph of the current cultural concept in the source language culture.

[0042] It is important to note that while a word's hypernyms may also have their own hypernyms, introducing too many higher-level hypernyms can cause the words in the final constructed cultural adaptation graph to deviate from the current cultural concepts. Therefore, the order of hypernyms queried in a multilingual semantic network needs to be reasonably controlled. In the embodiments of this invention, the order of hypernyms is controlled to three, meaning that hypernyms within the third order are queried.

[0043] Meanwhile, to facilitate the subsequent determination of the final adaptation set, when querying all hyponyms of each synonym in the source language culture in step S22 above, the level of the queried hyponyms also needs to be controlled accordingly. The lowest queried hyponym level for each synonym in the source language culture needs to be the same as the level of the current cultural concept. Thus, all leaf nodes in the cultural adaptation graph are at the same hyponym / hypernym level as the current cultural concept.

[0044] Furthermore, the aforementioned multilingual semantic network can be any semantic network that possesses a relational graph of hypernyms and hyponyms. In the embodiments of the present invention, the aforementioned multilingual semantic network employs ConceptNet and WordNet, wherein in ConceptNet, " / r / IsA" is used to query hypernyms, " / r / Synonym" is used to query synonyms, and in WordNet, "hyponyms()" is used to query hyponyms.

[0045] S3. For each current cultural concept in the S2 traversal process, extract all other nodes at the same hierarchical level as the current cultural concept from the corresponding cultural adaptation graph. Take the cultural concepts represented by all the extracted nodes as the adaptation set of the current cultural concept in the source language culture. At the same time, set a confidence level for each cultural concept in the adaptation set. The closer the path distance between each cultural concept in the adaptation set and the current cultural concept in the cultural adaptation graph, the higher the confidence level set.

[0046] S4. Using a hierarchical code-switching strategy, code-switching is performed on language sample data under the source language culture at the cultural concept level, phrase level, and word level in sequence to obtain augmented language samples that integrate concepts from other languages ​​and cultures, which are then used to train a multimodal and multilingual pre-trained model.

[0047] In an embodiment of the present invention, in step S4 above, each time code-switching is performed through a hierarchical code-switching strategy, an augmented language sample can be obtained. The specific steps of obtaining an augmented language sample are as follows:

[0048] S41: Randomly sample one target language culture L from all target language cultures, then sample a cultural concept C1 from the set of cultural concepts in target language culture L, and then sample a cultural concept C2 from the adaptation set CA of cultural concept C1 in the source language culture. When sampling cultural concept C2 from the adaptation set CA, the probability of each cultural concept in the set being sampled is positively correlated with its confidence level. That is, the higher the confidence level of a certain cultural concept determined in step S3, the higher the probability of this cultural concept being sampled, and the easier it is to be sampled.

[0049] S42: Retrieve language sample S1 containing cultural concept C2 from the language sample data under the source language culture, and then perform code mixing at the cultural concept level on language sample S1, replacing cultural concept C2 in language sample S1 with cultural concept C1 sampled in S41 to form language sample S2.

[0050] S43: Perform phrase-level code mixing on the language sample S2 obtained in S42, sample phrases from the source language culture from the language sample S2, sample a target language culture, and replace each sampled phrase with a synonym of the phrase in the sampled target language culture to form language sample S3.

[0051] In an embodiment of the present invention, before performing phrase-level code mixing in step S43, it is necessary to first statistically analyze the n-gram phrase structure of the language sample data of the source language culture, thereby selecting high-frequency phrases from the language sample data of the source language culture to construct a source language phrase set, and translating each phrase in the source language phrase set into other target language cultures to form synonyms; when performing phrase-level code mixing, it is necessary to determine whether there are any phrases in the language sample S2 that exist in the source language phrase set. If they exist, it is determined whether each phrase in the source language phrase set should be replaced according to a preset replacement probability. If there are phrases that need to be replaced, a target language culture is sampled and synonyms of the phrases that need to be replaced are found in this target language culture, and each phrase that needs to be replaced is replaced with the found synonyms.

[0052] S44: Perform word-level code mixing on the language sample S3 obtained in S43, sample words from the source language culture from the language sample S3, sample a target language culture, and replace each sampled word with a synonym of the word in the sampled target language culture to form the final augmented language sample S4.

[0053] In an embodiment of the present invention, before performing word-level code mixing in step S44, it is necessary to select high-frequency words from the language sample data of the source language culture to construct a source language word set, and translate each word in the source language word set into other target language cultures to form synonyms; when performing word-level code mixing, it is necessary to determine whether there are any words in the language sample S2 that exist in the source language word set. If they exist, it is determined whether each word in the source language word set should be replaced according to a preset replacement probability. If there are words that need to be replaced, a target language culture is sampled and synonyms of the words to be replaced are found in this target language culture, and each word to be replaced is replaced with the found synonyms.

[0054] Since each code-switching operation using the hierarchical code-switching strategy yields an augmented language sample, steps S41 to S44 can be repeated N times as needed, based on the total number N of augmented language samples obtained. In steps S41 to S44, the sampling probabilities for the target language culture, cultural concepts, phrases, and words all need to be pre-set.

[0055] The augmented language samples obtained in S4 above can be added to the original source language culture's language sample data for cross-cultural and cross-linguistic model training. Therefore, this embodiment of the invention can further provide a cross-cultural and cross-linguistic model training method, which involves training a multimodal multilingual pre-trained model on a dataset containing the augmented language samples obtained by the aforementioned augmentation method, thereby obtaining a model with better cross-cultural and cross-linguistic capabilities. Here, the multimodal multilingual pre-trained model refers to a model that has been pre-trained on large-scale image and text data, and can be selected according to actual needs, such as mUNITER, xUNITER, M3P, or UC2 models. The loss function used for training the multimodal multilingual pre-trained model can be cross-entropy loss.

[0056] The present invention will now apply the hierarchical code-mixing augmentation method based on cross-cultural adaptation described in S1 to S4 above to a specific practical case to demonstrate its specific implementation and technical effects.

[0057] Example

[0058] In this embodiment, the implementation framework of the hierarchical code-mixing augmentation method based on cross-cultural adaptation is as described in steps S1 to S4 above, and is referred to below as the method of the present invention. The specific implementation details of each step are described in detail below.

[0059] First, it is necessary to collect cultural concepts from different countries. Three types of public online resources can be considered: (1) Wikipedia. For example, the article "Culture of Turkey" on Wikipedia lists many cultural concepts, including food, festivals, architecture, etc. (2) Official websites. Most countries provide official websites to introduce their culture. (3) Search engines. Some websites retrieved by search engines, such as travel guides, introduce local culture. The method of this invention requires the organization and merging of data from the three online resources to establish a set of cultural concepts. Then, a cultural adaptation graph is constructed using semantic networks and dictionaries. Specifically, the relationship between "above," "below," and synonyms in open and freely available semantic networks, such as ConceptNet and WordNet, is used to construct an adaptation mapping between different cultural concepts. Taking the Chinese "erhu" as an example, for examples of cultural adaptation of "erhu," please refer to [link to relevant examples]. Figure 2 The erhu only has the pinyin "erhu" and no corresponding English translation. Therefore, one feasible way to explain the meaning of "erhu" to people living in English-speaking countries who are unfamiliar with Chinese culture is to describe it as a Chinese violin—this is a form of cultural adaptation. This invention proposes a general cultural adaptation method and constructs a cultural adaptation graph using the relationships between hypernyms, hyponyms, and synonyms in open and freely accessible semantic dictionaries. Each leaf node in the cultural adaptation graph is potentially a cultural adaptation of the erhu. The shorter the path distance, the higher the accuracy. The erhu has a hypernym "bowed string instrument," whose English synonym is "". "Bowed string instrument" has many hyponyms, such as "violin" and "cello". Therefore, the cultural adaptations of the erhu could be "violin" and "cello". Furthermore, the hypernym of "bowed string instrument" is "instrument," which can be seen as a second-order hypernym of the erhu, with the English synonym "Musical Instrument". "Saxophone" and "Oboe" can also be considered potential cultural adaptations of the erhu. In fact, every leaf node in the cultural adaptation graph can be a cultural adaptation of the erhu. The shorter the path distance, the higher the accuracy. For example, "violin" and "cello" are better than "Saxophone" and "Oboe". This embodiment can give all leaf nodes and their path distances to the erhu using a simple iterative traversal algorithm.

[0060] Once the aforementioned cultural adaptation is achieved, it can be used to improve cross-cultural competence. The specific implementation is described below. First, the method in this embodiment requires selecting a benchmark that can test the model's cross-cultural competence. This embodiment considers a visual reasoning task across language and culture, trained using English NLVR2 and tested using MaRVL. See [link to documentation] Figure 3 Examples are shown, illustrating sampling from NLVR2 and MaRVL. NLVR2 is a semantically diverse dataset used for reasoning natural language descriptions of photographs. The task is to determine whether a description about a pair of images is correct. MaRVL is similar to NLVR2, except that NLVR2 descriptions are in English, while MaRVL spans five different languages ​​(Chinese, Tamil, Swahili, Indonesian, and Turkish). Therefore, the objective of this task is, given two images and a description, the model needs to evaluate the validity of the description given the images, which can be transformed into a classification problem. Previous state-of-the-art methods have found that both multilingual and monolingual models perform reasonably well in English (NLVR2). However, when these models are evaluated on languages ​​in MaRVL, the performance of the multilingual baseline in the zero-shot setting drops sharply, only slightly better than random selection. Further analysis reveals two sources of difficulty that make MaRVL challenging: 1) cross-cultural transfer (involving concepts beyond the distribution of the English dataset) and 2) cross-linguistic transfer.

[0061] With the help of cultural adaptation, cross-cultural difficulties can be transformed into cross-linguistic difficulties. Figure 4 This embodiment presents an overview of its method: First, data is collected from network resources to establish a cultural concept set, where different ellipses represent different cultural concepts. Then, a specified cultural concept set is sampled, and cultural adaptation is performed based on a probability distribution. Subsequently, the cultural adaptation set is sampled, with shorter path distances resulting in higher sampling probabilities. This embodiment uses the sampled cultural adaptations to retrieve data from the training dataset and performs code-switching on the retrieved sentences at three levels. The target languages ​​to be replaced in the code-switched sentences include Chinese, Swahili, Turkish, and Indonesian. Previous cross-language methods typically perform code-switching at the word level; to improve cross-cultural capabilities, this embodiment uses cultural concepts for code-switching. Figure 4 In this example, "violin" undergoes code-switching through its Chinese cultural adaptation to the erhu (a two-stringed bowed instrument). Simultaneously, this task is rich in phrases highly relevant to visual reasoning, such as "in the picture" and "on the left," which inspires the method in this embodiment to propose phrase-level code-switching. The alignment of these phrases is crucial for improving cross-language transfer models. Figure 4In the text, "onthe left" is code-switched at the phrase level using "kushoto". Finally, "image" and "player" are code-switched at the word level using "resim" and "peserta" respectively.

[0062] Multilingual, multimodal pre-trained models often exhibit inconsistent performance across different languages ​​and cultures on test datasets, potentially due to imbalanced pre-training resource allocation. Self-supervised pre-training objectives are difficult to address. The method of this invention reduces language and cultural bias by adjusting the sampling distribution. In fact, the method can improve model performance on specific language or cultural topics in a nearly controllable manner. For example, if the model is to be used for Turkish, the sampling probability for Turkish can be increased. Similarly, if the model is to be used in a specific context that might include many concepts related to traditional Chinese musical instruments, the method can increase the sampling probability for Chinese instruments. In summary, for a given language or cultural category, the method of this invention can significantly improve model capabilities in a very fine-grained manner.

[0063] To test the practical effectiveness of the multi-label-based masked language modeling technique described above, this embodiment selected a series of different target language cultures: Indonesian, Swahili, Tamil, Turkish, Chinese, Japanese, and German. Each language culture contains ten chapters: Festivals, Music, Religion and Beliefs, Flora and Fauna, Food, Clothing, Architecture, Agriculture, Tools, and Sports. Each chapter contains several to dozens of cultural concepts. For cultural adaptation, the " / r / IsA" function in ConceptNet can be used to query hypernyms in the target language, and the " / r / Synonym" function can be used to query synonyms in English. The "hyponyms()" function in WordNet is used to query hyponyms of English concepts. This embodiment considers at most third-order hypernyms. The sampling probability decays exponentially with path distance. For phrase level, this embodiment counts n-gram phrases, where n ranges from 2 to 5. This embodiment selects phrases related to reasoning and translates them into other languages ​​using Google Translate. For word level, this embodiment uses " / r / Synonym" in ConceptNet for querying. This embodiment trains the model on the NLVR2 dataset and tests it on MaRVL. To compare with the baseline, this embodiment uses almost the same experimental setup and hyperparameters, i.e., augmenting the dataset with code-transformation data. This embodiment also reduces training time while keeping the total number of training steps the same.

[0064] This embodiment evaluates four currently released multilingual multimodal pre-trained models: (1) mUNITER; (2) xUNITER; (3) M3P; and (4) UC2. This embodiment conducts two sets of experiments on mUNITER under a zero-shot setting: one set studies the impact of the code-switching data ratio, and the other set studies the impact of the total number of code-switching data on model performance when the ratio is constant. In Table 1, x:y represents the ratio of original English and code-switched data. For the first set of experiments, this embodiment sets four different ratios x:y = 1:1, 2:1, 3:1, and 4:1. The results in Table 1 show that, regardless of whether it is an English test dataset or other language test datasets, the smaller the ratio of code-switching data, the better the model performance under a zero-shot setting. To investigate the impact of the total number of code-switching data on the model performance with a constant ratio, this embodiment sets three different ratios x:y = 3:1, 9:3, and 15:5. Table 1 shows that the more code-switching data, the better the model performance. Based on the results of these two sets of experiments, this embodiment decides to expand the dataset to 20 times its original size and choose x:y = 16:4 as the configuration for subsequent zero-shot and few-shot experiments. In this setting, each code-switching data example corresponds to an average of 4 English data examples. This embodiment believes that, using images and partial English tags as anchors, the model will learn implicit alignments between English and other languages. A higher proportion of English data allows the model to focus on task learning, while a sufficient number of other language data examples are beneficial for implicit alignment.

[0065] Table 2 lists the zero-shot setting performance of the four models, showing that the method of this invention consistently outperforms the baseline statistically, significantly narrowing the gap between translation test performance and transmission performance. This proves the effectiveness of the method of this invention. Compared with the baseline, the method of this invention improves M3P and mUNITER by approximately 3–4 points, while UC2 and xUNITER only improve by 2–3 points. This may be attributed to UC2 ​​and xUNITER learning better aligned representations during the pre-training phase. Table 3 shows the sample performance results on three languages, indicating that the method of this invention also achieves state-of-the-art performance on the few-shot setting. To investigate the effect of the three-level code conversion method, this embodiment conducted an ablation study on mUNITER, selecting 16:4 as the experimental setting. Table 4 shows the statistical averages of different random seeds. The results show that all three methods are beneficial to improving model performance.

[0066] Table 1 compares model performance under different proportions of English data and code-switching.

[0067]

[0068] Table 2 shows the experimental results of the four models under the zero-sample setting.

[0069]

[0070] Table 3 shows the experimental results of the four models under a small sample setting.

[0071]

[0072] Table 4. Ablation experiment results of different levels of code mixing strategies.

[0073]

[0074] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A hierarchical code-mixing augmentation method based on cross-cultural adaptation, characterized in that, The method for augmenting language sample data from a source language culture comprises the following steps: S1: For all target languages ​​and cultures to be augmented, obtain the cultural concepts under each language and culture and form a cultural concept set; In S1, the language sample data is a visual reasoning task dataset. Each sample in the dataset contains a pair of images, a text statement describing the pair of images, and a label indicating whether the text statement correctly describes the pair of images. When using a hierarchical code-switching strategy for code conversion, the object of conversion is the text statement describing the pair of images in the sample. S2: Traverse each cultural concept in the cultural concept set, and for the current cultural concept that has been traversed, construct a cultural adaptation graph of the current cultural concept in the source language culture using the hypernym and hyponym relationship graph in the multilingual semantic network; S3: For each current cultural concept in the S2 traversal process, extract all other nodes at the same hierarchical level as the current cultural concept from the corresponding cultural adaptation graph. Take the cultural concepts represented by all the extracted nodes as the adaptation set of the current cultural concept in the source language culture. At the same time, set a confidence level for each cultural concept in the adaptation set. The closer the path distance between each cultural concept in the adaptation set and the current cultural concept in the cultural adaptation graph, the higher the confidence level set. S4: Using a hierarchical code-switching strategy, code-switching is performed on language sample data under the source language culture at the cultural concept level, phrase level, and word level in sequence to obtain augmented language samples that integrate concepts from other languages ​​and cultures, which are used to train multimodal and multilingual pre-trained models. In step S4, each step of obtaining an augmented language sample through code conversion using a hierarchical code mixing strategy is as follows: S41: Randomly sample one target language culture L from all target language cultures, then sample a cultural concept C1 from the set of cultural concepts in the target language culture L, and then sample a cultural concept C2 from the adaptation set CA of cultural concept C1 in the source language culture. When sampling cultural concept C2 from the adaptation set CA, the probability of each cultural concept in the set being sampled is positively correlated with its confidence level. S42: Retrieve language sample S1 containing cultural concept C2 from the language sample data under the source language culture, and then perform code mixing at the cultural concept level on language sample S1, replacing cultural concept C2 in language sample S1 with cultural concept C1 sampled in S41 to form language sample S2. S43: Perform phrase-level code mixing on the language sample S2 obtained in S42, sample phrases from the source language culture from the language sample S2, sample a target language culture, and replace each sampled phrase with a synonym of the phrase in the sampled target language culture to form language sample S3. S44: Perform word-level code mixing on the language sample S3 obtained in S43, sample words from the source language culture from the language sample S3, sample a target language culture, and replace each sampled word with a synonym of the word in the sampled target language culture to form the final augmented language sample S4.

2. The hierarchical code-mixing augmentation method based on cross-cultural adaptation as described in claim 1, characterized in that, In S1, the source language culture is English culture, and the target language culture is other languages ​​and cultures from different countries or regions.

3. The hierarchical code-mixing augmentation method based on cross-cultural adaptation as described in claim 1, characterized in that, In step S2, the specific steps for constructing a cultural adaptation graph for each traversed current cultural concept are as follows: S21: For the current cultural concept that has been traversed, based on the target language culture to which the current cultural concept belongs, query its multi-level superordinate words in the superordinate and hypoordinate word relationship graph of the multilingual semantic network. S22: For each hypernym found in S21, continue to search for the synonyms of the hypernym in the source language culture in the relation graph, and then search for all the hyponyms of each synonym in the source language culture. S23: The current cultural concept that will be traversed, as well as all the hypernyms, hyponyms, and synonyms found in S21 and S22, will be used as nodes of the graph. All nodes will be connected according to their word order relationships to form a cultural adaptation graph of the current cultural concept in the source language culture.

4. The hierarchical code-mixing augmentation method based on cross-cultural adaptation as described in claim 3, characterized in that, In S21, the multi-level hypernym is a third-level hypernym.

5. The hierarchical code-mixing augmentation method based on cross-cultural adaptation as described in claim 1, characterized in that, In S2, the multilingual semantic network uses ConceptNet and WordNet. In ConceptNet, " / r / IsA" is used to query hypernyms and " / r / Synonym" is used to query synonyms, while in WordNet, "hyponyms()" is used to query hyponyms.

6. The hierarchical code-mixing augmentation method for cross-cultural adaptation as described in claim 1, characterized in that, In step S43, before performing phrase-level code mixing, high-frequency phrases need to be selected from the language sample data of the source language culture to construct a source language phrase set, and each phrase in the source language phrase set is translated into other target language cultures to form synonyms. When performing phrase-level code mixing, it is necessary to determine whether there are any phrases in the language sample S2 that exist in the source language phrase set. If they exist, it is determined whether each phrase in the source language phrase set should be replaced according to a preset replacement probability. If there are phrases that need to be replaced, a target language culture is sampled and synonyms of the phrases that need to be replaced are found in this target language culture. Each phrase that needs to be replaced is replaced with the found synonyms.

7. The hierarchical code-mixing augmentation method for cross-cultural adaptation as described in claim 1, characterized in that, In step S44, before performing word-level code mixing, high-frequency words need to be selected from the language sample data of the source language culture to construct a source language word set, and each word in the source language word set is translated into other target language cultures to form synonyms. When performing word-level code mixing, it is necessary to determine whether there are any words in the language sample S2 that exist in the source language word set. If they exist, it is determined whether to replace each word in the source language word set according to a preset replacement probability. If there are words that need to be replaced, a target language culture is sampled and synonyms of the words to be replaced are found in this target language culture. Each word to be replaced is replaced with the found synonym.

8. A cross-cultural, cross-linguistic model training method, characterized in that, The multimodal, multilingual pre-trained model is trained on a dataset containing augmented language samples obtained by any of the augmentation methods described in claims 1 to 7, thereby obtaining a model with better cross-cultural and cross-linguistic capabilities.

9. The cross-cultural and cross-linguistic model training method as described in claim 8, characterized in that, The multimodal, multilingual pre-trained model is a mUNITER, xUNITER, M3P, or UC2 model.

10. The cross-cultural and cross-linguistic model training method as described in claim 9, characterized in that, The loss function used for training is cross-entropy loss.

Images