Spanish multimodal dictionary construction method and system based on large language model

Abstract

The present invention relates to the technical field of natural language processing and multimodal data processing, and in particular to a method and system for constructing a Spanish multimodal dictionary based on a large language model, which includes modules such as corpus processing, semantic stratification, cross-modal mapping, multimodal feature extraction, semantic alignment, and knowledge graph association. By calling a large language model to extract semantic features and cultural background information of entries, a semantic stratification set is generated, and semantic alignment and knowledge graph association are performed in combination with multimodal resources, ultimately achieving automatic generation of dictionary entries. The present invention can improve the efficiency of multimodal data integration, the depth of cultural semantic mining, and the accuracy of cross-modal alignment, providing Spanish learners with rich and accurate learning resources while reducing the cost of manual intervention.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing and multimodal data processing, and specifically relates to a method and system for constructing a Spanish multimodal dictionary based on a large language model. Background Art

[0002] With the rapid development of multimodal data processing and natural language processing technologies, knowledge representation and lexicon construction methods based on multimodal features have gradually become research hotspots in areas such as cross-language learning and cultural semantic mining. However, existing multimodal lexicon construction technologies have limitations in terms of automation, in-depth cultural semantic mining, and cross-modal alignment accuracy, and cannot fully meet the multimodal resource needs of learners of specific languages, such as Spanish.

[0003] Publication number CN106202281B discloses a multimodal data representation learning method and system. Through a graph random walk model and a data reconstruction model, the method fuses the feature representations of multimodal data to generate low-dimensional discriminant representations and dictionary representations. However, when processing multimodal data, this technical solution focuses more on data heterogeneity and missing data, and its ability to model cultural semantic information is relatively weak. Furthermore, the generation process of its dictionary representation does not fully utilize the semantic parsing capabilities of large language models, resulting in a lack of contextualized expression of dictionary entries, making it difficult to meet the needs of Spanish learners for cultural annotations and pragmatic scenarios.

[0004] The above issues indicate that existing multimodal dictionary construction technologies still have room for improvement in areas such as automated generation, cultural semantic mining, and cross-modal alignment accuracy. Therefore, the present invention provides a method and system for constructing a Spanish multimodal dictionary based on a large language model. This method aims to utilize the large language model to analyze the semantics and cultural context of terms, design a cross-modal alignment algorithm to associate semantic vectors of text and images / audio, and enhance the dictionary's contextualized expression through a cultural knowledge graph, thereby achieving efficient and accurate multimodal Spanish dictionary construction. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for constructing a Spanish multimodal dictionary based on a large language model, so as to solve the technical problems in the prior art, such as insufficient automatic association of multimodal data, limited depth of cultural semantic mining, and low cross-modal alignment accuracy, caused by the reliance of traditional Spanish dictionaries on manual compilation and fixed modal association methods.

[0006] In view of the above problems, the present invention provides a method and system for constructing a Spanish multimodal dictionary based on a large language model.

[0007] In a first aspect, the present invention provides a method for constructing a Spanish multimodal dictionary based on a large language model, wherein the method for constructing a Spanish multimodal dictionary based on a large language model is implemented by a Spanish multimodal dictionary construction system based on a large language model, wherein the method for constructing a Spanish multimodal dictionary based on a large language model comprises: inputting a target term and its original corpus, calling a pre-trained large language model to extract the semantic feature vector and cultural background information of the target term, and generating K term semantic sets; performing semantic hierarchical annotation on the K term semantic sets respectively, and determining K semantic hierarchical sets, wherein the K A semantic layer set includes K semantic layer labels and K context weight values; a cross-modal semantic mapping analysis is performed based on the K semantic layer labels and the K context weight values ​​to determine M modal association directions and M modal association weights; feature vectors of multimodal resources such as images, audio, and video are collected to generate M multimodal feature sets; semantic alignment is performed on the M multimodal feature sets based on the M modal association directions and the M modal association weights to generate M aligned feature sets; the M aligned feature sets are associated with cultural knowledge graph nodes to generate dictionary entries and store them in a database.

[0008] In a second aspect, the present invention further provides a Spanish multimodal dictionary construction system based on a large language model, which is used to execute the Spanish multimodal dictionary construction method based on a large language model as described in the first aspect, wherein the Spanish multimodal dictionary construction system based on a large language model includes: a corpus processing module, the corpus processing module is used to input the target term and its original corpus, call the pre-trained large language model to extract the semantic feature vector and cultural background information of the target term, and generate K term semantic sets; a semantic layering module, the semantic layering module is used to perform semantic layering annotation on the K term semantic sets respectively, and determine K semantic layering sets, wherein the K semantic layering sets include K semantic layering labels and K context weight values; modal mapping A mapping module, the modal mapping module is used to perform cross-modal semantic mapping analysis based on the K semantic hierarchical labels and the K context weight values, and determine M modal association directions and M modal association weights; a multimodal feature extraction module, the multimodal feature extraction module is used to collect feature vectors of multimodal resources such as images, audio, and video, and generate M multimodal feature sets; a semantic alignment module, the semantic alignment module is used to perform semantic alignment on the M multimodal feature sets based on the M modal association directions and the M modal association weights, and generate M aligned feature sets; a knowledge graph association module, the knowledge graph association module is used to associate the M aligned feature sets with cultural knowledge graph nodes, generate dictionary entries and store them in a database.

[0009] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0010] By inputting the target term and its original corpus, calling the pre-trained large language model to extract the semantic feature vector and cultural background information of the target term, K term semantic sets are generated; the K term semantic sets are semantically hierarchically annotated respectively to determine K semantic hierarchical sets, wherein the K semantic hierarchical sets include K semantic hierarchical labels and K contextual weight values; cross-modal semantic mapping analysis is performed based on the K semantic hierarchical labels and the K contextual weight values ​​to determine M modal association directions and M modal association weights; feature vectors of multimodal resources such as images, audio, and video are collected to generate M multimodal feature sets; semantic alignment is performed on the M multimodal feature sets based on the M modal association directions and the M modal association weights to generate M aligned feature sets; the M aligned feature sets are associated with cultural knowledge graph nodes to generate dictionary entries and store them in a database. In other words, by using a large language model to analyze the semantics and cultural background of terms, and combining cross-modal semantic mapping with knowledge graph association technology, efficient integration and precise alignment of multimodal data can be achieved, which improves the cultural semantic expression depth of the dictionary and the utilization rate of multimodal resources, reduces the cost of manual intervention, and provides Spanish learners with richer and more accurate learning resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 1 is an overall flow chart of the method of the present invention.

[0012] Figure 2 This is an architecture diagram of the system of the present invention.

[0013] Figure 3 Schematic diagram of the cross-modal semantic alignment algorithm

[0014] Figure 4 This is an example diagram of the cultural knowledge graph.

[0015] Figure 5 Schematic diagram of the output of each stage in the Spanish multimodal dictionary construction method based on a large language model. DETAILED DESCRIPTION

[0016] The present invention provides a method and system for constructing a Spanish multimodal dictionary based on a large language model. The core of the system is to achieve multimodal semantic analysis and precise alignment of Spanish terms by combining large language models, multimodal data processing and cultural knowledge graph technology. Figures 1 to 5 , describe the specific implementation methods of the present invention in detail.

[0017] In practice, the target term and its original corpus are first input and preprocessed through the corpus processing module. As the initial step in the system, the corpus processing module cleans and standardizes the externally input target terms to ensure the quality of the data for subsequent processing. On this basis, the corpus processing module calls on the pre-trained large language model to extract the semantic feature vector and cultural background information of the target term and generates a semantic set of K terms. The core of this process is to use the powerful semantic parsing capabilities of the large language model to extract multi-level semantic information and cultural background-related content from the original corpus. For example, if the target term is "fiesta", the corpus processing module will extract the semantic features of the word in different contexts, such as festivals and social gatherings, and combine the cultural background information to annotate its special meaning in Spanish culture.

[0018] Next, the semantic stratification module performs semantic stratification annotation on the K semantic sets of terms generated by the corpus processing module to determine K semantic stratification sets. Each semantic stratification set contains K semantic stratification labels and K contextual weight values. The semantic stratification module automatically assigns contextual weight values ​​by analyzing the frequency of use and contextual relationships of terms in different contexts, thereby achieving a hierarchical division of the semantics of terms. For example, "fiesta" may be assigned a higher weight value in a religious context, but a lower weight value in daily social scenarios. The semantic stratification module and the corpus processing module are directly connected through a data stream. The semantic feature vectors and cultural background information output by the corpus processing module are the basic data source for the semantic stratification module to perform annotation.

[0019] Subsequently, the modal mapping module performs cross-modal semantic mapping analysis based on the K semantic layer labels and K contextual weight values ​​generated by the semantic layer module to determine M modal association directions and M modal association weights. The modal mapping module generates specific modal association directions and weight values ​​by calculating the association strength between the semantic layer labels and multimodal resources. For example, for the word "fiesta", the modal mapping module may identify a strong association with the celebration scene in the image resource, the music sound effects in the audio resource, and the dance movement in the video resource, and generate the corresponding modal association direction and weight value accordingly. The modal mapping module is connected to the semantic layer module through a data interface, receiving the semantic layer labels and contextual weight values ​​as input, and passing the analysis results to the multimodal feature extraction module.

[0020] The multimodal feature extraction module is responsible for collecting feature vectors from multimodal resources such as images, audio, and video, and generating M multimodal feature sets. The multimodal feature extraction module extracts features from multimodal resources by invoking a pre-trained deep learning model. For example, when processing images related to "fiesta," the multimodal feature extraction module extracts feature vectors such as color distribution and character posture; when processing audio resources, it extracts feature vectors such as musical rhythm and pitch changes. The multimodal feature extraction module is connected to the modal mapping module via a data pipeline. The modal association directions and weight values ​​output by the modal mapping module guide the multimodal feature extraction module in selecting appropriate resources for feature extraction.

[0021] Based on the M modal association directions and M modal association weights generated by the modal mapping module, the semantic alignment module semantically aligns the M multimodal feature sets generated by the multimodal feature extraction module to generate M aligned feature sets. The semantic alignment module determines the optimal alignment scheme by calculating the similarity between the multimodal feature vector and the semantic feature vector. For example, for the image resource of "fiesta", the semantic alignment module may prioritize images containing features such as crowds and lighting decorations for alignment. The semantic alignment module is connected to the multimodal feature extraction module via a data stream, receiving the multimodal feature set as input and passing the alignment results to the knowledge graph association module.

[0022] The knowledge graph association module associates the M aligned feature sets generated by the semantic alignment module with the cultural knowledge graph nodes, generating dictionary entries that are stored in the database. The knowledge graph association module queries the cultural knowledge graph to find cultural background nodes related to the target term and links the aligned feature sets to these nodes. For example, for the word "fiesta," the knowledge graph association module might associate it with cultural nodes such as "La Tomatina" or "Semana Santa," a traditional Spanish festival, to form a complete dictionary entry. The knowledge graph association module is connected to the semantic alignment module via a data interface, receiving the aligned feature sets as input and storing the generated dictionary entries in the database.

[0023] During the entire system operation, the functional modules work closely together through clear data flows and connection relationships. The corpus processing module, as the starting point, is responsible for inputting the target terms and their original corpus, and generating semantic feature vectors and cultural background information; the semantic stratification module receives the output of the corpus processing module and performs semantic stratification annotation; the modal mapping module analyzes cross-modal semantic mapping based on the output of the semantic stratification module; the multimodal feature extraction module extracts feature vectors of multimodal resources based on the results of the modal mapping module; the semantic alignment module performs semantic alignment on the multimodal feature set; finally, the knowledge graph association module associates the aligned feature set with the cultural knowledge graph node, generates a dictionary entry and stores it in the database. The entire process is as follows: Figure 1 As shown, the complete steps from corpus input to dictionary entry generation are clearly demonstrated.

[0024] also, Figure 2 The system architecture diagram is presented, clarifying the connections between the various functional modules and the data flow. The corpus processing module, semantic layering module, modality mapping module, multimodal feature extraction module, semantic alignment module, and knowledge graph association module are interconnected through data pipelines, forming a closed-loop data processing chain. Figure 3 The implementation process of the cross-modal semantic alignment algorithm is further described in detail, including the alignment logic of semantic feature vectors and multimodal feature vectors and the application of modal association directions and weights. Figure 4 It shows the organizational form of the cultural knowledge graph, and reflects the hierarchical expression of cultural background information in the knowledge graph and the association logic between nodes. Figure 5 It presents the specific output form of each stage, from the generation of semantic sets to the final dictionary entry storage.

[0025] Through the above-mentioned specific implementation methods, the present invention realizes the automated construction of a Spanish multimodal dictionary based on a large language model, solves the technical problems caused by the traditional dictionary's reliance on manual compilation and fixed modal association methods, improves the integration efficiency and semantic alignment accuracy of multimodal data, and provides Spanish learners with richer and more accurate learning resources.

[0026] In order to better enable relevant personnel in this technical field to fully understand and implement the present invention, the operating principle and implementation steps of the present invention are supplemented below with reference to a specific application scenario.

[0027] When constructing a Spanish multimodal dictionary, the target term "fiesta" and its original corpus are first input through the corpus processing module. The corpus processing module cleans and standardizes the external input corpus, removes irrelevant characters, unifies the format, and calls a pre-trained large language model to extract the semantic feature vector and cultural background information of the target term. For example, for "fiesta", the large language model will analyze its semantic features in different contexts, such as festivals and social gatherings, and annotate its special meaning in Spanish culture, such as "religious festivals" or "folk activities." This process is based on the context-awareness of the large language model. By learning from large-scale corpus, a multi-level semantic set K is generated, each corresponding to a specific semantic dimension. This step realizes the automated extraction of semantic features from the original corpus, laying the foundation for subsequent processing.

[0028] Next, the semantic stratification module receives the K semantic sets of terms output by the corpus processing module and performs semantic stratification annotation on them. For example, for the word "fiesta", the semantic stratification module analyzes its frequency of use and contextual relationships in different contexts, assigning a higher contextual weight value to "religious festival" and a lower weight value to "daily gathering". This process automatically generates a semantic stratification set K by calculating the similarity and contextual relevance between semantic labels. Each set contains a corresponding semantic stratification label and contextual weight value. The core of the semantic stratification module is to use statistical methods and semantic analysis algorithms to ensure the accuracy and rationality of semantic hierarchy division, thereby improving the accuracy of subsequent cross-modal mapping.

[0029] Subsequently, the modal mapping module performs cross-modal semantic mapping analysis based on the K semantic layer labels and K contextual weight values ​​generated by the semantic layer module. For example, for the "religious festival" semantic label of "fiesta", the modal mapping module calculates the strength of its association with the celebratory scenes in the image resources, the religious music in the audio resources, and the ritual actions in the video resources, and generates the corresponding modal association direction and weight value. By introducing the attention mechanism, the modal mapping module dynamically adjusts the association weights of different modal resources to ensure that highly relevant modal resources are selected first. This process achieves accurate mapping from semantic layering to multimodal resources, providing a clear direction for subsequent feature extraction.

[0030] The multimodal feature extraction module collects feature vectors of multimodal resources such as images, audio, and video based on the M modal association directions and weight values ​​generated by the modal mapping module. For example, when processing image resources related to "fiesta," the multimodal feature extraction module calls a pre-trained deep learning model to extract features such as color distribution and character posture in the image; when processing audio resources, it extracts features such as musical rhythm and pitch changes. The multimodal feature extraction module uses a feature extraction algorithm to generate M multimodal feature sets, each of which corresponds to a feature vector for a specific modal resource. The core of this process is to leverage the efficient feature extraction capabilities of deep learning models to ensure that the feature representation of multimodal data is highly discriminative and representative.

[0031] The semantic alignment module receives the M multimodal feature sets generated by the multimodal feature extraction module and performs semantic alignment based on the M modal association directions and weight values ​​generated by the modal mapping module. For example, for the image resource "fiesta", the semantic alignment module calculates the similarity between the image feature vector and the semantic feature vector, giving priority to images containing features such as crowds and lighting decorations for alignment. The semantic alignment module ensures the optimal alignment of multimodal features with semantic features by introducing the cosine similarity algorithm and weighted matching strategy. This process achieves precise matching from multimodal features to semantic features, significantly improving the accuracy of cross-modal alignment.

[0032] The knowledge graph association module associates the M aligned feature sets generated by the semantic alignment module with cultural knowledge graph nodes. For example, for the word "fiesta," the knowledge graph association module queries the cultural knowledge graph to find relevant cultural context nodes, such as "La Tomatina" or "Semana Santa," and links the aligned feature set to these nodes. Using graph database query technology, the knowledge graph association module quickly locates relevant cultural nodes and associates them with the aligned feature set to form a complete dictionary entry. This process enables in-depth mining of cultural context information from multimodal features, enhancing the dictionary's ability to express cultural semantics.

[0033] Throughout the system's operation, the various functional modules collaborate closely through clear data flows. The corpus processing module takes in the target terms and their original corpus, generating semantic feature vectors and cultural context information. The semantic stratification module performs hierarchical annotation on the received semantic sets. The modality mapping module analyzes cross-modal semantic mappings based on the semantic stratification results. The multimodal feature extraction module extracts feature vectors of multimodal resources based on the modality mapping results. The semantic alignment module semantically aligns multimodal feature sets. Finally, the knowledge graph association module associates the aligned feature sets with cultural knowledge graph nodes, generates dictionary entries, and stores them in the database.

[0034] Through the above-described specific implementations, the present invention achieves the automated construction of a Spanish multimodal dictionary based on a large language model. Each module, through specific technical means, addresses the technical issues inherent in traditional dictionaries, which rely on manual compilation and fixed modal associations. This significantly improves the efficiency of multimodal data integration and the accuracy of semantic alignment, providing Spanish learners with richer and more accurate learning resources.

[0035] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0036] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A Spanish multimodal dictionary construction method based on a large language model, characterized in that: include: Input the target term and its original corpus, call the pre-trained large language model to extract the semantic feature vector and cultural background information of the target term, and generate K term semantic sets; Performing semantic hierarchical annotation on the K semantic sets of terms respectively to determine K semantic hierarchical sets, wherein the K semantic hierarchical sets include K semantic hierarchical labels and K context weight values; Performing cross-modal semantic mapping analysis based on the K semantic layer labels and the K context weight values ​​to determine M modal association directions and M modal association weights; Collect feature vectors of multimodal resources including images, audio, and video to generate M multimodal feature sets; Semantically aligning the M multimodal feature sets based on the M modal association directions and the M modal association weights to generate M aligned feature sets; The M alignment feature sets are associated with cultural knowledge graph nodes to generate dictionary entries and store them in a database.

2. The method for constructing a Spanish multimodal dictionary based on a large language model according to claim 1, wherein: include: Traversing the K semantic sets of terms to perform semantic distribution calculation and generate K semantic distribution values; Analyze the K semantic distribution values ​​and a preset distribution threshold to determine K context weight values; Distributing the K context weight values ​​and the K term semantic sets to K semantic layer channels to generate the K semantic layer labels; The K semantic layer labels are added to the K semantic layer sets.

3. The method for constructing a Spanish multimodal dictionary based on a large language model according to claim 1, wherein: include: Performing semantic hierarchical division on the K semantic sets of terms according to a hierarchical structure to generate K semantic hierarchical identifiers; Assigning weights to the K semantic level identifiers respectively to generate K context weight values; The K context weight values ​​are added to the K semantic layer sets.

4. The method for constructing a Spanish multimodal dictionary based on a large language model according to claim 2, wherein: include: Calling the K semantic spaces of the K semantic layer channels respectively; Inputting the K semantic sets of terms into the K semantic spaces to obtain K semantic scatter point sets; Randomly extracting n semantic scattered points from the K semantic scattered point sets as K semantic center sets; Iteratively searching the K semantic center sets in the K semantic spaces according to the K semantic distribution values ​​until a preset number of iterations or a preset search stop constraint is satisfied, thereby obtaining K iterative semantic center sets; Screening the K iterative semantic center sets to generate K target semantic centers; The K semantic labels corresponding to the K target semantic centers are used as the K semantic hierarchical labels.

5. The method for constructing a Spanish multimodal dictionary based on a large language model according to claim 4, wherein: include: In the K semantic spaces, the K semantic center sets are respectively randomly moved in directions according to the K semantic distribution values ​​to obtain K stage semantic center sets; Determine whether the K stage clustering density sets of the K stage semantic center sets are greater than or equal to the K clustering density sets of the K semantic center sets. If so, take the K stage semantic center set iterations as the starting point, and perform search iterations in the K semantic spaces according to the K semantic distribution values ​​until a preset number of iterations or a preset search stop constraint is met to obtain the K iterative semantic center sets.

6. The method for constructing a Spanish multimodal dictionary based on a large language model according to claim 4, wherein: The preset search stop constraint is that the difference in cluster density between two adjacent search iterations is less than or equal to a preset difference threshold.

7. The method for constructing a Spanish multimodal dictionary based on a large language model according to claim 1, wherein: include: Performing feature adjustment on the M multimodal feature sets based on the M modal association directions and the M modal association weights to obtain M adjusted feature sets; Taking a preset semantic similarity interval and a preset cultural relevance interval as alignment targets, and combining the K semantic hierarchical labels, performing fitness identification on the M adjustment feature sets to obtain M adjustment fitnesses; When the M adjustment fitnesses satisfy a preset fitness, the M adjustment feature sets are used as the M alignment feature sets.

8. A Spanish multimodal dictionary construction system based on a large language model, configured to implement the steps of the Spanish multimodal dictionary construction method based on a large language model according to any one of claims 1 to 7, characterized in that: include: A corpus processing module is used to input the target term and its original corpus, call the pre-trained large language model to extract the semantic feature vector and cultural background information of the target term, and generate K term semantic sets; A semantic layering module, configured to perform semantic layering annotation on each of the K semantic sets of terms to determine K semantic layering sets, wherein the K semantic layering sets include K semantic layering labels and K context weight values; A modality mapping module, configured to perform cross-modality semantic mapping analysis based on the K semantic layer labels and the K context weight values ​​to determine M modality association directions and M modality association weights; A multimodal feature extraction module, which is used to collect feature vectors of multimodal resources such as images, audio, and video to generate M multimodal feature sets; a semantic alignment module, configured to semantically align the M multimodal feature sets based on the M modal association directions and the M modal association weights to generate M aligned feature sets; The knowledge graph association module is used to associate the M alignment feature sets with cultural knowledge graph nodes, generate dictionary entries and store them in a database.

9. The Spanish multimodal dictionary construction system based on a large language model according to claim 8, characterized in that: The corpus processing module further includes: The standardization submodule is used to clean and format the target terms and their original corpus; The semantic parsing submodule is used to call the pre-trained large language model to extract the semantic feature vector and cultural background information of the target term.

10. The Spanish multimodal dictionary construction system based on a large language model according to claim 8, characterized in that: The semantic layering module further includes: A semantic distribution calculation submodule, configured to calculate the semantic distribution values ​​of the K semantic sets of terms; The context weight assignment submodule is used to generate K context weight values ​​according to K semantic distribution values.

Images