Multi-modal language feature analysis method and system

Abstract

The invention discloses a multi-modal language feature analysis method and system, belongs to the technical field of natural language processing and data mining, and can enable southern Hunan historical and cultural resources to be more efficiently researched and utilized. Comprising the following steps: collecting multi-modal data of southern Hunan historical culture resources; performing cleaning operation on the acquired multi-modal data, and removing redundant data with cosine similarity higher than a set threshold value by using a preset deduplication algorithm; key words are extracted from the cleaned multi-modal data; calculating a correlation degree between each keyword and a preset historical and cultural subject term to screen out a core keyword; the relevance between the extracted core keywords is analyzed, and a data relation graph is constructed; calculating the node weight of each node in the data relation graph; screening the data based on the node weight, and screening out high-relevance data conforming to a set standard; and constructing a knowledge graph of the southern Hunan historical and cultural resources on the basis of the high-relevance data conforming to the set standard.

Description

Technical Field

[0001] The present invention relates to the technical fields of natural language processing and data mining, and in particular to a multimodal language feature analysis method and system. Background Art

[0002] Southern Hunan's historical and cultural resources are an integral part of Chinese history and culture, serving as a crucial vehicle for promoting patriotism and inheriting the spirit. With relevant departments vigorously promoting cultural heritage and development, southern Hunan's historical and cultural resources, with their rich historical heritage and unique regional characteristics, have become a key area for research and utilization. However, the acquisition and analysis of these resources still present numerous technical and practical difficulties.

[0003] On the one hand, the historical and cultural resources of southern Hunan are widely distributed, coming from a variety of sources, including documentary materials, video records, physical remains, oral histories, and so on. Information from these different sources is often scattered across different platforms and institutions. Collecting and organizing these resources requires researchers to invest considerable time and effort. This inefficient resource acquisition method limits the progress of academic research, cultural dissemination, and educational practice.

[0004] On the other hand, the historical and cultural resources of southern Hunan are diverse in form and complex in content, mixed with a large amount of redundant, repetitive, and even erroneous information, making it difficult for users to quickly access valuable content. This disorganized information not only increases the cost of resource screening and organization, but also hinders users from deeply utilizing historical and cultural resources and fully exploring their cultural connotations and value.

[0005] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of this application. Summary of the Invention

[0006] This application provides a multimodal language feature analysis method and system, which can enable the historical and cultural resources of southern Hunan to be studied and utilized more efficiently.

[0007] To achieve the above objectives, the present application discloses the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a multimodal language feature analysis method, comprising the following steps:

[0009] Use web crawler technology to collect multimodal data of historical and cultural resources in southern Hunan from pre-set data sources;

[0010] The acquired multimodal data is cleaned, and the data is vectorized using a preset deduplication algorithm. The cosine similarity between vectors is calculated, and redundant data with a cosine similarity higher than a set threshold is removed.

[0011] Extract keywords from the cleaned multimodal data;

[0012] Calculate the relevance of each keyword to the preset historical and cultural keywords to screen out the core keywords;

[0013] Analyze the correlation between each extracted core keyword and construct a data relationship diagram;

[0014] Calculate the node weight of each node in the data relationship graph. The node weight is used to indicate the importance of the node in the data relationship graph.

[0015] Filter data based on node weights to select highly relevant data that meets the set standards;

[0016] Based on highly correlated data that meets the set standards, a knowledge graph of southern Hunan’s historical and cultural resources is constructed.

[0017] In the embodiment of the present application, web crawler technology is used to automatically collect data from multiple preset data sources, effectively solving the problem of wide distribution and inefficient acquisition of historical and cultural resources in southern Hunan, reducing the time and energy of researchers in manually collecting resources, and improving resource acquisition efficiency.

[0018] During the data cleaning step, the solution employs a deduplication algorithm to identify and remove duplicate or redundant data by vectorizing the data and calculating cosine similarity between the vectors. Southern Hunan's historical and cultural resources are diverse and sourced from various sources, so duplicate and low-quality content is inevitable. By calculating cosine similarity, highly similar data is eliminated, retaining only content with unique value. This resolves the issue of information redundancy and duplication, reduces the interference of redundant information on subsequent processing, and alleviates the burden of information sifting for users faced with such a large volume of data.

[0019] By using a series of operations such as keyword extraction, correlation calculation, data relationship diagram construction, node weight calculation and data screening, and knowledge graph construction, valuable core information can be extracted from complex data, ensuring that the content received by users is core data with high research value after multiple screenings. This provides high-quality materials for the in-depth research and promotion of southern Hunan's historical and cultural resources, and facilitates users to make in-depth use of them and explore their cultural connotations and values. In this way, southern Hunan's historical and cultural resources can be studied and utilized more efficiently, improving the efficiency of obtaining southern Hunan's historical and cultural resources, saving researchers' time and energy, and promoting the progress of academic research, cultural communication, and educational practice.

[0020] In some possible implementations of the first aspect, the step of extracting keywords from the cleaned multimodal data includes:

[0021] For text data, we use natural language processing technology, lexical analysis, syntactic analysis, and semantic analysis tools, combined with stop word lists to remove meaningless words, and extract keywords based on the word frequency-inverse document frequency algorithm or the TextRank algorithm;

[0022] For video data, we first use video keyframe extraction technology to obtain keyframe images, then perform text recognition on the keyframe images, convert the recognized text into text data, and then operate according to the method of extracting keywords from text data; at the same time, we use video content analysis technology to extract scene and action feature information in the video, and convert this feature information into semantic information as supplementary keywords;

[0023] For image data, we use optical character recognition technology to identify the text content in the image and use the recognized text as text data for keyword extraction. In addition, we use image feature extraction algorithms to extract the visual features of the image, convert the visual features into semantic descriptions, and then extract relevant keywords.

[0024] For audio data, we first perform speech recognition and convert the audio into text format. Then, we apply the same keyword extraction method as for text data. This allows us to fully explore the key information in each modality of data, allowing the extracted keywords to more comprehensively and accurately reflect the data content, improving the accuracy and effectiveness of data processing and facilitating the subsequent in-depth analysis and utilization of historical and cultural resources.

[0025] In some possible implementations of the first aspect, the calculation formula for the correlation between each keyword and the preset historical and cultural theme word is:

[0026]

[0027] Where:

[0028] is the vector of keywords, is the vector of historical and cultural keywords;

[0029] f k is the frequency of the keyword, is the average frequency of all keywords in the document;

[0030] α and β are weight coefficients, and satisfy α+β=1.

[0031] Among them, by introducing the vector of historical and cultural keywords It can focus on a specific cultural background more efficiently according to the set subject scope, avoid processing irrelevant content, and improve the accuracy of extraction. By introducing the second word frequency ratio, the importance weight of keywords can be adjusted dynamically. Specifically, unlike the traditional method that only uses semantic similarity, this method can also consider the frequency of occurrence of keywords to ensure that in diverse data scenarios, high-frequency important words can be highlighted, and low-frequency but valuable keywords are not ignored. For example, in historical and cultural resources, the keyword "historical migration" may have a higher word frequency, while the keyword "Xiangnan Battle" may have a lower word frequency. Through the formula, although the former has a high word frequency, the weight is only increased within a reasonable range, while the latter can also obtain appropriate weight due to its high semantic relevance to the subject word.

[0032] In some possible implementations of the first aspect, the node weight is calculated using the following formula:

[0033] NodeWeigt=Rel·log(1+f k )

[0034] Where:

[0035] Rel is the keyword relevance calculated by the formula in claim 4;

[0036] f k is the total frequency of the keyword appearing in the entire data set;

[0037] log(1+f k ) is used to smooth the frequency values of high-frequency and low-frequency keywords.

[0038] In this way, the semantic relevance and word frequency correction value of keywords are combined to comprehensively evaluate the importance of keywords from two dimensions. Among them, the relevance of keywords ensures semantic relevance, and the introduction of natural logarithms avoids the unreasonable expansion of the weights of high-frequency words. Specifically, in historical and cultural resources, the weights of words with high word frequency but average semantic relevance (such as "meeting") will be adjusted to a reasonable range by the natural logarithm; while the weights of words with low word frequency but high semantic relevance will be increased. In this way, it will help to screen out data that is truly highly relevant and important, and improve the quality of data processing and the accuracy of knowledge graph construction.

[0039] In some possible implementations of the first aspect, the step of screening data based on node weights to select highly correlated data that meets set criteria includes:

[0040] Calculate the sum of all keyword node weights in each data.

[0041] The total weight value is compared with the set weight threshold, and data records exceeding the preset weight threshold are retained, while data records below the preset weight threshold are removed. In this way, highly relevant data that meets the set standards can be quickly and accurately screened from a large amount of data, further improving data quality, providing a guarantee for building an accurate and valuable knowledge map of southern Hunan's historical and cultural resources, and facilitating users to obtain more valuable information.

[0042] In some possible implementations of the first aspect, the data relationship graph is constructed by the following steps:

[0043] Use core keywords as nodes;

[0044] By counting the co-occurrence frequency of two keywords in the same data record, if the co-occurrence frequency is higher than the set threshold, an edge is established between the two nodes;

[0045] The weight of an edge is determined by co-occurrence frequency or semantic similarity.

[0046] In some possible implementations of the first aspect, the deduplication algorithm is a DeduplicationNet model based on a twin neural network. Compared to traditional deduplication methods, this model can more accurately capture similarities between data, improve deduplication accuracy, more effectively remove redundant data, and enhance data quality.

[0047] In some possible implementations of the first aspect, a distributed crawler system is constructed based on the Scrapy framework through distributed crawler technology, multiple crawler nodes are configured, and a crawling task range is set for each node, and different data sources are accessed respectively. The data sources include but are not limited to the cultural websites of relevant departments, the official websites of local museums, and academic databases, and the historical and cultural resource data of southern Hunan covering documentary materials, image records, descriptions of physical remains, and oral history are collected. In this way, multiple nodes can access different data sources at the same time, realizing the parallelization of data collection. In this way, network resources and computing resources can be fully utilized to improve the efficiency of data collection.

[0048] In some possible implementations of the first aspect, the steps of constructing a knowledge graph of historical and cultural resources in southern Hunan based on highly relevant data that meets set standards include:

[0049] Analyze the composition and internal relationships of highly relevant data that meet the set standards and build a knowledge representation model for historical and cultural resources;

[0050] Clearly and formally express the concepts, attributes, and relationships between concepts required to construct the knowledge graph, form a knowledge description specification, cover the definitions of concepts, attributes, and relationships between concepts, construct the model layer of the knowledge graph, and complete the abstract description of historical and cultural resource knowledge;

[0051] Based on the knowledge representation model of the knowledge graph pattern layer, build entity recognition, relationship extraction and attribute acquisition models;

[0052] Use entity recognition, relationship extraction and attribute acquisition models to carry out entity recognition, attribute acquisition and relationship extraction between entities, and then build the data layer of the historical and cultural resource knowledge graph;

[0053] The historical and cultural resource knowledge obtained in the knowledge acquisition process is integrated, stored in a graph database, and visualized to obtain a knowledge graph of southern Hunan’s historical and cultural resources.

[0054] In a second aspect, an embodiment of the present application provides a multimodal language feature analysis system, comprising:

[0055] The first acquisition module is used to collect multimodal data of historical and cultural resources in southern Hunan from a preset data source using web crawler technology;

[0056] The first cleaning module is used to clean the acquired multimodal data by using a preset deduplication algorithm to vectorize the data and calculate the cosine similarity between vectors, thereby removing redundant data with a cosine similarity higher than a set threshold;

[0057] The first extraction module is used to extract keywords from the cleaned multimodal data;

[0058] The first screening module is used to calculate the relevance of each keyword with the preset historical and cultural keywords to screen out the core keywords;

[0059] The first construction module is used to analyze the correlation between each extracted core keyword and construct a data relationship graph;

[0060] A first calculation module is used to calculate the node weight of each node in the data relationship graph, where the node weight is used to represent the importance of the node in the data relationship graph;

[0061] The second screening module is used to screen the data based on the node weight and select the highly relevant data that meets the set standards;

[0062] The second construction module is used to construct a knowledge graph of southern Hunan’s historical and cultural resources based on highly correlated data that meets the set standards.

[0063] In some possible implementations of the second aspect, the first cleaning module is specifically used to: for text data, use natural language processing technology, apply lexical analysis, syntactic analysis and semantic analysis tools, combine with stop word lists to remove meaningless words, and extract keywords based on word frequency-inverse document frequency algorithm or TextRank algorithm; for video data, first obtain key frame images through video key frame extraction technology, then perform text recognition processing on the key frame images, convert the recognized text into text data, and then operate according to the method of extracting keywords from text data; at the same time, use video content analysis technology to extract scene and action feature information in the video, and perform semantic conversion of these feature information as supplementary keywords; for image data, use optical character recognition technology to identify the text content in the image, and use the recognized text as text data for keyword extraction; in addition, use image feature extraction algorithm to extract visual features of the image, convert the visual features into semantic descriptions, and then extract relevant keywords; for audio data, first perform speech recognition, convert the audio into text format, and then use the same keyword extraction method as text data.

[0064] In some possible implementations of the second aspect, the first screening module is specifically configured to calculate the relevance of each keyword to a preset historical and cultural keyword using the following formula:

[0065]

[0066] Where:

[0067] is the vector of keywords, is the vector of historical and cultural keywords;

[0068] f k is the frequency of the keyword, is the average frequency of all keywords in the document;

[0069] α and β are weight coefficients, and satisfy α+β=1.

[0070] In some possible implementations of the second aspect, the first calculation module is specifically configured to calculate the node weight using the following formula:

[0071] NodeWeigt=Rel·log(1+f k )

[0072] Where:

[0073] Rel is the keyword relevance calculated by the formula in claim 4;

[0074] f k is the total frequency of the keyword appearing in the entire data set;

[0075] log(1+f k ) is used to smooth the frequency values of high-frequency and low-frequency keywords.

[0076] In some possible implementations of the second aspect, the second screening module is specifically used to: calculate the sum of the weights of all keyword nodes in each data, compare the total weight value with the set weight threshold, retain data records that exceed the preset weight threshold, and remove data records that are lower than the preset weight threshold.

[0077] In some possible implementations of the second aspect, the first construction module is specifically used to: take the core keywords as nodes; by counting the co-occurrence frequency of two keywords in the same data record, if the co-occurrence frequency is higher than a set threshold, an edge is established between the two nodes; the weight of the edge is determined by the co-occurrence frequency or semantic similarity.

[0078] In some possible implementations of the second aspect, the deduplication algorithm is a DeduplicationNet model based on a twin neural network.

[0079] In some possible implementations of the second aspect, the first acquisition module is specifically used to build a distributed crawler system based on the Scrapy framework through distributed crawler technology, configure multiple crawler nodes, and set the crawling task scope for each node, and access different data sources respectively. The data sources include but are not limited to cultural websites of relevant departments, official websites of local museums and academic databases, and collect historical and cultural resource data in southern Hunan covering documentary materials, image records, descriptions of physical relics, and oral history.

[0080] In some possible implementations of the second aspect, the second construction module is specifically used to: analyze the composition and internal correlation of highly correlated data that meets the set standards, and construct a historical and cultural resource knowledge representation model; clearly formalize the concepts, attributes and relationships between concepts required to construct the knowledge graph, form a knowledge description specification, cover the definitions of concepts, attributes and relationships between concepts, construct a model layer of the knowledge graph, and complete the abstract description of historical and cultural resource knowledge; construct entity recognition, relationship extraction and attribute acquisition models based on the knowledge representation model of the knowledge graph model layer; use entity recognition, relationship extraction and attribute acquisition models to carry out entity recognition, attribute acquisition and entity relationship extraction, and then construct the data layer of the historical and cultural resource knowledge graph; integrate the historical and cultural resource knowledge obtained in the knowledge acquisition link, store it in a graph database, and visualize it to obtain a knowledge graph of southern Hunan's historical and cultural resources.

[0081] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors; a storage device on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any technical solution of the first aspect.

[0082] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described in any technical solution of the first aspect is implemented.

[0083] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in any technical solution of the first aspect.

[0084] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0086] Figure 1 A flowchart of a multimodal language feature analysis method provided in some embodiments of the present application;

[0087] Figure 2 A schematic diagram of the process of constructing a knowledge graph of historical and cultural resources in southern Hunan, provided for some embodiments of this application;

[0088] Figure 3 A schematic diagram of the structure of a multimodal language feature analysis system provided in some embodiments of the present application;

[0089] Figure 4 It is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present application. DETAILED DESCRIPTION

[0090] Specific embodiments of the present invention will now be mentioned in detail. Although the present invention is described in conjunction with these specific embodiments, it should be appreciated that the present invention is not intended to be limited to these specific embodiments. On the contrary, these embodiments are intended to cover substitutions, changes, or equivalent embodiments that may be included within the spirit and scope of the invention defined by the claims. In the following description, a large number of specific details are set forth in order to provide a comprehensive understanding of the present invention. The present invention may be implemented without some or all of these specific details.

[0091] When used in conjunction with "including," "methods comprising," or similar language in this specification and the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0092] Application Overview: Southern Hunan's historical and cultural resources are an integral part of Chinese history and culture, serving as a crucial vehicle for promoting patriotism and inheriting the spirit. With relevant departments vigorously promoting cultural heritage and development, Southern Hunan's historical and cultural resources, with their rich historical heritage and unique regional characteristics, have become a key area for research and utilization. However, the acquisition and analysis of Southern Hunan's historical and cultural resources currently face numerous technical and practical difficulties.

[0093] On the one hand, the historical and cultural resources of southern Hunan are widely distributed, coming from a variety of sources, including documentary materials, video records, physical remains, oral histories, and so on. Information from these different sources is often scattered across different platforms and institutions. Collecting and organizing these resources requires researchers to invest considerable time and effort. This inefficient resource acquisition method limits the progress of academic research, cultural dissemination, and educational practice.

[0094] On the other hand, the historical and cultural resources of southern Hunan are diverse in form and complex in content, mixed with a large amount of redundant, repetitive, and even erroneous information, making it difficult for users to quickly access valuable content. This disorganized information not only increases the cost of resource screening and organization, but also hinders users from deeply utilizing historical and cultural resources and fully exploring their cultural connotations and value.

[0095] In response to the above technical problems, the overall idea of the technical solution provided by this application is as follows: a multimodal language feature analysis method is provided, comprising the following steps: using web crawler technology to collect multimodal data of historical and cultural resources in southern Hunan from a preset data source; performing a cleaning operation on the acquired multimodal data, using a preset deduplication algorithm to vectorize the data, and then calculating the cosine similarity between vectors, and removing redundant data with a cosine similarity higher than a set threshold; extracting keywords from the cleaned multimodal data; calculating the correlation between each keyword and the preset historical and cultural keywords to screen out core keywords; analyzing the correlation between each extracted core keyword and constructing a data relationship graph; calculating the node weight of each node in the data relationship graph, and the node weight is used to indicate the importance of the node in the data relationship graph; screening the data based on the node weight to screen out high-correlation data that meets the set standards; and constructing a knowledge graph of historical and cultural resources in southern Hunan based on the high-correlation data that meets the set standards.

[0096] This method uses web crawler technology to automatically collect data from multiple preset data sources, effectively solving the problem of wide distribution and inefficient acquisition of historical and cultural resources in southern Hunan, reducing the time and energy researchers spend on manually collecting resources, and improving resource acquisition efficiency.

[0097] During the data cleaning step, the solution employs a deduplication algorithm to identify and remove duplicate or redundant data by vectorizing the data and calculating cosine similarity between the vectors. Southern Hunan's historical and cultural resources are diverse and sourced from various sources, so duplicate and low-quality content is inevitable. By calculating cosine similarity, highly similar data is eliminated, retaining only content with unique value. This resolves the issue of information redundancy and duplication, reduces the interference of redundant information on subsequent processing, and alleviates the burden of information sifting for users faced with such a large volume of data.

[0098] By using a series of operations such as keyword extraction, correlation calculation, data relationship diagram construction, node weight calculation and data screening, and knowledge graph construction, valuable core information can be extracted from complex data, ensuring that the content received by users is core data with high research value after multiple screenings. This provides high-quality materials for the in-depth research and promotion of southern Hunan's historical and cultural resources, and facilitates users to make in-depth use of them and explore their cultural connotations and values. In this way, southern Hunan's historical and cultural resources can be studied and utilized more efficiently, improving the efficiency of obtaining southern Hunan's historical and cultural resources, saving researchers' time and energy, and promoting the progress of academic research, cultural communication, and educational practice.

[0099] After introducing the basic principles of this application, various non-limiting implementation methods of this application will be specifically introduced in conjunction with the accompanying drawings. Figure 1, the embodiment of the present application provides a multimodal language feature analysis method, comprising the following steps:

[0100] S101: using web crawler technology to collect multimodal data of historical and cultural resources in southern Hunan from a preset data source;

[0101] Specifically, in some embodiments, a distributed crawler system is constructed based on the Scrapy framework through distributed crawler technology, multiple crawler nodes are configured, and a crawling task range is set for each node, and different data sources are accessed respectively. The data sources include but are not limited to the cultural websites of relevant departments, the official websites of local museums, and academic databases, and the historical and cultural resource data of southern Hunan covering documentary materials, image records, descriptions of physical remains, and oral history are collected. In this way, multiple nodes can access different data sources at the same time, realizing the parallelization of data collection. In this way, network resources and computing resources can be fully utilized to improve the efficiency of data collection.

[0102] Of course, the present application is not limited thereto. In other embodiments, the web crawler technology can also be a single crawler program.

[0103] S102: performing a cleaning operation on the acquired multimodal data, using a preset deduplication algorithm to vectorize the data and calculate the cosine similarity between the vectors, and removing redundant data with a cosine similarity higher than a set threshold;

[0104] In some embodiments, the deduplication algorithm is a DeduplicationNet model based on a twin neural network. Compared to traditional deduplication methods, this model can more accurately capture similarities between data, improve deduplication accuracy, more effectively remove redundant data, and enhance data quality.

[0105] S103: Extract keywords from the cleaned multimodal data;

[0106] Specifically, in some embodiments, keywords are extracted from the cleaned multimodal data through the following steps:

[0107] The first step is to use natural language processing technology for text data, using lexical analysis, syntactic analysis, and semantic analysis tools, combined with stop word lists to remove meaningless words, and extract keywords based on the word frequency-inverse document frequency algorithm or the TextRank algorithm;

[0108] The second step is to extract keyframe images from the video data using video keyframe extraction technology. Then, text recognition is performed on the keyframe images, converting the recognized text into text data. The process is then repeated using the same keyword extraction method as for text data. At the same time, video content analysis technology is used to extract scene and action feature information from the video. This feature information is then semantically converted to supplementary keywords.

[0109] In the third step, for image data, optical character recognition technology is used to identify the text content in the image, and the recognized text is used as text data for keyword extraction. In addition, an image feature extraction algorithm is used to extract the visual features of the image, convert the visual features into semantic descriptions, and then extract relevant keywords.

[0110] The fourth step involves performing speech recognition on the audio data, converting it into text format, and then applying the same keyword extraction method used for text data. This allows us to fully explore the key information in each modality of data, ensuring that the extracted keywords more comprehensively and accurately reflect the data content, improving the accuracy and effectiveness of data processing and facilitating the subsequent in-depth analysis and utilization of historical and cultural resources.

[0111] S104: Calculate the relevance of each keyword with preset historical and cultural keywords to screen out core keywords;

[0112] Specifically, in some embodiments, the calculation formula for the correlation between each keyword and the preset historical and cultural theme words is:

[0113]

[0114] Where:

[0115] is the vector of keywords, is the vector of historical and cultural keywords;

[0116] f k is the frequency of the keyword, is the average frequency of all keywords in the document;

[0117] α and β are weight coefficients, and satisfy α+β=1.

[0118] Among them, by introducing the vector of historical and cultural keywords It can focus on a specific cultural background more efficiently according to the set subject scope, avoid processing irrelevant content, and improve the accuracy of extraction. By introducing the second word frequency ratio, the importance weight of keywords can be adjusted dynamically. Specifically, unlike the traditional method that only uses semantic similarity, this method can also consider the frequency of occurrence of keywords to ensure that in diverse data scenarios, high-frequency important words can be highlighted, and low-frequency but valuable keywords are not ignored. For example, in historical and cultural resources, the keyword "historical migration" may have a higher word frequency, while the keyword "Xiangnan Battle" may have a lower word frequency. Through the formula, although the former has a high word frequency, the weight is only increased within a reasonable range, while the latter can also obtain appropriate weight due to its high semantic relevance to the subject word.

[0119] S105: Analyze the correlation between each extracted core keyword and construct a data relationship diagram;

[0120] Specifically, in some embodiments, the data relationship graph can be constructed by the following steps:

[0121] The first step is to use core keywords as nodes;

[0122] The second step is to count the co-occurrence frequency of two keywords in the same data record. If the co-occurrence frequency is higher than the set threshold, an edge is established between the two nodes.

[0123] In the third step, the weight of the edge is determined by the co-occurrence frequency or semantic similarity.

[0124] S106: Calculate the node weight of each node in the data relationship graph, where the node weight is used to indicate the importance of the node in the data relationship graph;

[0125] Specifically, in some embodiments, the node weight is calculated using the following formula:

[0126] NodeWeigt=Rel·log(1+f k )

[0127] Where:

[0128] Rel is the keyword relevance calculated by the formula in claim 4;

[0129] f k is the total frequency of the keyword appearing in the entire data set;

[0130] log(1+f k ) is used to smooth the frequency values of high-frequency and low-frequency keywords.

[0131] In this way, the semantic relevance and word frequency correction value of keywords are combined to comprehensively evaluate the importance of keywords from two dimensions. Among them, the relevance of keywords ensures semantic relevance, and the introduction of natural logarithms avoids the unreasonable expansion of the weights of high-frequency words. Specifically, in historical and cultural resources, the weights of words with high word frequency but average semantic relevance (such as "meeting") will be adjusted to a reasonable range by the natural logarithm; while the weights of words with low word frequency but high semantic relevance will be increased. In this way, it will help to screen out data that is truly highly relevant and important, and improve the quality of data processing and the accuracy of knowledge graph construction.

[0132] S107: Filter the data based on the node weights to select highly relevant data that meets the set criteria;

[0133] Specifically, in some embodiments, the following steps may be performed to filter data based on node weights to select highly relevant data that meets set criteria:

[0134] The first step is to calculate the sum of the weights of all keyword nodes in each data.

[0135] The second step is to compare the total weight value with the set weight threshold, retaining data records that exceed the preset weight threshold and removing data records that fall below the preset weight threshold. This allows for the rapid and accurate screening of highly relevant data that meets the set criteria from a large amount of data, further improving data quality and ensuring the construction of an accurate and valuable knowledge map of southern Hunan's historical and cultural resources, allowing users to access more valuable information.

[0136] S108: Construct a knowledge graph of southern Hunan’s historical and cultural resources based on highly correlated data that meets the set standards.

[0137] Specifically, in some embodiments, a knowledge graph of the historical and cultural resources of southern Hunan can be constructed based on highly relevant data that meets set standards through the following steps:

[0138] The first step is to analyze the composition and internal relationships of highly relevant data that meet the set standards and build a knowledge representation model for historical and cultural resources;

[0139] The second step is to clearly formalize the concepts, attributes, and relationships between concepts required to build the knowledge graph, form a knowledge description specification, cover the definitions of concepts, attributes, and relationships between concepts, build the model layer of the knowledge graph, and complete the abstract description of historical and cultural resource knowledge;

[0140] The third step is to build entity recognition, relationship extraction and attribute acquisition models based on the knowledge representation model of the knowledge graph pattern layer;

[0141] The fourth step is to use the entity recognition, relationship extraction and attribute acquisition models to carry out entity recognition, attribute acquisition and relationship extraction between entities, and then build the data layer of the historical and cultural resource knowledge graph;

[0142] The fifth step is to integrate the historical and cultural resource knowledge obtained in the knowledge acquisition process, store it in a graph database, and visualize it to obtain a knowledge graph of southern Hunan’s historical and cultural resources.

[0143] The application of the knowledge graph of historical and cultural resources in southern Hunan is aimed at users' needs for convenient acquisition and in-depth utilization of knowledge about historical and cultural resources in southern Hunan. Based on the already constructed knowledge graph of historical and cultural resources in southern Hunan, knowledge query and knowledge question and answer of historical and cultural resources in southern Hunan are realized.

[0144] See also Figure 2Based on the same inventive concept as the multimodal language feature analysis method in the aforementioned embodiment, the present embodiment provides a multimodal language feature analysis system, including:

[0145] The first acquisition module 201 is used to collect multimodal data of historical and cultural resources in southern Hunan from a preset data source using web crawler technology;

[0146] The first cleaning module 202 is used to perform a cleaning operation on the acquired multimodal data. It uses a preset deduplication algorithm to vectorize the data and then calculate the cosine similarity between the vectors to remove redundant data with a cosine similarity higher than a set threshold.

[0147] The first extraction module 203 is used to extract keywords from the cleaned multimodal data;

[0148] The first screening module 204 is used to calculate the relevance between each keyword and the preset historical and cultural keywords to screen out the core keywords;

[0149] The first construction module 205 is used to analyze the correlation between each extracted core keyword and construct a data relationship graph;

[0150] A first calculation module 206 is used to calculate the node weight of each node in the data relationship graph, where the node weight is used to represent the importance of the node in the data relationship graph;

[0151] The second screening module 207 is used to screen the data based on the node weights to select the highly relevant data that meets the set standards;

[0152] The second construction module 208 is used to construct a knowledge graph of the historical and cultural resources of southern Hunan based on highly correlated data that meets the set standards.

[0153] In some embodiments, the first cleaning module 202 is specifically used to: for text data, use natural language processing technology, apply lexical analysis, syntactic analysis and semantic analysis tools, combine with stop word lists to remove meaningless words, and extract keywords based on word frequency-inverse document frequency algorithm or TextRank algorithm; for video data, first obtain key frame images through video key frame extraction technology, then perform text recognition processing on the key frame images, convert the recognized text into text data, and then operate according to the method of extracting keywords from text data; at the same time, use video content analysis technology to extract scene and action feature information in the video, and perform semantic conversion of these feature information as supplementary keywords; for image data, use optical character recognition technology to recognize the text content in the image, and use the recognized text as text data for keyword extraction; in addition, use image feature extraction algorithm to extract visual features of the image, convert the visual features into semantic descriptions, and then extract relevant keywords; for audio data, first perform speech recognition, convert the audio into text format, and then use the same keyword extraction method as text data.

[0154] In some embodiments, the first screening module 204 is specifically configured to calculate the relevance of each keyword to a preset historical and cultural keyword using the following formula:

[0155]

[0156] Where:

[0157] is the vector of keywords, is the vector of historical and cultural keywords;

[0158] f k is the frequency of the keyword, is the average frequency of all keywords in the document;

[0159] α and β are weight coefficients, and satisfy α+β=1.

[0160] In some embodiments, the first calculation module 206 is specifically configured to calculate the node weight using the following formula:

[0161] NodeWeigt=Rel·log(1+f k )

[0162] Where:

[0163] Rel is the keyword relevance calculated by the formula in claim 4;

[0164] f k is the total frequency of the keyword appearing in the entire data set;

[0165] log(1+f k ) is used to smooth the frequency values of high-frequency and low-frequency keywords.

[0166] In some embodiments, the second screening module 207 is specifically used to: calculate the sum of the weights of all keyword nodes in each data, compare the total weight value with the set weight threshold, retain data records that exceed the preset weight threshold, and remove data records that are lower than the preset weight threshold.

[0167] In some embodiments, the first construction module 205 is specifically used to: use the core keywords as nodes; by counting the co-occurrence frequency of two keywords in the same data record, if the co-occurrence frequency is higher than a set threshold, an edge is established between the two nodes; the weight of the edge is determined by the co-occurrence frequency or semantic similarity.

[0168] In some embodiments, the deduplication algorithm is a DeduplicationNet model based on a twin neural network.

[0169] In some embodiments, the first acquisition module 201 is specifically used to build a distributed crawler system based on the Scrapy framework through distributed crawler technology, configure multiple crawler nodes, and set the crawling task scope for each node, and access different data sources respectively. The data sources include but are not limited to the cultural websites of relevant departments, the official websites of local museums and academic databases, and collect historical and cultural resource data of southern Hunan covering documentary materials, image records, descriptions of physical relics, and oral history.

[0170] In some embodiments, the second construction module 208 is specifically used to: analyze the composition and internal correlation of highly correlated data that meets the set standards, and construct a historical and cultural resource knowledge representation model; clearly formalize the concepts, attributes and relationships between concepts required to construct the knowledge graph, form a knowledge description specification, cover the definitions of concepts, attributes and relationships between concepts, construct the model layer of the knowledge graph, and complete the abstract description of historical and cultural resource knowledge; construct entity recognition, relationship extraction and attribute acquisition models based on the knowledge representation model of the knowledge graph model layer; use entity recognition, relationship extraction and attribute acquisition models to carry out entity recognition, attribute acquisition and entity relationship extraction, and then construct the data layer of the historical and cultural resource knowledge graph; integrate the historical and cultural resource knowledge obtained in the knowledge acquisition link, store it in the graph database, and visualize it to obtain a knowledge graph of southern Hunan's historical and cultural resources.

[0171] It is understandable that the modules and references recorded in the multimodal language feature analysis system Figure 1The steps in the multimodal language feature analysis method described above correspond to each other. Therefore, the operations, features and beneficial effects described above for the method are also applicable to the multimodal language feature analysis system and the modules included therein, and will not be repeated here.

[0172] See also Figure 4 , based on the inventive concept of a multimodal language feature analysis method in the aforementioned embodiment, an embodiment of the present application provides an electronic device. The electronic device may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device includes a processing device 301 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in ROM 302 (read-only memory) or the program loaded from storage device 308 into RAM 303 (random access memory). In RAM 303, various programs and data required for the operation of the electronic device are also stored. The processing device 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output interface (i.e., an I / O interface 305) is also connected to the bus 304.

[0173] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. The communication device 309 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data.

[0174] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network via the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present application are performed.

[0175] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present application, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In some embodiments of the present application, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0176] In some embodiments, the client and server can communicate using any currently known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an adhoc peer-to-peer network), as well as any currently known or future developed network.

[0177] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: uses web crawler technology to collect multimodal data of historical and cultural resources in southern Hunan from a preset data source; cleans the acquired multimodal data, uses a preset deduplication algorithm to vectorize the data, and then calculates the cosine similarity between vectors, removing redundant data with a cosine similarity higher than a set threshold; extracts keywords from the cleaned multimodal data; calculates the correlation between each keyword and the preset historical and cultural keywords to screen out core keywords; analyzes the correlation between each extracted core keyword and constructs a data relationship graph; calculates the node weight of each node in the data relationship graph, and the node weight is used to represent the importance of the node in the data relationship graph; screens the data based on the node weight to screen out highly correlated data that meets the set standards; and constructs a knowledge graph of historical and cultural resources in southern Hunan based on the highly correlated data that meets the set standards.

[0178] Computer program code for performing the operations of some embodiments of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0179] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0180] The modules described in some embodiments of the present application can be implemented by software or by hardware. The modules described can also be set in a processor: for example, they can be described as: a first acquisition module, a first cleaning module, a first extraction module, a first screening module, a first construction module, a first calculation module, a second screening module, and a second construction module. Among them, the names of these modules do not constitute a limitation on the modules themselves under certain circumstances. For example, the first acquisition module can also be described as a "multimodal data acquisition module for historical and cultural resources in southern Hunan."

[0181] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0182] Some embodiments of the present application further provide a computer program product, including a computer program, which implements any of the above-mentioned multimodal language feature analysis methods when executed by a processor.

[0183] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A multimodal language feature analysis method, characterized in that: The following steps are involved: Use web crawler technology to collect multimodal data of historical and cultural resources in southern Hunan from pre-set data sources; Performing a cleaning operation on the obtained multimodal data, using a preset deduplication algorithm to vectorize the data and then calculate the cosine similarity between the vectors, and removing redundant data with a cosine similarity higher than a set threshold; Extract keywords from the cleaned multimodal data; Calculating the relevance between each of the keywords and the preset historical and cultural keywords to screen out the core keywords; Analyze the correlation between each of the extracted core keywords and construct a data relationship graph; Calculating a node weight of each node in the data relationship graph, wherein the node weight is used to represent the importance of the node in the data relationship graph; Filter the data based on the node weights to select highly relevant data that meets the set standards; Based on the highly correlated data that meets the set standards, a knowledge graph of southern Hunan’s historical and cultural resources is constructed.

2. The multimodal language feature analysis method according to claim 1, characterized in that: For the cleaned multimodal data, the steps for extracting keywords include: For text data, we use natural language processing technology, lexical analysis, syntactic analysis, and semantic analysis tools, combined with stop word lists to remove meaningless words, and extract keywords based on the word frequency-inverse document frequency algorithm or the TextRank algorithm; For video data, we first use video keyframe extraction technology to obtain keyframe images, then perform text recognition on the keyframe images, convert the recognized text into text data, and then operate according to the method of extracting keywords from text data; at the same time, we use video content analysis technology to extract scene and action feature information in the video, and convert this feature information into semantic information as supplementary keywords; For image data, we use optical character recognition technology to identify the text content in the image and use the recognized text as text data for keyword extraction. In addition, we use image feature extraction algorithms to extract the visual features of the image, convert the visual features into semantic descriptions, and then extract relevant keywords. For audio data, speech recognition is first performed to convert the audio into text format, and then the same keyword extraction method as for text data is used.

3. The multimodal language feature analysis method according to claim 1, characterized in that: The calculation formula for the correlation between each keyword and the preset historical and cultural theme words is: Where: is the vector of keywords, is the vector of historical and cultural keywords; f k is the frequency of the keyword, is the average frequency of all keywords in the document; α and β are weight coefficients, and satisfy α+β=1.

4. The multimodal language feature analysis method according to claim 3, characterized in that: The node weight is calculated by the following formula: NodeWeigt=Rel·log(1+f k ) Where: Rel is the keyword relevance calculated by the formula in claim 4; f k is the total frequency of the keyword appearing in the entire data set; log(1+f k ) is used to smooth the frequency values of high-frequency and low-frequency keywords.

5. The multimodal language feature analysis method according to claim 4, characterized in that: The steps for filtering data based on node weights and selecting highly relevant data that meets the set criteria include: Calculate the sum of all keyword node weights in each data. The total weight value is compared with the set weight threshold, and data records exceeding the preset weight threshold are retained, while data records below the preset weight threshold are removed.

6. The multimodal language feature analysis method according to any one of claims 1 to 5, characterized in that: The data relationship diagram is constructed by the following steps: Use core keywords as nodes; By counting the co-occurrence frequency of two keywords in the same data record, if the co-occurrence frequency is higher than the set threshold, an edge is established between the two nodes; The weight of an edge is determined by co-occurrence frequency or semantic similarity.

7. A multimodal language feature analysis system, characterized in that: include: The first acquisition module is used to collect multimodal data of historical and cultural resources in southern Hunan from a preset data source using web crawler technology; A first cleaning module is configured to perform a cleaning operation on the acquired multimodal data, vectorize the data using a preset deduplication algorithm, calculate the cosine similarity between vectors, and remove redundant data with a cosine similarity higher than a set threshold; The first extraction module is used to extract keywords from the cleaned multimodal data; The first screening module is used to calculate the relevance between each keyword and a preset historical and cultural theme word to screen out core keywords; The first construction module is used to analyze the correlation between each of the extracted core keywords and construct a data relationship graph; A first calculation module is used to calculate the node weight of each node in the data relationship graph, wherein the node weight is used to represent the importance of the node in the data relationship graph; A second screening module is used to screen the data based on the node weights to select highly relevant data that meets the set standards; The second construction module is used to construct a knowledge graph of the historical and cultural resources of southern Hunan based on the highly correlated data that meets the set standards.

8. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which implements the method according to any one of claims 1 to 6 when executed by a processing device.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processing device, the method according to any one of claims 1 to 6 is implemented.

Images