A semantic correlation analysis method based on big data

Abstract

The application relates to the technical field of semantic analysis, and discloses a semantic correlation analysis method based on big data, which comprises the following steps: step one: according to big data, the domain type of a document to be analyzed is identified, and the document is divided into multiple analysis pages according to the number of pages; step two: the most frequently-occurring words in multiple analysis pages are obtained, recorded as high-frequency words, and the high-frequency words of multiple analysis sub-pages are integrated to construct a high-frequency word database; and step three: according to the frequency of each high-frequency word in the high-frequency word database in multiple analysis pages, the analysis pages are classified into associated pages corresponding to the high-frequency words. Through the identification of the domain type of the document to be analyzed and the division of the analysis pages, the high-frequency words are extracted in combination with a domain dictionary, and the related domain new words appearing in the document are simultaneously mined, and then the correlation matrix of the analysis pages and the high-frequency words is constructed to calculate and obtain the correlation degree scores of the analysis pages and the high-frequency words.

Description

Technical Field

[0001] This invention relates to the field of semantic analysis technology, specifically to a semantic correlation analysis method based on big data. Background Technology

[0002] In the field of text semantic processing, semantic association analysis typically takes text data as the processing object. Through steps such as text cleaning, word extraction, and association calculation, it achieves preliminary analysis of semantic relationships within or between documents, and explores potential associations between words and between texts. This provides data support for applications such as information retrieval, text classification, and topic mining. For example, in academic literature analysis, existing technologies can preliminarily determine the degree of topic association between different documents by statistically analyzing the co-occurrence frequency of keywords.

[0003] However, existing semantic association analysis techniques have some shortcomings in practical applications. When conducting semantic association analysis, existing techniques mostly rely on pre-set lexicons and lack dynamic judgment mechanisms based on the stability of word structure and the diversity of collocations. They are prone to missing new words in emerging fields, which affects the timeliness and accuracy of association analysis.

[0004] To address this, the present invention proposes a semantic correlation analysis method based on big data to overcome the shortcomings of existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a semantic correlation analysis method based on big data. The objective of this invention can be achieved through the following technical solutions: A semantic correlation analysis method based on big data includes the following steps: Step 1: Identify the domain type of the document to be analyzed based on big data, and divide it into multiple analysis pages according to the number of pages; Step 2: Obtain the most frequently occurring words in multiple analysis pages and record them as high-frequency words. Integrate the high-frequency words from multiple subpages to be analyzed to construct a high-frequency word database. Step 3: Based on the frequency of each high-frequency word in the high-frequency word database appearing in multiple analysis pages, classify the analysis pages into associated pages corresponding to the high-frequency words; Step 4: Extract key features corresponding to the high-frequency words in each of the associated pages, and obtain the strength between the associated pages based on the key features in each of the associated pages; Step 5: Generate the corresponding association graph based on the association strength between the associated pages.

[0006] Preferably, the method for performing step two is as follows: Construct a domain dictionary based on the domain type of the document to be analyzed; Obtain all text from the analysis page, and remove some text from all text on the analysis page according to preset filtering rules; Extract all text after removing some text by single character, obtain the single character that appears most frequently, obtain the characters before and after the single character that appears most frequently in the paragraph, form core words, and obtain the frequency of the core words in the analysis page, and record them as high-frequency words; Obtain the semantic similarity between the core words and the words in the domain dictionary, determine whether the core words are recorded words in the domain dictionary, if the core words are related to the recorded words in the domain dictionary, they are judged as known words, if they are not related, obtain the core words to determine whether they are new words in the domain, and update the domain dictionary.

[0007] Preferably, the method for determining whether core words are new words in the domain is as follows: The core words are broken down into multiple combined characters, and the frequency of the core words and multiple combined characters appearing in all text on the analysis page is obtained. The frequency ratio is obtained by comparing the frequency of the core words in all text with the frequency of multiple combined characters in all text on the analysis page. Define at least two adjacent single-character words on the left and right as adjacent conjunctions, and count the number of different at least two single-character words on the left and right of the core words; Find the minimum number of characters in at least two distinct characters that are adjacent to the core word; The frequency ratio and minimum quantity are compared with their respective set thresholds. If the frequency ratio is greater than the set frequency threshold and the minimum quantity is greater than the set quantity threshold, the core word is determined to be a newly added domain word.

[0008] Preferably, the method for performing step three is as follows: An association matrix is ​​constructed based on the high-frequency words in the high-frequency word database and all analysis pages. In the association matrix, rows represent the analysis pages and columns represent words in the high-frequency word database. The correlation scores between each high-frequency word and each analysis page are obtained based on the constructed correlation matrix. High-frequency words on the analysis page whose relevance scores are greater than a set relevance threshold are designated as relevant high-frequency words on the analysis page.

[0009] Preferably, the method for obtaining the correlation score is as follows: The ratio of the number of times each high-frequency word appears on each analysis page to all words on each analysis page is set as the relative frequency of the high-frequency word on the analysis page; The document frequency of high-frequency words is calculated based on the number of pages analyzed and the relative frequency of each high-frequency word. The correlation score between high-frequency words and each analyzed page is calculated based on the obtained document frequency and relative frequency.

[0010] Preferably, the method for performing step four is as follows: Obtain various features of high-frequency words in each analysis page: obtain the short sentence generation features of short sentences containing high-frequency words in the analysis page, obtain the semantic features of words that are semantically similar to high-frequency words, obtain the paragraph position where high-frequency words appear in the analysis page, and obtain the number of times high-frequency words appear with other high-frequency words in the same analysis page; The extracted multi-type features are normalized to generate a comprehensive feature vector for each analysis page. Based on the comprehensive feature vector generated, obtain the cosine value of the angle between any two analyzed pages, and obtain the correlation strength between any two analyzed pages based on the obtained cosine value of the angle.

[0011] Preferably, the method for generating the feature vector is as follows: The numerical features are normalized, and the short sentence features and semantic features are encoded and converted into binary vectors. Weight coefficients are assigned to different features, and the feature vector of a single feature is obtained by combining the weight coefficients with the normalized value or binary vector of the feature. The comprehensive feature vector of the analysis page is obtained by combining multiple individual feature vectors.

[0012] Preferably, a visual association graph is generated by using related pages and high-frequency words as core nodes, with the strength of association between pages as edges, and a hierarchical filtering function is set up.

[0013] The beneficial effects of this invention are: 1. This invention identifies the domain type of the document to be analyzed and divides it into analysis pages. Then, it extracts high-frequency words by combining the domain dictionary and simultaneously mines new words related to the domain that appear in the document. Subsequently, it constructs an association matrix between the analysis pages and high-frequency words, calculates the association score between the analysis pages and high-frequency words, classifies the analysis pages of the document to be analyzed, generates a comprehensive feature vector for each analysis page, obtains the association strength between each analysis page through cosine similarity, and finally generates a visualized association map. This solves the problems of traditional semantic analysis, such as the inability to effectively identify emerging domain words and the shallow semantic association between pages.

[0014] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the steps of a semantic correlation analysis method based on big data according to the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Please see Figure 1 As shown, this invention is a semantic correlation analysis method based on big data, comprising the following steps: Step 1: Identify the domain type of the document to be analyzed based on big data, and divide it into multiple analysis pages according to the number of pages; Step 2: Obtain the most frequently occurring words from multiple analysis pages and record them as high-frequency words. Integrate the high-frequency words from multiple subpages to be analyzed to build a high-frequency word database. Step 3: Based on the frequency of each high-frequency word in the high-frequency word database appearing on multiple analysis pages, classify the analysis pages into related pages corresponding to the high-frequency words; Step 4: Extract key features corresponding to high-frequency words from each associated page, and obtain the strength between each associated page based on the key features in each associated page; Step 5: Generate the corresponding association graph based on the association strength between each related page.

[0019] Specifically, based on big data resources covering terminology databases across various industries and fields, the domain type of the document to be analyzed is identified through multi-dimensional feature analysis. First, core vocabulary, professional terms, and sentence structure features are extracted from the document. This is then used in conjunction with the trained language model BERT for preliminary judgment. The results are corrected by comparing the semantic similarity with benchmark documents in the domain to determine the technical field to which the document belongs. Subsequently, the document is divided into multiple analysis pages according to page count. If a single page is too short, such as containing only half a page of text or having many charts, it is merged with the previous adjacent page. The method for performing the work in step two is as follows: Construct a domain dictionary based on the domain type of the document to be analyzed; Acquire all text of the analysis page, and remove part of the text from all text of the analysis page according to preset filtering rules; Extract all the text after removing part of the text character by single character, obtain the single character with the highest occurrence frequency, acquire the adjacent characters before and after the single character with the highest occurrence frequency in the paragraph to form core words, acquire the occurrence frequency of the core words in the analysis page, and record them as high-frequency words; Acquire the semantic similarity between the core words and the words in the domain dictionary, determine whether the core words are recorded words in the domain dictionary, if a core word is related to a recorded word in the domain dictionary, it is determined as a known word, if not, determine whether the core word is a new domain word, and update the domain dictionary; Specifically, according to the domain type identified by step 1 from the document to be analyzed, a domain dictionary covering terms, historical high-frequency words and professional vocabularies of the domain is constructed; after acquiring all text content of each analysis page, meaningless symbols are removed according to preset filtering rules, such as special punctuation, numerical symbols, general stop words such as "de", "zai", "yu" and the like, and domain-specific redundant vocabularies, so as to retain text with actual semantics; single characters are extracted one by one from the cleaned text, the single character with the highest occurrence frequency is counted, and 1 to 2 adjacent characters before and after said single character are combined to form a candidate core word; through the number of characters of the core word and the total number of characters of the analysis page, the proportion of the core word in the total number of words of the analysis page is calculated, and the core word with the highest proportion is recorded as a high-frequency word; The semantic similarity between the core words and the words in the domain dictionary is calculated through a mature word vector model; when the cosine similarity is greater than or equal to 0.7, it is determined that the core word in the analysis page is related to the word in the domain dictionary and classified as a known word; if not, it is further determined whether it is a new domain word, and finally the confirmed new word is supplemented into the domain dictionary, with its frequency and data of the first occurrence page recorded synchronously.

[0020] The method for acquiring and determining whether a core word is a new domain word is as follows: decomposing the core word into a plurality of combined characters, acquiring the occurrence frequency of the core word and the plurality of combined characters in all text of the analysis page, comparing the occurrence frequency of the core word in all text with the occurrence frequency of the plurality of combined characters in all text of the analysis page to obtain a frequency ratio, setting at least two single characters adjacent to the left and right of the core word as adjacent connecting words, counting the number of different at least two single characters adjacent to the left and right of the core word, and acquiring the minimum value of the number of different at least two single characters adjacent to the left and right of the core word; Comparing the acquired frequency ratio and the minimum number value with their respective set thresholds, if the frequency ratio is greater than the set frequency threshold and the minimum number value is greater than the set number threshold, the core word is determined as a new domain vocabulary.

[0021] Specifically, the core words are split into all possible binary subdivisions, such as splitting blockchain into zone and blockchain, or zone and chain, and decomposing them into multiple combined characters. The frequency of the core words as a whole in all text on the analysis page and the frequency of each combined character individually in all text are counted. The ratio of the overall frequency of the core words to the product of the frequencies of each combined character is calculated. Continuing with the above example, the ratio of the proportion of blockchain in all characters on the analysis page to the proportion of zone and blockchain in all characters on the analysis page is taken as the minimum value of the ratio among all splitting methods. Then, taking the core word as the center, take at least two single characters on its left and right sides as adjacent connecting words. For example, if the core word is "meta-universe", and the analysis page shows adjacent virtual and digital on the left and adjacent technology and platform on the right, count the total number of different adjacent characters on the left and the total number of different adjacent characters on the right, and take the smaller value of the two. Finally, the calculated frequency ratio is compared with the preset frequency threshold, and the minimum number of adjacent characters is compared with the preset number threshold. The frequency threshold is adjusted according to the characteristics of the field, such as setting the field to 1.5 and the number threshold to 3. If the frequency ratio is greater than the frequency threshold and the minimum number is greater than the number threshold, the core word is determined to be a newly added domain word. If either condition is not met, it is determined to be a non-domain new word or the splitting method and statistical data need to be re-verified. When it is determined to be a newly added domain word, the core word is recorded in the domain dictionary for use in the next semantic analysis.

[0022] The method for performing the work in step three is as follows: A correlation matrix is ​​constructed based on the high-frequency words in the high-frequency word database and all analysis pages. In the correlation matrix, rows represent analysis pages and columns represent words in the high-frequency word database. The correlation score between each high-frequency word and each analysis page is obtained based on the constructed correlation matrix. High-frequency words in the analysis pages whose correlation scores are greater than a set correlation threshold are set as associated high-frequency words of the analysis pages.

[0023] Construct an association matrix using all high-frequency words in the high-frequency word database as columns and all analyzed pages as rows, with the matrix element values ​​being... That is, the first The first page The frequency of occurrence of each high-frequency word is used to calculate the relevance score between each high-frequency word and the analyzed pages through an association matrix. The relevance score is obtained by setting the ratio of the number of times each high-frequency word appears on each analyzed page to the total number of words on each analyzed page as the relative frequency of the high-frequency word on that page. The document frequency of the high-frequency words is calculated based on the number of analyzed pages and the relative frequency of each high-frequency word. The relevance score between the high-frequency words and each analyzed page is then calculated based on the obtained document frequency and relative frequency, using the formula: ; in, For the first High-frequency words With the Analysis page The correlation score, To analyze the page The total number of words, The total number of documents to be analyzed. For containing high-frequency words The total number of pages; Taking a document in the field of computer science as an example, the document is divided into 5 analysis pages, with a total of [number of pages]. We selected high-frequency keywords: "machine learning," and calculated their relevance scores to each page. Page 1 contains a total of 200 words. High-frequency words appear 10 times in page 1, therefore the relative frequency of high-frequency words in page 1 is... ; Page 3 contains a total of 300 words. High-frequency words appear 15 times in page 3, therefore the relative frequency of high-frequency words in page 3 is... .

[0024] If the high-frequency word is found on three pages (page 1, page 3, and page 5) of the document to be analyzed, then: .

[0025] The relevance scores between pages 1 and 2 and high-frequency words are both: ; Taking the high-frequency term "deep learning" as another example, the total number of pages remains 5: Page 2 contains a total of 250 words. The high-frequency words appear 8 times in the deep learning analysis. ; If only pages 2 and 4 contain information about deep learning, then... ; The correlation score between page 2 and high-frequency words in deep learning. .

[0026] Subsequently, the relevance threshold is dynamically set according to the characteristics of the domain to which the analysis page belongs. For example, the threshold for professional documents is set to 0.02, and for general documents it is set to 0.01. High-frequency words with relevance scores exceeding the threshold in the analysis page are set as the relevance high-frequency words of the analysis page. If a page has multiple high-frequency words that meet the conditions, they are all marked as its relevance high-frequency words.

[0027] Step four involves extracting key features corresponding to high-frequency words from each associated page. The strength between associated pages is then determined based on these key features. Key features are obtained by: generating sentence features from short phrases containing high-frequency words on the analysis page; acquiring semantically similar words to high-frequency words to form semantic features; determining the paragraph positions where high-frequency words appear on the analysis page; and determining the frequency of high-frequency words appearing on the same analysis page with other high-frequency words. The extracted key features are then normalized to generate a comprehensive feature vector for each analysis page. The feature vector generation method involves: normalizing numerical features; encoding sentence and semantic features into binary vectors; assigning weight coefficients to different features; calculating the feature vector of a single feature by combining the weight coefficients with the normalized numerical value or binary vector of the feature; and combining multiple single feature vectors to obtain the comprehensive feature vector of the analysis page.

[0028] The method for obtaining the strength between related pages is as follows: First, obtain multi-dimensional features related to high-frequency words. Extract structurally complete key phrases from the text containing high-frequency words. For example, use big data-driven semantic analysis to generate phrase features. Based on semantic similarity, obtain synonyms with semantic similarity greater than or equal to 0.8 with high-frequency words in the analysis page to form semantic features. Record the position and frequency of high-frequency words in different paragraphs such as title, abstract, and body text. Set different weight coefficients for different paragraphs. For example, the weight coefficient in the title is 2, the weight coefficient in the abstract is 1.5, and the weight coefficient in the body text is 1. Count the number of times high-frequency words co-occur with other related high-frequency words on the same page and calculate the relative co-occurrence density. The contribution density is equal to the number of co-occurrences divided by the total number of words in the analysis page. Then, the extracted features are normalized. Numerical features such as co-occurrence density and position weight are mapped to the [0,1] interval. Textual features such as short sentences and semantic features are converted into binary vectors. If a feature exists on the page, it is recorded as 1; if it does not exist, it is recorded as 0. A comprehensive feature vector is generated for each analysis page. Each feature component in the vector is assigned a different weight coefficient according to its importance. The weight coefficient for semantic features is 0.4, the weight coefficient for short sentence features is 0.25, the weight coefficient for position features is 0.15, and the weight coefficient for co-occurrence features is 0.2. Finally, the cosine value of the angle between the comprehensive feature vectors of any two analysis pages is calculated. This value is directly used as the correlation strength between the two pages, and a strength grading standard is set according to the domain characteristics. When the cosine value of the angle is greater than or equal to 0.7, the two pages are considered to be strongly correlated. When the cosine value of the angle is between 0.3 and 0.7 (inclusive), the two pages are considered to be moderately correlated. When the cosine value of the angle is less than 0.3, the two pages are considered to be weakly correlated.

[0029] Step five involves using related pages and high-frequency words as core nodes, with the strength of the association between pages as edges, and setting up a hierarchical filtering function to generate a visual association graph.

[0030] Linked pages and high-frequency words are used as the core nodes of the visualized association graph. Page nodes are dynamically adjusted based on the total number of times the analyzed page is linked to other analyzed pages. The node diameter increases by 10% for every 5 additional links, and the more densely linked page nodes are visually more prominent. The size of high-frequency word nodes is determined by a weighted value calculated based on their total frequency of occurrence across all analyzed pages and a domain weight coefficient. The weight coefficient for high-frequency words in the core domain is set to 1.2, and the weight coefficient for high-frequency words in the general domain is 1.0, ensuring that core thematic words are easier to identify. The node colors are finely differentiated according to the sub-category of the domain to which the high-frequency words belong. For example, in the computer field, artificial intelligence corresponds to dark blue, blockchain to emerald green, and big data to orange. Each sub-category is further subdivided into 3 or 5 different shades to distinguish secondary thematic branches, making the semantic connections in different directions within the domain immediately clear. The edges of the graph directly map the strength of the association between any two related pages. The thickness of the lines is positively correlated with the strength of the association. Strong associations (cosine of the angle ≥ 0.7) use a dark, thick line with a width of 2 in RGB color. Medium associations (0.3 ≤ cosine of the angle < 0.7) use a medium-colored line with a width of 1. Weak associations (cosine of the angle < 0.3) use a light, thin line with a width of 0.5. The color of all edges is consistent with the main color of the corresponding high-frequency word node, clearly reflecting the theme of each association. Meanwhile, the graph has built-in multi-dimensional hierarchical filtering functions, allowing users to customize the correlation strength threshold range (0-1) via a slider, select one or more domain subcategories via a drop-down menu, and choose to display page nodes, high-frequency word nodes, or both simultaneously via radio buttons to meet the analysis needs of different scenarios. In addition, the graph has convenient interactive operation functions. The mouse wheel can achieve free scaling according to the corresponding ratio. When dragging a node, the system will automatically adjust the length and angle of the surrounding lines. Clicking on a page node will display a pop-up window showing the list of its high-frequency words, the weight ratio of the core features, and the correlation score ranking. Clicking on a high-frequency word node will display a pop-up window showing the complete list of related pages, the total frequency of occurrence, the page number of the first occurrence, whether it is a new domain word, and related words with high semantic similarity. It presents key correlation information in all aspects and finally generates a visual correlation graph with a clear structure, smooth interaction, and rich information dimensions, intuitively and concretely representing the semantic correlation network within the document.

[0031] The above description is merely an example and illustration of the concept of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the concept of the invention or exceed the scope defined in the claims, they should all fall within the protection scope of the present invention.

Claims

1. A semantic correlation analysis method based on big data, characterized in that, Includes the following steps: Step 1: Identify the domain type of the document to be analyzed based on big data, and divide it into multiple analysis pages according to the number of pages; Step 2: Obtain the most frequently occurring words from multiple analysis pages, record them as high-frequency words, and integrate the high-frequency words from multiple analysis pages to construct a high-frequency word database; Step 3: Based on the frequency of each high-frequency word in the high-frequency word database appearing in multiple analysis pages, classify the analysis pages into associated pages corresponding to the high-frequency words; Step 4: Extract features corresponding to the high-frequency words in each of the associated pages. The features include: short sentence features generated by extracting structurally complete short sentences from text containing high-frequency words, semantic features formed by words with similar semantics to high-frequency words, and the occurrence position and frequency of high-frequency words in different paragraphs. The association strength between the associated pages is obtained based on the features in each associated page. The specific method is as follows: The features corresponding to the extracted high-frequency words are normalized to generate a comprehensive feature vector for each page. The cosine of the angle between any two pages is obtained from the generated comprehensive feature vector, and the correlation strength between any two pages is obtained from the obtained cosine of the angle. Step 5: Generate the corresponding association graph based on the association strength between the associated pages.

2. The semantic correlation analysis method based on big data according to claim 1, characterized in that, The method for performing step two is as follows: Construct a domain dictionary based on the domain type of the document to be analyzed; Obtain all text from the analysis page, and remove some text from all text on the analysis page according to preset filtering rules; Extract all text after removing some text by single character, obtain the single character that appears most frequently, obtain the characters before and after the single character that appears most frequently in the analysis page, form core words, and obtain the frequency of the core words in the analysis page, and record them as high-frequency words; Obtain the semantic similarity between the core words and the words in the domain dictionary, determine whether the core words are recorded words in the domain dictionary, if the core words are related to the recorded words in the domain dictionary, they are judged as known words, if they are not related, obtain the core words to determine whether they are new words in the domain, and update the domain dictionary.

3. The semantic correlation analysis method based on big data according to claim 2, characterized in that, The method for determining whether core words are new words in the domain is as follows: The core words are broken down into multiple combined characters, and the frequency of the core words and multiple combined characters appearing in all text on the analysis page is obtained. The frequency ratio is obtained by comparing the frequency of the core words in all text with the frequency of multiple combined characters in all text on the analysis page. Define at least two adjacent single-character words on the left and right as adjacent conjunctions, and count the number of different at least two single-character words adjacent to the left and right of the core words; Find the minimum number of characters in at least two distinct characters that are adjacent to the core word; The frequency ratio and minimum quantity are compared with their respective set thresholds. If the frequency ratio is greater than the set frequency threshold and the minimum quantity is greater than the set quantity threshold, the core word is determined to be a newly added domain word.

4. The semantic correlation analysis method based on big data according to claim 1, characterized in that, The method for performing step three is as follows: An association matrix is ​​constructed based on the high-frequency words in the high-frequency word database and all analysis pages. In the association matrix, rows represent the analysis pages and columns represent words in the high-frequency word database. The correlation scores between each high-frequency word and each analysis page are obtained based on the constructed correlation matrix. High-frequency words on the analysis page whose relevance scores are greater than a set relevance threshold are designated as relevant high-frequency words on the analysis page.

5. The semantic correlation analysis method based on big data according to claim 1, characterized in that, The method for performing step five is as follows: Based on related pages and high-frequency words as core nodes, and using the strength of the relationship between pages as edges, a hierarchical filtering function is set up to generate a visual relationship graph.

Images