A conference information processing method and system based on big data

Abstract

The application provides a conference information processing method and system based on big data, acquires a conference document to be processed and a conference document corpus to which the conference document belongs, calculates position and area weighted term frequency of a term, calculates within-class inverse document frequency of the term in a belonging class cluster based on a division result of a document theme class cluster, obtains initial weight of the term based on a product of the position and area weighted term frequency and the within-class inverse document frequency, screens out core terms with weight greater than a first preset threshold, constructs a weighted co-occurrence network graph of the core terms, takes the initial weight of each core term as initial weight in the weighted co-occurrence network graph, iteratively updates weight of each core term node according to connection relationship of nodes and weight of edges in the graph through a preset graph sorting iteration algorithm, and takes the obtained weight as weight of the core term when weight of each node converges or reaches a preset iteration number.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of data processing, and particularly relates to a conference information processing method and system based on big data. BACKGROUND

[0002] With the rapid development of information technology, the number of electronic documents generated by academic conferences has increased dramatically. These conference documents, such as papers, reports and presentations, contain a large amount of professional knowledge and cutting-edge information. Quickly and accurately extracting core information, such as keywords and topics, from a large number of conference documents is of great significance for scholars to conduct literature research, track discipline hotspots, and build knowledge graphs. Keyword extraction is one of the core technologies to achieve this goal. Traditional keyword extraction methods mainly rely on statistical models. Among them, the term frequency-inverse document frequency (TF-IDF) algorithm is the most widely used classical statistical method. If a word appears frequently (TF) in a document and rarely (IDF) in other documents in the entire document corpus, it is considered to have high importance for the document. However, it ignores the position information and structural information of the word within the document, and treats words in titles, abstracts and main texts equally, while words usually appearing in titles and abstracts can better summarize the main theme of the document. Moreover, the standard IDF calculation method is based on the entire corpus, which may lead to some important professional terms in a specific topic area being incorrectly assigned a lower weight. TF-IDF is a bag-of-words model that calculates the weight of each word in isolation, completely ignoring the semantic association and co-occurrence relationship between words.

[0003] Graph-based algorithms improve the above problems, such as the TextRank algorithm. This type of method abstracts the document as a word network graph and calculates the importance of nodes (words) in the graph through a PageRank-like algorithm, thereby obtaining the co-occurrence relationship between words. However, when constructing the network graph, they often treat all candidate words equally without fully utilizing the statistical characteristics of the words themselves as the initial importance judgment basis, or the edge weight setting is relatively simple and cannot accurately measure the association strength between words. Although some studies have attempted to combine TF-IDF with graph models, such as using TF-IDF values as the initial weight of the graph sorting algorithm, these methods mostly follow the standard TF-IDF calculation framework and fail to address the inherent shortcomings of TF-IDF in terms of word position, document structure, and theme sensitivity. Existing technologies still have deficiencies in integrating statistical characteristics of words and semantic structural information, making it difficult to comprehensively and accurately evaluate the importance of core terms in conference documents. SUMMARY

[0004] The application aims to provide a conference information processing method and system based on big data, which is used to solve the problem that the importance of core terms is difficult to determine in the existing conference information processing, resulting in inaccurate extraction of core information, comprising the following steps:

[0005] S1: obtaining a to-be-processed conference document and a conference document corpus to which the to-be-processed conference document belongs; for any term in the to-be-processed conference document, calculating the position and region weighted term frequency of the term based on a preset document region weight coefficient and an inverse sequence position attenuation function of the term in the document region ; the document region includes a title, an abstract and a body, and the weight coefficients of the three are sequentially decreased;

[0006] S2: converting each document in the conference document corpus into a document vector by using a document vectorization model, and performing a hierarchical clustering algorithm on the document vector to obtain a plurality of document topic clusters; based on the division result of the document topic clusters, calculating the within-cluster inverse document frequency of the term in the belonging cluster ;

[0007] S3: obtaining the initial weight of the term based on the product of the position and region weighted term frequency and the within-cluster inverse document frequency ; based on the initial weight, screening out core terms with a weight greater than a first preset threshold;

[0008] S4: constructing a weighted co-occurrence network graph of the core terms, the nodes in the graph being the core terms, and when the sentence distance of any two core terms in a document is less than a second preset threshold, a connection edge is established between the corresponding nodes, and the weight of the edge is the point mutual information value of the two core terms;

[0009] S5: taking the initial weight of each core term as the initial weight in the weighted co-occurrence network graph, and through a preset graph sorting iteration algorithm, iteratively updating the weight of each core term node according to the connection relationship of the nodes and the weight of the edges in the graph, and when the weight of each node converges or reaches a preset iteration number, taking the obtained weight as the weight of the core term.

[0010] Optionally, the calculation of the position and region weighted term frequency of the term based on the preset document region weight coefficient and the inverse sequence position attenuation function of the term in the document region comprises:

[0011] for each occurrence of the term in the document, a single weighting value is calculated, which is obtained by multiplying the weight coefficient of the region where the term is located and the position attenuation value in the region;

[0012] the position and region weighted term frequency of the term a sum of all single weighting values of the word item in the document;

[0013] wherein the weight coefficient of the document region is: the title weight coefficient is 1.0, the abstract weight coefficient is 0.8, and the text weight coefficient is 0.5; the position attenuation value is calculated by an inverse sequence position attenuation function , wherein p is a position sequence number of the word item in the region counted from back to front or from front to back, and p is counted from 1.

[0014] Optionally, based on the division result of the document topic cluster, an intra-cluster inverse document frequency of the word item in the belonging cluster is calculated , comprising:

[0015] A conference document corpus is trained by using a Doc2Vec model, and each document in the corpus is converted into a document vector.

[0016] A Ward minimum variance criterion-based agglomerative hierarchical clustering algorithm is applied to all document vectors to divide the documents into a preset number of document topic clusters.

[0017] The intra-cluster inverse document frequency of the word item in the belonging cluster is calculated according to the following formula: , wherein is the total number of documents in the cluster C to which the word item t belongs, and is the number of documents in the cluster C containing the word item t.

[0018] Optionally, based on the initial weight, core word items with a weight greater than a first preset threshold are screened out, comprising:

[0019] An arithmetic mean value of initial weights of all word items is calculated .

[0020] The first preset threshold is set to a preset multiple of the arithmetic mean value .

[0021] Word items with an initial weight greater than the first preset threshold are screened out as core word items.

[0022] Optionally, the weighted co-occurrence network graph of the core word items is constructed, comprising:

[0023] The second preset threshold is set to N, and when the sentence distance of any two core word items in the document is less than N, a connection edge is established between the corresponding nodes;

[0024] The weight of the edge is calculated by using a point mutual information value PMI, and the calculation formula is: .

[0025] ​In this context, a sliding window is constructed using a continuous sequence of sentences of length N from the document. This represents the total number of sliding windows in the document. Core term and The number of sliding windows that appear together and Each contains core terms and The number of sliding windows.

[0026] Optionally, the step of iteratively updating the weights of each core term node using a preset graph sorting iterative algorithm based on the connection relationships of nodes and the weights of edges in the graph includes:

[0027] A personalized graph sorting iterative algorithm is adopted, and a damping coefficient d is set;

[0028] The formula for updating the weights of each core term node is as follows: ,in Let i be the weight of node i. Let i be the set of nodes adjacent to node i. Let be the weight of the edge connecting node i and node j. Let be the normalized initial weight of node i. The normalized initial weight is obtained by dividing the initial weight of the core term by the sum of the initial weights of all core terms.

[0029] The iteration termination condition is set as follows: the change in the weight value of all nodes in two consecutive iterations is less than the preset value, or the number of iterations reaches the preset number.

[0030] This invention also provides a conference information processing system based on big data, comprising the following modules:

[0031] The position- and region-weighted word frequency calculation module is used to acquire the meeting document to be processed and its associated meeting document corpus; for any word in the meeting document to be processed, based on a preset document region weight coefficient and a word's inverse position decay function within the document region, the position- and region-weighted word frequency of the word is calculated. The document area includes a title, a summary, and the main text, with the weight coefficients of the three decreasing sequentially.

[0032] The intra-class inverse document frequency (INV) calculation module is used to convert each document in the conference document corpus into a document vector using a document vectorization model, and to perform a hierarchical clustering algorithm on the document vectors to obtain multiple document topic clusters; based on the partitioning results of the document topic clusters, the module calculates the intra-class INV of the term within its respective cluster. ;

[0033] The core term filtering module is used to weight term frequencies based on the location and region. With the inverse document frequency within the class The product of the terms yields the initial weight of the term; based on the initial weight, core terms with weights greater than a first preset threshold are selected.

[0034] The network graph construction module is used to construct a weighted co-occurrence network graph of the core terms. The nodes in the graph are core terms. When the sentence distance between any two core terms in the document is less than a second preset threshold, a connection edge is established between the corresponding nodes. The weight of the edge is the point mutual information value of the two core terms.

[0035] The core term weight calculation module is used to take the initial weight of each core term as the initial weight in the weighted co-occurrence network graph. Through a preset graph sorting iteration algorithm, the weight of each core term node is iteratively updated according to the connection relationship of the nodes in the graph and the weight of the edges. When the weight of each node converges or reaches the preset number of iterations, the obtained weight is used as the weight of the core term.

[0036] Optionally, the step of calculating the position of the term and its region-weighted frequency based on a preset document region weight coefficient and a word's reverse position decay function within the document region is performed. ,include:

[0037] For each occurrence of the term in the document, a single weighted value is calculated. The single weighted value is obtained by multiplying the weight coefficient of the region where the term is located by the positional decay value in the region.

[0038] The position and region-weighted word frequency of the terms This is the sum of all single-weighted values ​​of the term in the document;

[0039] The weighting coefficients for the document regions are as follows: title weighting coefficient is 1.0, abstract weighting coefficient is 0.8, and body text weighting coefficient is 0.5; the position decay value is obtained through a reverse position decay function. The calculation is performed where p is the position number of the term within the region, counted from back to front or from front to back, starting from 1.

[0040] Optionally, based on the document topic clustering results, the intra-category inverse document frequency of the term is calculated within its respective cluster. ,include:

[0041] The Doc2Vec model was used to train the conference document corpus, and each document in the corpus was converted into a document vector.

[0042] An agglomerative hierarchical clustering algorithm based on the Ward minimum variance criterion is applied to all document vectors to divide the documents into a predetermined number of document topic clusters;

[0043] The inverse document frequency of the term within its respective cluster The calculation formula is: ,in The total number of documents within the cluster C to which term t belongs. Let be the number of documents containing term t within cluster C.

[0044] Optionally, the step of filtering out core terms with weights greater than a first preset threshold based on the initial weights includes:

[0045] Calculate the arithmetic mean of the initial weights of all terms. ;

[0046] The first preset threshold is set to the arithmetic mean. The preset multiple;

[0047] Terms with an initial weight greater than the first preset threshold are selected as core terms.

[0048] Optionally, constructing the weighted co-occurrence network graph of the core terms includes:

[0049] The second preset threshold is set to N. When the distance between any two core terms in the document is less than N, a connection edge is established between the corresponding nodes.

[0050] The edge weights are calculated using the point mutual information (PMI) value, and the calculation formula is as follows: ;

[0051] In this context, a sliding window is constructed using a continuous sequence of sentences of length N from the document. This represents the total number of sliding windows in the document. Core term and The number of sliding windows that appear together and Each contains core terms and The number of sliding windows.

[0052] Optionally, the step of iteratively updating the weights of each core term node using a preset graph sorting iterative algorithm based on the connection relationships of nodes and the weights of edges in the graph includes:

[0053] A personalized graph sorting iterative algorithm is adopted, and a damping coefficient d is set;

[0054] The formula for updating the weights of each core term node is as follows: ,in Let i be the weight of node i. Let i be the set of nodes adjacent to node i. Let be the weight of the edge connecting node i and node j. Let be the normalized initial weight of node i. The normalized initial weight is obtained by dividing the initial weight of the core term by the sum of the initial weights of all core terms.

[0055] The iteration termination condition is set as follows: the change in the weight value of all nodes in two consecutive iterations is less than the preset value, or the number of iterations reaches the preset number.

[0056] This invention calculates word frequencies by introducing document region weights and term position decay functions, highlighting the value of important words in key regions such as titles and abstracts. Simultaneously, by performing topic clustering on documents and calculating inverse document frequencies within each cluster, it avoids the dilution of weights for core terms within specific topics by general domain terminology, thus improving the topic relevance of weight calculations. A weighted co-occurrence network is constructed using point mutual information, and iterative ranking is initiated with a high-quality initial weight graph, thereby integrating the statistical features, structural information, and semantic relationships between words. This invention can more accurately assess the importance of core terms in conference documents, improving the accuracy and effectiveness of keyword extraction. Attached Figure Description

[0057] Figure 1 This is a flowchart of the first specific embodiment;

[0058] Figure 2 This is a schematic diagram of the position decay function;

[0059] Figure 3 This is a schematic diagram of the document hierarchical clustering results;

[0060] Figure 4 A diagram illustrating the filtering of core terms. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0062] The first specific embodiment provides a meeting information processing method based on big data, such as... Figure 1 As shown, it includes the following steps:

[0063] S1: Obtain the meeting document to be processed and its associated meeting document corpus; for any term in the meeting document to be processed, calculate the position and region-weighted word frequency of the term based on the preset document region weight coefficient and the inverse position decay function of the term within the document region. The document area includes a title, a summary, and the main text, with the weight coefficients of the three decreasing sequentially.

[0064] Conference documents in PDF or XML format were batch-acquired from academic databases such as IEEE Xplore and ACM Digital Library using web crawlers or API interfaces to form a corpus. Each document was then parsed to extract the text content of three structured regions: title, abstract, and body. Next, natural language processing tools, such as Jieba for Chinese, were used to segment the text in each region, and a predefined stop word list was loaded to remove meaningless terms such as modal particles and conjunctions. Simultaneously, stemming or lemmatization was performed on English terms to complete data preprocessing. Weighting coefficients were assigned to the title, abstract, and body regions, for example, 1.0, 0.8, and 0.5 respectively. The reverse position decay function was defined as the reciprocal of the position number of each occurrence of a term within that region. Figure 2 A schematic diagram of a positional decay function; weighted term frequency of a term. This is the sum of the products of the decay function values ​​and the corresponding weight coefficients for all occurrences of the term in the title, abstract, and body text regions; for example, if the term model appears in the 5th position of the title and the 20th position of the body text, then its... It is 9 / 40.

[0065] S2: Using a document vectorization model, convert each document in the conference document corpus into a document vector, and perform a hierarchical clustering algorithm on the document vectors to obtain multiple document topic clusters; based on the partitioning results of the document topic clusters, calculate the in-class inverse document frequency of the term within its respective cluster. ;

[0066] A pre-trained BERT model was used as the document vectorization tool. The full text of each pre-processed document was input into the model, and the output vector corresponding to its CLS flag was extracted as the semantic vector representation of the document. Then, an agglomerative hierarchical clustering algorithm was applied to all document vectors, using Ward's minimum variance method as the distance metric between clusters. By analyzing the dendrogram and setting a distance threshold, the optimal number of clusters was determined, thereby dividing the corpus into multiple topic-related document sets, such as... Figure 3 As shown; in one embodiment, for any term, its intra-class inverse document frequency The term is calculated logarithmically as follows: the total number of documents in its cluster divided by the number of documents in that cluster that contain the term.

[0067] S3: Weighted word frequency based on the aforementioned location and region With the inverse document frequency within the class The product of the terms yields the initial weight of the term; based on the initial weight, core terms with weights greater than a first preset threshold are selected.

[0068] The calculated value of S1 Calculated with S2 Multiply to obtain the initial weight value of each term; sort all terms in the document from high to low according to the initial weight, and select the top 30 terms after sorting as core terms, with 30 being the first preset threshold.

[0069] S4: Construct a weighted co-occurrence network graph of the core terms, where nodes are core terms. When the distance between any two core terms in a document is less than a second preset threshold, establish a connection edge between the corresponding nodes. The weight of the edge is the point mutual information value of the two core terms.

[0070] The core terms selected by S3 are used as nodes in the network graph; a sliding window of size 5 is set as the sentence distance, i.e., the second preset threshold is 5; the entire document is traversed, and if two core terms appear in the same sliding window at the same time, an edge is established between their corresponding nodes; the weight of the edge is calculated using Point Mutual Information (PMI), which is the logarithm of the probability that the two core terms appear together in all sliding windows, divided by the product of their probabilities of appearing independently in any sliding window, where the probability is estimated by frequency.

[0071] S5: The initial weights of each core term are used as the initial weights in the weighted co-occurrence network graph. The weights of each core term node are iteratively updated according to the connection relationship of the nodes and the weights of the edges in the graph using a preset graph sorting iteration algorithm. When the weights of each node converge or reach the preset number of iterations, the obtained weights are used as the weights of the core terms.

[0072] The weighted TextRank algorithm is used as the graph sorting iterative algorithm. The initial weights of each core term calculated by S3 are assigned to the corresponding nodes in the graph as their initial scores. The damping coefficient is set to 0.85, and then iterative calculation is performed. In each iteration, the updated score of a node consists of two parts: one part is a fixed random jump probability, and the other part is the result of weighted summation of the scores of all its neighboring nodes according to the PMI weights of the connecting edges. This process is continuously iterated until the change in the scores of all nodes is less than a very small convergence threshold, such as 0.0001, or the preset maximum number of iterations, such as 100, is reached. At this point, the scores of each node are the final weights of the core terms.

[0073] In an optional embodiment, the position of the term and its region-weighted frequency are calculated based on a preset document region weight coefficient and a word's reverse position decay function within the document region. ,include:

[0074] For each occurrence of the term in the document, a single weighted value is calculated. The single weighted value is obtained by multiplying the weight coefficient of the region where the term is located by the positional decay value in the region.

[0075] The position and region-weighted word frequency of the terms This is the sum of all single-weighted values ​​of the term in the document;

[0076] The weighting coefficients for the document regions are as follows: title weighting coefficient is 1.0, abstract weighting coefficient is 0.8, and body text weighting coefficient is 0.5; the position decay value is obtained through a reverse position decay function. The calculation is performed where p is the position number of the term within the region, counted from back to front or from front to back, starting from 1.

[0077] For example, a term has a weighting factor of 1.0 if it appears in the title area, 0.8 if it appears in the abstract area, and 0.5 in the body text area. Furthermore, the later the term appears in the text, the smaller the attenuation value becomes; for example, the last word in the area, with position index p1, has an attenuation value of 0.5. Multiplying the region weight by the position attenuation value yields a single-weighted value. Then, summing the single-weighted values ​​for all occurrences of the term in the document gives the final position-region weighted term frequency. In one embodiment, p can also be used as the position number of the term in the region, counted from front to back. Whether to use back to front or front to back depends on the specific format of the document. For example, in some meeting conclusions or summary paragraphs, the last sentence is the most important, so back to front is used; if the preceding sentences are the most important, front to back is used.

[0078] In an optional embodiment, the inverse document frequency of the term within its respective cluster is calculated based on the document topic clustering results. ,include:

[0079] The Doc2Vec model was used to train the conference document corpus, and each document in the corpus was converted into a document vector.

[0080] An agglomerative hierarchical clustering algorithm based on the Ward minimum variance criterion is applied to all document vectors to divide the documents into a predetermined number of document topic clusters;

[0081] The inverse document frequency of the term within its respective cluster The calculation formula is: ,in The total number of documents within the cluster C to which term t belongs. Let be the number of documents containing term t within cluster C.

[0082] The Doc2Vec model is used to process the document corpus, transforming each document into a vector. Then, a Ward minimum variance criterion clustering algorithm is applied to divide these document vectors into different topic clusters. When calculating the inverse document frequency within a cluster, taking a specific cluster as an example, assuming that the cluster contains 100 documents, the frequency is... If the value is 100, and a certain term t appears in 10 of those documents, then... The value is 10. This term's... The value is the logarithm of the quotient of 100 divided by 11, reflecting the rarity of a term within a specific topic cluster.

[0083] In an optional embodiment, the step of filtering out core terms with weights greater than a first preset threshold based on the initial weights includes:

[0084] Calculate the arithmetic mean of the initial weights of all terms. ;

[0085] The first preset threshold is set to the arithmetic mean. The preset multiple;

[0086] Terms with an initial weight greater than the first preset threshold are selected as core terms.

[0087] To select the most important core terms, it is necessary to calculate the arithmetic mean of the initial weights of all candidate terms. Set the first preset threshold to a specific multiple of this average value, such as 1.5 times. Identify all terms with an initial weight greater than 1.5 times the arithmetic mean as core terms, while filtering out terms with lower weights, such as... Figure 4 As shown.

[0088] In an optional embodiment, constructing the weighted co-occurrence network graph of the core terms includes:

[0089] The second preset threshold is set to N. When the distance between any two core terms in the document is less than N, a connection edge is established between the corresponding nodes.

[0090] The edge weights are calculated using the point mutual information (PMI) value, and the calculation formula is as follows: ;

[0091] In this context, a sliding window is constructed using a continuous sequence of sentences of length N from the document. This represents the total number of sliding windows in the document. Core term and The number of sliding windows that appear together and Each contains core terms and The number of sliding windows.

[0092] The selected core terms are used as nodes in the network graph, and connections are established based on their co-occurrence relationships in the documents. For example, if two core terms are separated by less than N=5 sentences in the original text, an edge is drawn between the two nodes representing them. The weight of this edge, i.e., the strength of the association between the two terms, is quantified by the Point Mutual Information (PMI) value. It uses a sliding window of length 5 sentences to count the co-occurrence frequency of terms, thereby calculating the increase in their probability of co-occurrence relative to their individual probabilities of occurrence.

[0093] In an optional embodiment, the step of iteratively updating the weights of each core term node using a preset graph sorting iterative algorithm based on the connection relationships of nodes and the weights of edges in the graph includes:

[0094] A personalized graph sorting iterative algorithm is adopted, and the damping coefficient d is set to 0.85;

[0095] The formula for updating the weights of each core term node is as follows: ,in Let i be the weight of node i. Let i be the set of nodes adjacent to node i. Let be the weight of the edge connecting node i and node j. Let be the normalized initial weight of node i. The normalized initial weight is obtained by dividing the initial weight of the core term by the sum of the initial weights of all core terms.

[0096] The iteration termination condition is set as follows: the change in the weight value of all nodes in two consecutive iterations is less than the preset value, or the number of iterations reaches the preset number.

[0097] A personalized graph sorting iterative algorithm is used to update the final weights of each core term node in the network. In each iteration, the weight update of a node depends not only on its own initial weight but also on the weights of all its neighboring nodes. The damping coefficient d is set to 0.85, meaning that 85% of a node's weight comes from the contributions of its neighbors. The iteration process continues until the change in the weight values ​​of all nodes in two consecutive calculations is less than, for example, 0.0001, or the total number of iterations reaches, for example, 100. At this point, the process stops and the final weight results are output. After obtaining the weights of the core terms, other processing can be performed, including summary generation.

[0098] This invention also provides a conference information processing system based on big data, comprising the following modules:

[0099] The position- and region-weighted word frequency calculation module is used to acquire the meeting document to be processed and its associated meeting document corpus; for any word in the meeting document to be processed, based on a preset document region weight coefficient and a word's inverse position decay function within the document region, the position- and region-weighted word frequency of the word is calculated. The document area includes a title, a summary, and the main text, with the weight coefficients of the three decreasing sequentially.

[0100] The intra-class inverse document frequency (INV) calculation module is used to convert each document in the conference document corpus into a document vector using a document vectorization model, and to perform a hierarchical clustering algorithm on the document vectors to obtain multiple document topic clusters; based on the partitioning results of the document topic clusters, the module calculates the intra-class INV of the term within its respective cluster. ;

[0101] The core term filtering module is used to weight term frequencies based on the location and region. With the inverse document frequency within the class The product of the terms yields the initial weight of the term; based on the initial weight, core terms with weights greater than a first preset threshold are selected.

[0102] The network graph construction module is used to construct a weighted co-occurrence network graph of the core terms. The nodes in the graph are core terms. When the sentence distance between any two core terms in the document is less than a second preset threshold, a connection edge is established between the corresponding nodes. The weight of the edge is the point mutual information value of the two core terms.

[0103] The core term weight calculation module is used to take the initial weight of each core term as the initial weight in the weighted co-occurrence network graph. Through a preset graph sorting iteration algorithm, the weight of each core term node is iteratively updated according to the connection relationship of the nodes in the graph and the weight of the edges. When the weight of each node converges or reaches the preset number of iterations, the obtained weight is used as the weight of the core term.

[0104] In an optional embodiment, the position of the term and its region-weighted frequency are calculated based on a preset document region weight coefficient and a word's reverse position decay function within the document region. ,include:

[0105] For each occurrence of the term in the document, a single weighted value is calculated. The single weighted value is obtained by multiplying the weight coefficient of the region where the term is located by the positional decay value in the region.

[0106] The position and region-weighted word frequency of the terms This is the sum of all single-weighted values ​​of the term in the document;

[0107] The weighting coefficients for the document regions are as follows: title weighting coefficient is 1.0, abstract weighting coefficient is 0.8, and body text weighting coefficient is 0.5; the position decay value is obtained through a reverse position decay function. The calculation is performed where p is the position number of the term within the region, counted from back to front or from front to back, starting from 1.

[0108] In an optional embodiment, the inverse document frequency of the term within its respective cluster is calculated based on the document topic clustering results. ,include:

[0109] The Doc2Vec model was used to train the conference document corpus, and each document in the corpus was converted into a document vector.

[0110] An agglomerative hierarchical clustering algorithm based on the Ward minimum variance criterion is applied to all document vectors to divide the documents into a predetermined number of document topic clusters;

[0111] The inverse document frequency of the term within its respective cluster The calculation formula is: ,in The total number of documents within the cluster C to which term t belongs. Let be the number of documents containing term t within cluster C.

[0112] In an optional embodiment, the step of filtering out core terms with weights greater than a first preset threshold based on the initial weights includes:

[0113] Calculate the arithmetic mean of the initial weights of all terms. ;

[0114] The first preset threshold is set to the arithmetic mean. The preset multiple;

[0115] Terms with an initial weight greater than the first preset threshold are selected as core terms.

[0116] In an optional embodiment, constructing the weighted co-occurrence network graph of the core terms includes:

[0117] The second preset threshold is set to N. When the distance between any two core terms in the document is less than N, a connection edge is established between the corresponding nodes.

[0118] The edge weights are calculated using the point mutual information (PMI) value, and the calculation formula is as follows: ;

[0119] In this context, a sliding window is constructed using a continuous sequence of sentences of length N from the document. This represents the total number of sliding windows in the document. Core term and The number of sliding windows that appear together and Each contains core terms and The number of sliding windows.

[0120] In an optional embodiment, the step of iteratively updating the weights of each core term node using a preset graph sorting iterative algorithm based on the connection relationships of nodes and the weights of edges in the graph includes:

[0121] A personalized graph sorting iterative algorithm is adopted, and the damping coefficient d is set to 0.85;

[0122] The formula for updating the weights of each core term node is as follows: ,in Let i be the weight of node i. Let i be the set of nodes adjacent to node i. Let be the weight of the edge connecting node i and node j. Let be the normalized initial weight of node i. The normalized initial weight is obtained by dividing the initial weight of the core term by the sum of the initial weights of all core terms.

[0123] The iteration termination condition is set as follows: the change in the weight value of all nodes in two consecutive iterations is less than the preset value, or the number of iterations reaches the preset number.

[0124] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0125] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0126] The method and electronic device for providing product object information provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A big data-based conference information processing method, characterized by, The method comprises the following steps: S1: obtaining a conference document to be processed and a conference document corpus to which the conference document to be processed belongs; for any term in the conference document to be processed, calculating a position and region weighted term frequency of the term based on a preset document region weight coefficient and an inverse order position attenuation function of the term in the document region ; the document region includes a title, an abstract and a body, and the weight coefficients of the three are sequentially decreased; S2: converting each document in the conference document corpus into a document vector using a document vectorization model, and performing a hierarchical clustering algorithm on the document vectors to obtain a plurality of document topic clusters; based on a partition result of the document topic clusters, calculating an intra-cluster inverse document frequency of the word item within the cluster to which the word item belongs ; S3: multiplying the initial weight of the term by a weight of the term in the region based on the location and the inverse document frequency within the class of the term; filtering out core terms with weights greater than a first preset threshold based on the initial weights; S4: constructing a weighted co-occurrence network graph of the core terms, wherein the nodes in the graph are the core terms, and a connection edge is established between the corresponding nodes when the sentence distance of any two core terms in the document is less than a second preset threshold, and the weight of the edge is the point mutual information value of the two core terms; S5: taking the initial weights of the core terms as the initial weights in the weighted co-occurrence network graph, and iteratively updating the weights of the core term nodes according to the connection relationship of the nodes and the weight of the edge in the graph by using a preset graph ranking iteration algorithm, and taking the obtained weights as the weights of the core terms when the weights of the nodes converge or reach a preset iteration number.

2. The method of claim 1, wherein, The position and region weighted term frequency of the word item is calculated based on a preset document region weight coefficient and an inverse order position decay function of the word item in the document region , comprising: For each occurrence of the term in the document, a single weighting value is calculated, which is obtained by multiplying the weight coefficient of the region where the term is located by the position attenuation value in the region; position and region weighted term frequency of the term is the sum of all single-weighted values of the term in the document; Wherein, the weight coefficient of the document area is: the title weight coefficient is 1.0, the abstract weight coefficient is 0.8, and the text weight coefficient is 0.5; the position attenuation value is obtained by an inverse sequence position attenuation function The position sequence number p of the word item in the area is counted from back to front or from front to back, and p is counted from 1.

3. The method of claim 1, wherein, calculating an intra-cluster inverse document frequency of the word item within the cluster to which the word item belongs based on the partitioning of the document topic clusters comprising: training a conference document corpus by using a Doc2Vec model, and converting each document in the corpus into a document vector; applying a Ward minimum variance criterion-based agglomerative hierarchical clustering algorithm to all the document vectors to divide the documents into a preset number of document topic clusters; intra-class inverse document frequency of the term within the cluster of the class The formula for calculating the intra-class inverse document frequency of a term t within a cluster C is: where is the total number of documents within the cluster C to which the term t belongs, is the number of documents within the cluster C that contain the term t.

4. The method of claim 1, wherein, The filtering out of core terms with weights greater than a first preset threshold based on the initial weights comprises: computing an arithmetic mean of all initial weights of the terms ; setting the first preset threshold value as a preset multiple of the arithmetic mean value of the first preset threshold value. filtering out core terms with initial weights greater than the first preset threshold.

5. The method of claim 1, wherein, The constructing of the weighted co-occurrence network graph of the core terms comprises: setting the second preset threshold as N, and establishing a connection edge between the corresponding nodes when the sentence distance of any two core terms in the document is less than N; The weight of the edge is calculated by using the point mutual information value PMI, and the calculation formula is: ; wherein a sliding window is formed by a sequence of sentences of length N in the document, is the total number of sliding windows in the document, is the number of sliding windows containing the core term and is the number of sliding windows containing the core term and is the number of sliding windows containing the core term and is the number of sliding windows containing the core term 6. The method of claim 1, wherein, The iteratively updating of the weights of the core term nodes according to the connection relationship of the nodes and the weight of the edge in the graph by using a preset graph ranking iteration algorithm comprises: using a personalized graph ranking iteration algorithm and setting a damping coefficient d; The weight updating formula of each core term node is: wherein is a weight of a node i, is a set of nodes adjacent to the node i, is a weight of an edge connecting the node i and the node j, is a normalized initial weight of the node i, the normalized initial weight being obtained by dividing the initial weight of the core term by a sum of the initial weights of all core terms; The iteration termination condition is set as: the weight values of all the nodes change by less than a preset value in two consecutive iterations, or the iteration number reaches a preset number. 7.A big data-based conference information processing system, characterized by, The method comprises the following modules: The position and region weighted word frequency calculation module is configured to acquire a conference document to be processed and a conference document corpus to which the conference document to be processed belongs; for any word in the conference document to be processed, a position and region weighted word frequency of the word is calculated based on a preset document region weight coefficient and an inverse sequence position attenuation function of the word in the document region The document region includes a title, an abstract and a body, and the weight coefficients of the three are sequentially decreased. an intra-class inverse document frequency calculation module configured to convert each document in the conference document corpus into a document vector using a document vectorization model, and perform a hierarchical clustering algorithm on the document vectors to obtain a plurality of document topic clusters; and calculate an intra-class inverse document frequency of the word item within the cluster to which the word item belongs based on a division result of the document topic clusters ; a core term filtering module for calculating an initial weight of each term based on a product of a term frequency weighted by the location and the region and an inverse document frequency within the class of the term filtering out core terms with weights greater than a first preset threshold based on the initial weights; a network graph construction module, configured to construct a weighted co-occurrence network graph of the core terms, wherein the nodes in the graph are the core terms, and a connection edge is established between the corresponding nodes when the sentence distance of any two core terms in the document is less than a second preset threshold, and the weight of the edge is the point mutual information value of the two core terms; a core term weight calculation module, configured to take the initial weights of the core terms as the initial weights in the weighted co-occurrence network graph, and iteratively update the weights of the core term nodes according to the connection relationship of the nodes and the weight of the edge in the graph by using a preset graph ranking iteration algorithm, and take the obtained weights as the weights of the core terms when the weights of the nodes converge or reach a preset iteration number.

8. The system of claim 7, wherein, The position and region weighted term frequency of the word item is calculated based on a preset document region weight coefficient and an inverse order position decay function of the word item in the document region , comprising: For each occurrence of the term in the document, a single weighting value is calculated, which is obtained by multiplying the weight coefficient of the region where the term is located by the position attenuation value in the region; position and region weighted term frequency of the term is the sum of all single-weighted values of the term in the document; Wherein, the weight coefficient of the document area is: the title weight coefficient is 1.0, the abstract weight coefficient is 0.8, and the text weight coefficient is 0.5; the position attenuation value is obtained by an inverse sequence position attenuation function is calculated, where p is the position sequence number of the word item in the area according to the backward or forward sequence, and p is counted from 1.

9. The system of claim 7, wherein, calculating an intra-cluster inverse document frequency of the word item within the cluster to which the word item belongs based on the partitioning of the document topic clusters comprising: The Doc2Vec model is used to train a conference document corpus, and each document in the corpus is converted into a document vector; A condensed hierarchical clustering algorithm based on Ward's minimum variance criterion is applied to all document vectors to divide the documents into a preset number of document topic clusters; intra-class inverse document frequency of the term within the cluster of the class The formula for calculating the intra-class inverse document frequency of a term t within a cluster C is: where Nc(t) is the total number of documents within the cluster C to which the term t belongs, Nc(t) is the number of documents within the cluster C that contain the term t.

10. The system of claim 7, wherein, The core word items with a weight greater than a first preset threshold are screened out based on the initial weight, and the screening includes: computing an arithmetic mean of all initial weights of the terms ; setting the first preset threshold value as a preset multiple of the arithmetic mean value ​ The word items with an initial weight greater than the first preset threshold are screened as core word items.

Images