A method and system for processing ancient document climate record data

Abstract

The present invention discloses a method and system for processing ancient document climate record data, comprising: preprocessing an original document in the form of classical Chinese to obtain climate characteristic words contained in the original document; determining keywords from the climate characteristic words contained in the original document, and obtaining a two-dimensional word frequency matrix based on the keywords; inputting the two-dimensional word frequency matrix into a topic model of ancient document climate record data to obtain the number of climate record topics involved in the original document, keywords of each climate record topic and the probability that the original document belongs to each climate record topic; determining the specific meaning of the climate record topic based on the keywords of the climate record topic with the highest probability, and obtaining the climate event type of the original document.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer natural language processing, and specifically relates to a method and system for processing ancient document climate record data. Background Art

[0002] Ancient document climate records refer to text data in the form of classical Chinese that record various climate events such as drought, rainstorm, high temperature, cold, etc. Compared with substitute data, the records in ancient document climate records have clearer climate significance and time resolution. However, in the past, when scholars used ancient document climate records, they mostly manually consulted the ancient document climate records or data compilation results of the research period and research area, and the process was often very complicated and repetitive. Moreover, these climate events are often recorded in historical books and local chronicles. The recording languages ​​and recording methods in different historical books and local chronicles are often quite different. Even for the same historical period and the same climate event, different ancient document climate records often have different focuses. As a result, when different scholars interpret the climate events in the ancient document climate records, the climate record types and climate event levels they may obtain are not exactly the same. This greatly hinders the application of ancient document climate records in the study of ancient climate change. Therefore, a convenient, fast and unified method for collating ancient document climate records is urgently needed.

[0003] With the maturity of computer science, especially natural language processing related technologies, many studies have attempted to use topic models such as LDA (latent Dirichlet allocation) to classify Chinese texts. However, the topic modeling of ancient document climate records is relatively lacking, and is mainly focused on the sentiment analysis of classical poetry, author attribution and other fields. The topic modeling of ancient document climate records needs further exploration.

[0004] When extracting keywords from documents, the existing processing methods use the TF-IDF (term frequency-inverse document frequency) method to count the word frequency and select the words with larger TF-IDF values ​​in the document as the keywords of the document. However, this method is very dependent on the word segmentation results. If a word is split into two words during word segmentation, the two words cannot be glued together during keyword extraction, which may lead to the loss of key information in the document and affect subsequent topic modeling. In addition, the text data that the existing processing methods can process are usually modern Chinese and English. Due to the differences in stop words and syntactic structures, the existing processing methods cannot directly process ancient document climate record data. Summary of the invention

[0005] Purpose of the invention: In order to solve the problem that existing processing methods cannot directly process ancient document climate record data, and to solve the problem that existing processing methods overly rely on word segmentation results and thus have the risk of losing key information, the present invention proposes a processing method and system for ancient document climate record data.

[0006] Technical solution: A method for processing ancient document climate record data, comprising the following steps:

[0007] Step 1: Preprocess the original document in classical Chinese to obtain the climate characteristic words contained in the original document;

[0008] Step 2: Determine keywords from the climate characteristic words contained in the original document, and construct a two-dimensional word frequency matrix based on the keywords;

[0009] Step 3: Input the two-dimensional word frequency matrix into the topic model of the ancient document climate record data to obtain the number of climate record topics involved in the original document, the keywords of each climate record topic, and the probability that the original document belongs to each climate record topic;

[0010] Step 4: Determine the specific meaning of the climate record topic based on the keywords of the climate record topic with the highest probability, and obtain the climate event type of the original document.

[0011] Furthermore, the aforementioned preprocessing of the original document in the form of classical Chinese to obtain the climate characteristic words contained in the original document specifically includes:

[0012] Perform data cleanup on the original documents in classical Chinese and filter out invalid characters;

[0013] Segment the original document after data cleaning to obtain multiple words;

[0014] Use the self-built stop word list of classical Chinese to filter the multiple words obtained by word segmentation to obtain the filtered words;

[0015] Tag the filtered words with parts of speech, and further filter the filtered words according to their parts of speech to obtain secondary filtered words;

[0016] Entity recognition is performed on the secondary filtered words, and according to the entity recognition results, the secondary filtered words are filtered to obtain the climate characteristic words contained in the original document.

[0017] Further, the self - built classical Chinese stop - word list includes: '之' (zhi), '兹' (zi), '此' (ci), '彼' (bi), '谁' (shui), '孰' (shu), '何' (he), '奚' (xi), '曷' (he), '焉' (yan), '或' (huo), '有' (you), '某' (mou), '莫' (mo), '弗' (fu), '毋' (wu), '勿' (wu), '不' (bu), '得' (de), '亦' (yi), '乃' (nai), '於' (yu), '于' (yu), '乎' (hu), '在' (zai), '而' (er), '以' (yi), '为' (wei), '其' (qi), '唯' (wei).

[0018] Further, to determine keywords from the climate - feature words contained in the original document and construct a two - dimensional word - frequency matrix based on the keywords, the specific operations include:

[0019] Using the TextRank algorithm, each climate - feature word is regarded as a node in pagerank; according to the nodes, the original document is segmented to obtain multiple words;

[0020] Set the window size, and determine the co - occurrence relationship between each pair of words through the window. There is an undirected and unweighted edge between the nodes corresponding to two words with a co - occurrence relationship, and a word - connection graph is drawn accordingly;

[0021] Based on the word - connection graph, calculate the weights of each word according to the following formula:

[0022]

[0023] Among them, represents the weight of word in the current round of iteration, represents the weight of word in the current round of iteration, represents the damping coefficient; represents the set of words that have a co - occurrence relationship with word ; represents the set of words in that have a co - occurrence relationship with each word; represents the weight of the edge connecting word and word represents the weight of the edge connecting word and word ; represents any word in the set of words that have a co - occurrence relationship with word ; represents any word that has a co - occurrence relationship with word ;

[0024] Sort the weights of the words contained in the original document, and take the words with weights greater than the set weight threshold as keywords;

[0025] A two-dimensional word frequency matrix is ​​constructed based on the keywords.

[0026] Furthermore, only when there are multiple adjacent keywords, the multiple adjacent keywords are merged into one keyword.

[0027] Furthermore, the topic model of the ancient document climate record data is obtained according to the following steps:

[0028] Step 1: Build the latent Dirichlet allocation model according to the following sub-steps;

[0029] Sub-step 1: The probability of each document belonging to each climate record topic is expressed as:

[0030]

[0031]

[0032] In the formula, Representation Document The distribution parameters of climate record themes, express The hyperparameters of the distribution, Indicates the number of documents; ;

[0033] Sub-step 2: The keyword distribution probability of each climate record theme is:

[0034]

[0035]

[0036] In the formula, Indicates the climate record subject The keyword distribution parameters, express The hyperparameters of the distribution, , represents the number of climate record themes;

[0037] Sub-step 3: Assume that the total number of keywords in each document is , taking the probability that a document belongs to a certain climate record topic as the probability that each keyword in the document belongs to the climate record topic, the process obeys the multinomial distribution, expressed as:

[0038]

[0039] In the formula, Representation Document Keywords The corresponding multinomial distribution of climate record themes;

[0040] Sub-step 4: Restore the original document based on the probability that each document belongs to each climate record topic and the probability that each keyword in the document belongs to each climate record topic. This process follows a multinomial distribution and is expressed as:

[0041]

[0042] In the formula, Representation Document Keywords in The multinomial distribution of keywords in the corresponding climate record topic, where the document Keywords in The corresponding multinomial distribution of climate record topics is given by get, Representation Document Keywords Corresponding multinomial distribution of climate record topics Keyword distribution parameters of ;

[0043] Sub-step 5: For any document, the probability of completely restoring the original document is expressed as:

[0044]

[0045] in, Representation Document Keywords The corresponding multinomial distribution of climate record topics, Representation Document of distributed, Representation Document Keywords in The corresponding multinomial distribution of keywords in the climate record topics;

[0046] Step 2: Use the grid search method to iteratively optimize the constructed latent Dirichlet allocation model to obtain the topic model of ancient document climate record data.

[0047] Furthermore, after step 4, the following steps are also included: performing word segmentation processing on the original document to obtain multiple words; according to the climate event type of the original document, using the corresponding pre-created climate event level screening terms to match the multiple words obtained by the word segmentation processing; if there is any word corresponding to a certain climate event level screening term, the corresponding climate event level label is assigned to the original document; if not, a normal year label is assigned to the document; if there are multiple words corresponding to multiple climate event level screening terms, the label assigned to the original document is the highest level of climate event level.

[0048] Furthermore, after step 4, the method further includes the following steps:

[0049] Perform word segmentation and entity recognition on the original document to identify the ancient place names contained in the original document;

[0050] According to the correspondence between ancient place names and current place names, the ancient place names are replaced with the longitude and latitude of the current place names;

[0051] By using GIS software, the original document is loaded onto the map as point data according to the longitude and latitude of the current place name, and attribute information is added to the point data, wherein the attribute information includes: climate event type.

[0052] The present invention discloses a processing system for ancient document climate record data, comprising:

[0053] A climate event processing module is used to process the original document in the form of classical Chinese using a method for processing ancient document climate record data to determine the climate event type of the original document;

[0054] The climate event classification module is used to perform word segmentation processing on the original document to obtain multiple words. According to the climate event type of the original document, the corresponding pre-created climate event level screening terms are used to match the multiple words obtained by the word segmentation processing. If there is any word corresponding to a certain climate event level screening term, the corresponding climate event level label is assigned to the original document; if not, the document is assigned a normal year label; if there are multiple words corresponding to multiple climate event level screening terms, the label assigned to the original document is the highest level of climate event level; the method for processing ancient document climate record data is the above-mentioned method for processing ancient document climate record data disclosed above.

[0055] The present invention discloses a processing system for ancient document climate record data, comprising:

[0056] A climate event processing module is used to process the original document in the form of ancient document classical Chinese using a method for processing ancient document climate record data to determine the climate event type of the original document;

[0057] The visualization module is used to perform word segmentation and entity recognition on the original document, identify the ancient place names contained in the original document; replace the ancient place names with the longitude and latitude of the current place names according to the correspondence between the ancient place names and the current place names; load the original document as point data onto the map according to the longitude and latitude of the current place names through GIS software, and add attribute information to the point data, wherein the attribute information includes: climate event type;

[0058] The method for processing climate record data from ancient documents is the method for processing climate record data from ancient documents disclosed above.

[0059] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0060] (1) The method of the present invention uses the TextRank algorithm to select words with high importance as keywords according to their importance in the sentence. When several keywords are adjacent, these keywords are automatically merged into one keyword, thereby reducing excessive reliance on the original word segmentation results.

[0061] (2) The method of the present invention converts natural language data into mathematical language data by preprocessing the ancient document climate record data, and then inputs the mathematical language data into the latent Dirichlet allocation model. This process can be understood as using the latent Dirichlet allocation model to organize and analyze various ancient document climate record data, and obtain the organized analysis results, that is, the output of the latent Dirichlet allocation model, which is stored in the database with a preset structure, so as to achieve the unification of the climate event type, climate event level and location information of the ancient document climate record data, facilitate scientific research query and analysis, and promote the application of ancient document climate record data in analyzing the characteristics of ancient Chinese climate;

[0062] (3) The method of the present invention solves the problem that the recording languages ​​and recording methods in different historical books and local chronicles are often quite different. Even for the same historical period and the same climate event, the records of different ancient document climate records often have different emphases, resulting in different scholars interpreting climate events in ancient document climate records. The climate record types and climate event levels may not be exactly the same. Specifically, taking drought as an example, when a drought event occurs, the records of different materials have different emphases. For example, the grain department may record "bad year" or "poor harvest", the meteorological department may record "no rain since winter", and other departments or historical books may record "grass withering" or "praying for rain". Although different historical books and local chronicles will record the impact of drought on different aspects, these words used in the records will be connected with "drought", and will be identified as keywords with high weights. Therefore, the present invention can assist in identifying the type of climate event records in ancient document climate records to a certain extent through the recognition and interpretation of the corresponding words of the climate event level;

[0063] (4) The method of the present invention processes the document of the determined climate record theme to obtain the climate record theme to which the document belongs, thereby realizing the organization of climate record data in ancient documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flowchart of a method for processing ancient document climate record data proposed by the present invention;

[0065] Figure 2Concept model diagram for topic modeling based on TextRank and LDA;

[0066] Figure 3 This is a diagram of word connections after word segmentation. DETAILED DESCRIPTION

[0067] The technical solution of the present invention is now further described in conjunction with the accompanying drawings and embodiments.

[0068] Example 1: Figure 1 As shown, this embodiment proposes a method for processing ancient document climate record data, which mainly includes the following steps:

[0069] Step 1: Get There are pieces of ancient document climate record data, each of which is stored in the corpus in the form of a document. Due to the limitations of stop words and syntactic structure, the latent Dirichlet allocation model cannot be directly used for topic modeling of ancient document climate record data. Therefore, it is necessary to preprocess each piece of ancient document climate record data to convert natural language data into mathematical language data. For the convenience of expression, the ancient document climate record data stored in the corpus in the form of a document is now called the original document:

[0070] Clean the original document and filter out invalid characters;

[0071] Segment the original document after data cleaning to obtain multiple words;

[0072] Use the self-built stop word list of classical Chinese to filter the multiple words obtained by word segmentation to obtain the filtered words;

[0073] Tagging the filtered words with parts of speech, further filtering the filtered words according to their parts of speech, filtering out interference words such as prepositions, pronouns, spaces, function words, adverbs, and numerals that have no climate-indicative meaning, and obtaining secondary filtered words;

[0074] The named entity recognition module of Python is used to perform entity recognition on the secondary filtered words. According to the entity recognition results, that is, whether it is a proper noun, specific named entities such as specific names of people, places, things and dates are filtered out from the secondary filtered words to obtain words without sentence structure and function words, and only meaningful climate characteristic words are retained.

[0075] Using the graph-based ranking algorithm for keyword extraction (TextRank algorithm), each climate feature word is used as a node in PageRank. First, the node is used to segment the original document, and the window size is set to k, the co-occurrence relationship between each word is determined through the window. There is an undirected and unweighted edge between the nodes corresponding to the two words with co-occurrence relationship, so as to draw a word connection graph.

[0076] Based on the word connection graph, the weight of each word in the sentence is calculated according to the following formula:

[0077]

[0078] in, Expressing words The weights in this round of iteration are Expressing words In the weights iterated in this round, the sum on the right represents the contribution of each adjacent word to the importance of the current word. Represents the damping coefficient, used for smoothing, generally set to 0.85; Expression and words A collection of words that have co-occurrence relationships; express The set of words that co-occur with each word in the Expressing words With words The weight of the edge connecting Expressing words and words The weight of the edge connecting Expression and words Any word in the set of words with co-occurrence relationship, Expression and words Any word with co-occurrence relationship;

[0079] The weight of each word is calculated, and the weight represents the importance of the word. The most important words can be used as keywords, so that the keyword extraction is completed according to the weight of each word.

[0080] After keyword extraction, you can get If there are several adjacent keywords in the original document, these adjacent keywords will be merged to become one keyword. The merging of keywords can overcome the limitation of over-reliance on word segmentation results of other algorithms such as TP-IDF algorithm to a certain extent, and can also enhance the connotation of keywords and better reflect the information contained in the original document.

[0081] A two-dimensional word frequency matrix is ​​established using keywords to convert natural language data into mathematical language data.

[0082] Step 2: Construct a latent Dirichlet allocation model. The specific operations include:

[0083] Assume that each original document contains keywords, and the number of climate record topics is Based on this, construct Figure 2 The Latent Dirichlet Allocation Topic Model shown; Figure 2 The directed edges shown in represent probabilistic dependencies, the rectangular box represents the repetition of calculations within the rectangular box, and the number in the lower right corner of the rectangular box represents the number of repetitions.

[0084] The climate record topic distribution of each document follows the Dirichlet distribution, which is expressed as:

[0085]

[0086] In the formula, Representation Document The distribution parameters of climate record themes, , express Hyperparameters of the distribution, which are usually assumed to be 1 in the absence of prior knowledge; Figure 2 The displayed content is: point to And repeat times, indicating that according to the hyperparameters Generated Thematic distribution of climate records for documents.

[0087] The probability of each document belonging to each climate record topic is expressed as:

[0088]

[0089] In the formula, Representation Document The distribution parameters of climate record themes, .

[0090] The keyword distribution of each climate record topic follows the Dirichlet distribution, which is expressed as:

[0091]

[0092] In the formula, Indicates the climate record subject The keyword distribution parameters, express Hyperparameters of the distribution, which are usually assumed to be 1 in the absence of prior knowledge; Figure 2 The content shown is: Hyperparameters point to And repeat times, indicating that according to the hyperparameters generate Keyword distribution of climate record topics.

[0093] The keyword distribution probability for each climate record theme is:

[0094]

[0095] In the formula, Indicates the climate record subject The keyword distribution parameters, .

[0096] The probability that a document belongs to a certain climate record topic is taken as the probability that each keyword in the document belongs to the climate record topic. This process follows a multinomial distribution and is expressed as:

[0097]

[0098] In the formula, Representation Document Keywords The corresponding multinomial distribution of climate record themes; Figure 2 The displayed content is: point to And repeat times, indicating that according to Generated The multinomial distribution of climate record topics corresponding to the keywords.

[0099] According to the probability that each keyword in the document belongs to each climate record theme and the keyword distribution parameters of each climate record theme, the original document is restored. This process obeys the multinomial distribution and is expressed as:

[0100]

[0101] In the formula, Representation Document Keywords in The multinomial distribution of keywords in the corresponding climate record topic, where the document Keywords in The corresponding multinomial distribution of climate record topics is given by get, Representation Document Keywords Corresponding multinomial distribution of climate record topics Keyword distribution parameters of ;

[0102] Figure 2 The displayed content is: point to ,and indivual Also points to , indicating that the document Keywords Corresponding climate record topics and documents Keywords Keyword distribution parameters corresponding to the climate record theme Co-generation .

[0103] Therefore, for any document, the probability of obtaining the correct correspondence between the new document and the original document is:

[0104]

[0105] in, Representation Document Keywords The corresponding multinomial distribution of climate record topics, Representation Document of distributed, Representation Document Keywords in The corresponding multinomial distribution of keywords in the climate record topic. In summary, the probability of completely restoring the original document is:

[0106]

[0107] in, Indicates that the original document is in the hyperparameter of Distribution and multinomial distribution of climate record topics corresponding to keywords in the original document with climate record topics in hyperparameters of Distribution and multinomial distribution of keywords in climate record topics The joint probability density distribution under , each distribution is independent of each other.

[0108] The probability of completely recovering the original document can be calculated by Sampling method for estimation.

[0109] Step 3: Use the corpus to train the latent Dirichlet allocation model constructed in step 2 to obtain the topic model of the ancient document climate record data. The specific operations include:

[0110] The two-dimensional word frequency matrix obtained based on the corpus is input into the latent Dirichlet allocation model constructed in step 2. The hyperparameter combinations involved in the latent Dirichlet allocation model are iteratively optimized through the grid search method to maximize the probability of completely restoring the original document. The hyperparameter combination with the highest probability of completely restoring the original document is the optimal hyperparameter combination, and the latent Dirichlet allocation model corresponding to the optimal hyperparameter combination is the topic model of the ancient document climate record data.

[0111] The combination of hyperparameters involved in the latent Dirichlet allocation model is determined by the number of climate record topics. and learning rate .

[0112] Based on the keywords of each climate record theme, the specific meaning of the climate record theme, that is, the climate event type, can be determined. The climate event type is used as the label of the original document and stored in the database with a preset structure mapping, thereby determining the climate event type of the ancient document climate record data.

[0113] Step 4: Obtain the ancient document climate record data for the climate event type to be classified, and preprocess the ancient document climate record data as shown in step 1 to obtain a two-dimensional word frequency matrix corresponding to the ancient document climate record data;

[0114] Step 5: Input the two-dimensional word frequency matrix corresponding to the climate record data of the ancient document into the topic model of the climate record data of the ancient document, and output the number of climate record topics contained in the climate record data of the ancient document and the keywords corresponding to each climate record topic; take the climate record topic with the highest probability as the climate record topic corresponding to the climate record data of the ancient document, and determine the specific meaning of the climate record topic based on the first multiple keywords of the climate record topic, thereby determining the climate event type of the climate record data of the ancient document.

[0115] This embodiment uses TextRank keyword extraction and Latent Dirichlet Allocation topic modeling to extract records about climate events from ancient document climate records and classify climate event types. Specifically, the TextRank method is used to extract keywords from ancient document climate records. The idea of ​​TextRank is to build a network through the adjacent relationship between words, and then use PageRank to iteratively calculate the rank value of each word. The keywords can be obtained by sorting the rank values. Compared with the existing method of using words with larger TF-IDF as keywords of documents, the TextRank method can first extract several keywords based on the adjacent relationship between words. If there are several adjacent keywords in the original document, these keywords can be merged into one keyword, which overcomes the limitation of over-reliance on the original word segmentation results to a certain extent. And this embodiment constructs a set of stop word lists for classical Chinese, so that the method of this embodiment is suitable for classical Chinese processing.

[0116] Embodiment 2: Based on Embodiment 1, this embodiment proposes a method for identifying the intensity level of climate events based on the semantic differential method.

[0117] In ancient documents, climate records are usually recorded in classical Chinese, and are subjective records and expressions of climate events. According to psychological theory, this subjective record and expression of climate events is not random. When describing climate events, the recorder often integrates multiple aspects of information to obtain a more complete impression, such as "this year there was drought in spring and summer, flood in autumn and winter, and locusts rose in large numbers", "In December, there were heavy snows for several days, and it was bitterly cold. The charcoal sellers in the capital sold them for 200 yuan each. Last winter was extremely cold, and there was still a flood. If the thaw occurs, it will surely overflow, causing more serious damage."

[0118] Therefore, this embodiment establishes a set of descriptive idioms for different intensity levels of different climate events to identify the intensity level of the ancient document climate record data that has determined the type of climate event. This embodiment divides the intensity level into 4 levels, with level 1 being a normal year, level 2 being slightly strong, level 3 being strong, and level 4 being extremely strong. The specific operations include: determining the descriptive idioms to be used according to the type of climate event in the ancient document climate record data;

[0119] The climate record data in ancient documents are segmented to obtain multiple words;

[0120] The words obtained by word segmentation are compared one by one with descriptive idioms. If there is a word corresponding to a descriptive idiom, the intensity level corresponding to the descriptive idiom is the intensity level of the climate event type; if not, the document is assigned a normal year label; if there are multiple intensity levels, the highest intensity level is taken as the intensity level of the climate event type.

[0121] Example 3: Based on Example 1 or Example 2, this example proposes a visualization method for ancient literature climate record data. The specific steps include: According to the correspondence between ancient place names and current place names, identify the ancient place names contained in each piece of ancient literature climate record data, and replace them with the longitude and latitude corresponding to the current place names, endowing each piece of ancient literature climate record data with longitude and latitude information. At the same time, store the intensity levels of the climate events obtained in Example 1 and / or the climate event types obtained in Example 2 as the attribute information of this ancient literature climate record data in the database, and perform geographical visualization. The advantage of doing this is that corresponding climate events, the intensity levels of corresponding climate events, the occurrence frequencies and distribution ranges of the intensities of corresponding climate events can be screened according to the attribute information.

[0122] Example 4: This example proposes a processing method for ancient literature climate record data, which mainly includes the following steps:

[0123] Step 1: Obtain the ancient literature climate record data set, and perform the following processing on the ancient literature climate record data in the ancient literature climate record data set:

[0124] For the convenience of expression, the ancient literature climate record data is now referred to as the original document;

[0125] Perform data cleaning on the original document, filtering out invalid characters such as numbers, English letters, etc., such as 1, #, *, %, $, etc.;

[0126] Perform word segmentation on the original document after data cleaning to obtain multiple words;

[0127] Use the self-built classical Chinese stop word list to filter out meaningless stop words from the multiple words obtained by word segmentation to obtain the filtered words; The stop words in the self-built classical Chinese stop word list in this example include: '之', '兹', '此', '彼', '谁', '孰', '何', '奚', '曷', '焉', '或', '有', '某', '莫', '弗', '毋', '勿', '不', '得', '亦', '乃', '於', '于', '乎', '在', '而', '以', '为', '其', '唯', etc.;

[0128] Perform part-of-speech tagging on the filtered words, and further filter the filtered words according to the part-of-speech of the words, filtering out interfering words such as prepositions, pronouns, spaces, function words, adverbs, numerals, etc. that have no climate indication meaning to obtain the secondarily filtered words;

[0129] The named entity recognition module (spacy module) of Python is used to perform entity recognition on the secondary filtered words. According to the entity recognition results, that is, whether it is a proper noun, specific named entities such as specific names of people, places, objects and dates are filtered out from the secondary filtered words to obtain words without sentence structure and function words, and only meaningful climate characteristic words are retained.

[0130] The above steps are now described with examples.

[0131] Suppose that the content of the climate record data in ancient documents is: "On December Jichou, it rained and the trees were frozen. On the month of Wuxu, frost and fog fell heavily, and the grass and trees were all like ice. On the morning of November 2nd, the sky was very foggy, and the trees were covered with ice. On the eleventh day, the frost and dew covered the grass and all turned into ice. On November Renzi, it rained and the trees were frozen. On Xinyou, it rained and the trees were frozen. On Guiyou, it rained and the trees were frozen. It was July, and it was as cold as deep winter. The cliffs were high and the valleys were steep. You couldn't see the sun when you looked up, and the cold was even worse. After leaving the valley and getting out of the flat land, the springs were clear and cold, and the grass was soft like velvet, which could be used for sleeping."

[0132] After filtering, the content obtained is: "In December, it rained on Jichou and iced the trees. In the month of Wuxu, there was heavy frost and fog and iced the trees. On the second day of November, the sky was very dark and foggy and iced the trees. On the eleventh day, there was frost and dew and iced the trees. In November, it rained on Renzi and iced the trees. In Xinyou, it rained on wood and ice. In Guiyou, it rained on wood and ice. In July, it was cold and deep in winter. Looking up at the high cliffs and steep valleys, you can see the sun. When you get out of the valley, you can find clear springs on the flat ground with cool grass and soft grass to sleep on."

[0133] After part-of-speech tagging and entity recognition filtering, the content obtained is: "Rain on the wood, frost, fog on the wood, heavy fog on the wood, frost on the wood, dew on the grass, rain on the wood, rain on the wood, rain on the wood, winter, high cliffs and steep valleys, cold water, clear springs, soft grass, to sleep on."

[0134] Step 2: Use the TextRank algorithm to treat each climate feature word as a node in PageRank. According to the node, perform word segmentation on each ancient document climate record data, and set the window size to k The purpose of the window is to select words in sequence. Words that exist in the same window are called co-occurrences. It can be simply understood that there is a connection between these words and there is an unweighted and undirected edge. In short, the window is used to determine the co-occurrence relationship of words. The window size is usually 2 to 4. If the window is too large, the word connection is too complex and it is not easy to converge.

[0135] The weight of each word is calculated as follows:

[0136]

[0137] in, Expressing words The weights in this round of iteration are Expressing words In the weights iterated in this round, the sum on the right represents the contribution of each adjacent word to the importance of the current word. Represents the damping coefficient, used for smoothing, generally set to 0.85; Expression and words A collection of words that have co-occurrence relationships; express The set of words that co-occur with each word in the Expressing words With words The weight of the edge connecting Expressing words and words The weight of the edge connecting Expression and words Any word in the set of words with co-occurrence relationship, Expression and words Any word with a co-occurrence relationship; e.g. Figure 3 As shown, take this sentence as an example: "That year, there was a severe drought and all the waterways were dry", after preprocessing it becomes "severe drought and dry waterways", and after word segmentation it becomes "severe drought and dry waterways", then when calculating the weight of the word "waterway", The word set represented is: {"drought" and "dryness"}; if the window size k When the window size is 2, there is no connection between "drought" and "dryness". k When it is 3, there is a connection between "severe drought" and "dryness"; for The word set represented is: the word set of "severe drought" and the word set of "dryness" in {"severe drought" and "dryness"}. In this example, the word set of "severe drought" only has {"waterway"} and the word set of "dryness" has {"waterway"}.

[0138] In the embodiment, the size of the given window is 2, and the sentence 1 "severe drought and dry waterways" and the sentence 2 "30,000 shi of drought relief grain" are slid in sequence, and the co-occurrence relationship between the words after word segmentation is determined line by line. For example, for sentence 2, the window 1 {drought relief grain}, the window 2 {30,000 shi of grain} and the window 3 {30,000 shi of relief grain} are obtained. Figure 3 The words and undirected edges shown. According to the above calculation, the weights of "severe drought and dry waterways" are 0.25676, 0.48648, 0.25676, and the weights of {30,000 shi of drought relief grain} are 0.17544, 0.32456, 0.32456, 0.17544. The weights of the words are sorted. The larger the weight, the higher the importance. Therefore, the words with large weights are taken as the keywords of the document.

[0139] The keywords of the sentence "Long drought, grass withered, taking advantage of the wind, arson, chaos, great, waterway, dry, pray for rain, drought begins" are [drought, grass, withered, taking advantage of the wind, arson, chaos, great, waterway, dry, pray for rain, drought begins, rain]; if there are several adjacent keywords in the original document, then these adjacent keywords are merged to form one keyword. Finally, the keywords of the sentence "Long drought, grass withered, taking advantage of the wind, arson, chaos, great, drought, waterway dry, drought begins" are [drought, grass withered, taking advantage of the wind, chaos, waterway dry, pray for rain, drought begins], which can overcome the limitations of the segmentation results and enhance the connotation of the keywords, so as to better reflect the main meaning of the climate record data in ancient documents.

[0140] Based on keywords, a two-dimensional word frequency matrix is ​​constructed to convert the natural language climate record data of ancient documents into mathematical language.

[0141] Step 3: Construct a latent Dirichlet allocation model. The specific operations include:

[0142] Assume that there are 16 documents in the ancient climate record data set, and 20 keywords are extracted from each document. Figure 2 The establishment of the latent Dirichlet allocation model shown in Figure 2 The directed edges shown in represent probabilistic dependencies, the rectangular box represents the repetition of calculations within the rectangular box, and the number in the lower right corner of the rectangular box represents the number of repetitions.

[0143] The climate record topic distribution of each document follows the Dirichlet distribution, which is expressed as:

[0144]

[0145] In the formula, Representation Document The distribution parameters of climate record themes, , express Hyperparameters of the distribution, which are usually assumed to be 1 in the absence of prior knowledge; Figure 2 The displayed content is: point to And repeat times, indicating that according to the hyperparameters Generated Thematic distribution of climate records for documents.

[0146] The probability of each document belonging to each climate record topic is expressed as:

[0147]

[0148] In the formula, Representation Document The distribution parameters of climate record themes, .

[0149] In this embodiment, the number of climate record topics is first set to It should be noted that these climate record themes do not necessarily correspond directly to drought, rainstorm, heat, etc. Due to the different number of climate record themes, there may be cross-integration or subdivision of climate record themes. In this step, the probability that the 16 documents in the corpus belong to different climate record themes will be determined. For example, when the number of climate record themes is set =3, document 5 in this embodiment: "In December, there were continuous heavy snows and bitter cold. Charcoal sellers in the capital city were charged 200 yuan each. Last winter was very cold and there were still floods. If the thaw occurs, it will surely overflow and cause more serious damage." The probabilities of belonging to the three climate record themes are 0.0312, 0.9376, and 0.0312.

[0150] The keyword distribution of each climate record topic follows the Dirichlet distribution, which is expressed as:

[0151]

[0152] In the formula, Indicates the climate record subject The keyword distribution parameters, express Hyperparameters of the distribution, which are usually assumed to be 1 in the absence of prior knowledge; Figure 2 The content shown is: Hyperparameters point to And repeat times, indicating that according to the hyperparameters generate Keyword distribution of climate record topics.

[0153] The keyword distribution probability for each climate record theme is:

[0154]

[0155] In each document, each keyword belongs to a certain climate record theme with a certain probability. When calculating, each keyword in each document has a clear climate record theme attribute, but the climate record theme attributes of these keywords in different documents may be different. It should be noted that some keywords, such as "reduced tax" and "great harvest", may not have a particularly clear meaning of indicating climate events, but in this step, they are still assigned a unique climate record theme attribute.

[0156] In this embodiment, the document has a certain probability of belonging to a certain climate record theme, so the keyword in the document is considered to belong to a certain climate record theme with this probability. Therefore, even the same keyword may be assigned different climate record themes in different documents. This process follows a multinomial distribution, which can be expressed as:

[0157]

[0158] In the formula, Representation Document Keywords The corresponding multinomial distribution of climate record themes; Figure 2 The displayed content is: point to And repeat times, indicating that according to Generated The multinomial distribution of climate record topics corresponding to the keywords.

[0159] According to the probability that each keyword in the document belongs to each climate record theme and the keyword distribution parameters of each climate record theme, the original document is restored. This process obeys the multinomial distribution and is expressed as:

[0160]

[0161] In the formula, Representation Document Keywords in The multinomial distribution of keywords in the corresponding climate record topic, where the document Keywords in The corresponding multinomial distribution of climate record topics is given by get, Representation Document Keywords Corresponding multinomial distribution of climate record topics Keyword distribution parameters of ;

[0162] Figure 2 The displayed content is: point to ,and indivual Also points to , indicating that the document Keywords Corresponding climate record topics and documents Keywords Keyword distribution parameters corresponding to the climate record theme Co-generation .

[0163] Therefore, for any document, the probability of obtaining the correct correspondence between the new document and the original document is:

[0164]

[0165] where represents the multinomial distribution of the climate record topics corresponding to the keywords in the document , represents the distribution of the document , represents the multinomial distribution of the keywords in the climate record topics corresponding to the keywords in the document .

[0166] In the embodiment, first, it is necessary to determine the keywords included in each document in the ancient literature climate record data set and the climate record topics corresponding to the keywords. For example, in Document 4 of this embodiment, some of the keywords "Shi Chi, official charcoal market, heavy snow, suffering from selling charcoal, weighing, money, last winter, freezing, thawing, snowing" correspond to "Climate Record Topic 2, Climate Record Topic 1, Climate Record Topic 1, Climate Record Topic 3, Climate Record Topic 1, Climate Record Topic 2, Climate Record Topic 1, Climate Record Topic 2, Climate Record Topic 3, Climate Record Topic 1". Then, according to the correspondence between the keywords and the climate record topics, the document is restored to obtain: "drought, rain and snow, official charcoal market, freezing, heavy snow, snowing, suffering from selling charcoal, praying for clear weather, continuous rain, relief". It can be seen that although the restored document is similar to the original document, the positions of the words and sentences have changed and even some words and sentences have been added or deleted.

[0167] In summary, the probability of completely restoring the original document is:

[0168]

[0169] where represents the joint probability density distribution of the distribution of the original document under the hyperparameter and the multinomial distribution of the climate record topics corresponding to the keywords in the original document , and the distribution of the climate record topics under the hyperparameter and the multinomial distribution of the keywords in the climate record topics . The distributions are independent of each other.

[0170] The probability of completely restoring the original document can be estimated by the method of sampling.

[0171] ​Step 4: Use the corpus to train the latent Dirichlet allocation model constructed in step 3 to obtain the topic model of the ancient document climate record data. The specific operations include:

[0172] The two-dimensional word frequency matrix obtained based on the ancient document climate record data set is input into the latent Dirichlet allocation model constructed in step 2. The hyperparameter combination involved in the latent Dirichlet allocation model is iteratively optimized through the grid search method to maximize the probability of completely restoring the original document. The hyperparameter combination with the highest probability of completely restoring the original document is the optimal hyperparameter combination, and the latent Dirichlet allocation model corresponding to the optimal hyperparameter combination is the topic model of the ancient document climate record data.

[0173] The probability of completely recovering a document depends on the hyperparameters of the latent Dirichlet allocation model. The more reliable the parameters are, the greater the probability of recovering the original document is, and the more reliable the climate record theme is.

[0174] The hyperparameters involved in the latent Dirichlet allocation model include the number of climate record topics and learning rate In an embodiment, a grid search method is used to search for the number of climate record topics. and learning rate Iterative optimization is performed, and the parameters with the highest probability of restoring the original document are the optimal hyperparameter combination.

[0175] In the grid search process, we first define the values ​​of different hyperparameters, including the learning rate The value ranges from 0.01 to 1, and the number of climate record topics The value of is 1 to 10. Then, initialize a latent Dirichlet allocation model and use the GridSearchCV class in the scikit-learn library to create a model object, passing in the latent Dirichlet allocation model to be searched and the value of the hyperparameter.

[0176] Next, call the fit method of the model object and pass in the ancient document climate record data set. In this step, GridSearchCV will automatically cross-validate all possible hyperparameter combinations, calculate the performance of each hyperparameter combination on the training data, and finally find the hyperparameter combination with the best performance in the cross-validation.

[0177] This embodiment uses the log likelihood score and perplexity to evaluate the performance of the trained latent Dirichlet allocation model.

[0178] The score of the log-likelihood estimate of the optimal latent Dirichlet allocation model trained in this embodiment is: -63877.2421, and the perplexity is: 65.2115, indicating that the topic model of the ancient document climate record data trained in this embodiment has good fitting ability and prediction performance.

[0179] Based on the topic model of ancient document climate record data, the number of climate record topics can be obtained and the previous Keywords. The keywords clarify the specific meaning of the climate record theme, determine the climate event type of the ancient document climate record data, and store the climate event type of the ancient document climate record data into the database with a preset structure mapping, thereby completing the classification of the climate event type of the ancient document climate record data.

[0180] Based on the topic model of the climate record data in ancient documents, as shown in Table 1, this embodiment obtains 3 climate record topics and the first 8 keywords of each climate record topic.

[0181] Table 1 Detailed list of the three climate record themes and the first eight keywords of each climate record theme

[0182]

[0183] In Table 2, each row represents the probability that a document belongs to a certain climate record topic. Taking the second row as an example, the probability that document 2 belongs to climate record topic 3 is 0.9782, which means that document 2 belongs to climate record topic 3. The specific meaning of climate record topic 3 can be determined based on all the keywords of climate record topic 3, but the first few keywords have a greater weight, so the specific meaning of climate record topic 3 can also be determined by the first 2 to 4 keywords.

[0184] Table 2 Probability table of 16 documents belonging to three climate record topics

[0185]

[0186] Step 5: Obtain the ancient document climate record data for the climate event type to be classified, and preprocess the ancient document climate record data as shown in step 1 to obtain a two-dimensional word frequency matrix corresponding to the ancient document climate record data;

[0187] Step 6: Input the two-dimensional word frequency matrix corresponding to the climate record data of the ancient document into the topic model of the climate record data of the ancient document, and output the number of climate record topics contained in the climate record data of the ancient document and the keywords corresponding to each climate record topic; take the climate record topic with the highest probability as the climate record topic corresponding to the climate record data of the ancient document, and determine the specific meaning of the climate record topic based on the first multiple keywords of the climate record topic, thereby determining the climate event type of the climate record data of the ancient document.

[0188] Step 7: Based on the semantic differential method, identify the intensity level of climate event types in ancient literature climate records, including:

[0189] Therefore, this embodiment establishes a set of descriptive idioms for different intensity levels of different climate events to identify the intensity level of the ancient document climate record data that has determined the type of climate event. This embodiment divides the intensity level into 4 levels, with level 1 being a normal year, level 2 being slightly strong, level 3 being strong, and level 4 being extremely strong. The specific operations include: determining the descriptive idioms to be used according to the type of climate event in the ancient document climate record data;

[0190] The climate record data in ancient documents are segmented to obtain multiple words;

[0191] The words obtained by word segmentation are compared one by one with descriptive idioms. If there is a word corresponding to a descriptive idiom, the intensity level corresponding to the descriptive idiom is the intensity level of the climate event type; if not, the document is assigned a normal year label; if there are multiple intensity levels, the highest intensity level is taken as the intensity level of the climate event type.

[0192] In this embodiment, descriptive idioms of different intensity levels of flood climate events are constructed, and the intensity levels of documents with climate record themes of floods are quantified. For example, "heavy rain for days, boats can't sail on land" corresponds to flood level 4; "sudden water rises in the mountains and ruins the fields" corresponds to flood level 3; "heavy rain in spring and summer damages crops" corresponds to flood level 2; "good harvest" and "prosperous year" with no flood records correspond to level 1, which is a normal year.

[0193] Step 8: According to the correspondence between ancient place names and current place names, the ancient place names contained in the ancient document climate record data are identified through word segmentation and named entity recognition, and replaced with the longitude and latitude corresponding to the current place names, and the longitude and latitude information of the ancient document climate record data is assigned. At the same time, the intensity level of climate events and climate event types is stored in the database as the attribute information of the ancient document climate record data for geographic visualization. The advantage of this is that the corresponding climate events, the intensity level of the corresponding climate events, the occurrence frequency and distribution range of the intensity of the corresponding climate events can be screened according to the attribute information.

Claims

1. A method for processing ancient document climate record data, characterized in that: It includes the following steps: Step 1: Preprocess multiple original documents in classical Chinese form to obtain the climate feature words contained in each original document; Step 2: Determine the keywords from the climate feature words contained in each original document. Only when there are multiple adjacent keywords, merge the multiple adjacent keywords into one keyword; construct a two-dimensional word frequency matrix based on the keywords; Step 3: Input the two-dimensional word frequency matrices of all the original documents into the topic model of ancient literature climate record data to obtain the number of climate record topics involved in the original documents, the keywords of each climate record topic, and the probabilities of the original documents belonging to each climate record topic; Step 4: Determine the specific meaning of the climate record topic according to the keywords of the climate record topic with the highest probability to obtain the climate event types of the original documents; The preprocessing of multiple original documents in classical Chinese form to obtain the climate feature words contained in each original document specifically includes the following operations: Perform data cleaning on the original documents in classical Chinese form to filter out invalid characters; Segment the original documents after data cleaning to obtain multiple words; Use the self-built stop word list for classical Chinese to filter the stop words from the multiple words obtained by segmentation to obtain the filtered words; Perform part-of-speech tagging on the filtered words, and further filter the filtered words according to the part-of-speech of the words to obtain the secondarily filtered words; Perform entity recognition on the secondarily filtered words, and filter the secondarily filtered words according to the entity recognition results to obtain the climate feature words contained in the original documents; The self-built stop word list for classical Chinese includes: '之', '兹', '此', '彼', '谁', '孰', '何', '奚', '曷', '焉', '或', '有', '某', '莫', '弗', '毋', '勿', '不', '得', '亦', '乃', '於', '于', '乎', '在', '而', '以', '为', '其', '唯'.

2. The method for processing ancient document climate record data according to claim 1, characterized in that: The determination of keywords from the climate feature words contained in the original documents and the construction of a two-dimensional word frequency matrix based on the keywords specifically include the following operations: Use the TextRank algorithm, with each climate feature word as a node in Pagerank; perform word segmentation processing on the original documents according to the nodes to obtain multiple words; Set the window size, determine the co-occurrence relationship between each pair of words through the window, and there is an undirected and unweighted edge between the nodes corresponding to the two words with a co-occurrence relationship, and thus draw a word connection graph; Based on the word connection graph, calculate the weights of each word according to the following formula: Among them, WS(V i ) indicates word V i In this round of iteration, the weight WS(V j ) indicates word V j In the weights iterated in this round, d represents the damping coefficient; In(V i ) indicates that the word V i A set of words with co-occurrence relationship; Out(V j ) represents In(V i ) is a set of words that have co-occurrence relationships with each word; ji V j With the word V i The weight of the edge connecting jk V k and the word V j The weight of the edge connecting j Expression and word V i Any word in the set of words with co-occurrence relationship, V k Expression and word V j Any word with co-occurrence relationship; Sort the weights of the words contained in the original documents, and use the words with weights greater than the set weight threshold as keywords; Construct a two-dimensional word frequency matrix based on the keywords.

3. The method for processing ancient document climate record data according to claim 1, characterized in that: The topic model of ancient literature climate record data is obtained according to the following steps: S00: Build a Latent Dirichlet Allocation model according to the following sub-steps; S00_1: The probability that each document belongs to each climate record topic is expressed as: i j ~Dirichlet(s) In the formula, θ j represents the climate record topic distribution parameter of document j, α represents the hyperparameter of Dirichlet distribution, M is the number of documents; j≤M; S00_2: The probability distribution of the keywords of each climate record topic is: In the formula, represents the keyword distribution parameter of climate record topic i, β represents the hyperparameter of Dirichlet distribution, i≤K, K represents the number of climate record topics; S00_3: Assume that the total number of keywords in each document is N m , taking the probability that a document belongs to a certain climate record topic as the probability that each keyword in the document belongs to the climate record topic, the process obeys the multinomial distribution, expressed as: With jt ~Multinomial(θ j ) In the formula, z jt represents the multinomial distribution of climate record topics corresponding to keyword t in document j; S00_4: According to the probability that each document belongs to each climate record topic and the probability that each keyword in the document belongs to each climate record topic, the original document is restored. This process follows a multinomial distribution and is expressed as: In the formula, w jt represents the multinomial distribution of keywords in the climate record topic corresponding to keyword t in document j, where the multinomial distribution of climate record topics corresponding to keyword t in document j is represented by z jt get, The multinomial distribution z of the climate record topics corresponding to keyword t in document j jt Keyword distribution parameters of ; S00_5: For any document, the probability of completely restoring the original document is expressed as: Among them, z jt represents the multinomial distribution of climate record topics corresponding to keyword t in document j, θ j represents the Dirichlet distribution of document j, w jt represents the multinomial distribution of keywords in the climate record topic corresponding to keyword t in document j; S10: The grid search method is used to iteratively optimize the constructed latent Dirichlet allocation model to obtain the topic model of the ancient document climate record data.

4. The method for processing ancient document climate record data according to claim 1, characterized in that: After step 4, the following steps are also included: Perform word segmentation on the original document to obtain multiple words; According to the climate event type of the original document, the corresponding pre-created climate event level screening terms are used to match the multiple words obtained by word segmentation processing. If there is any word corresponding to a certain climate event level screening term, the corresponding climate event level label is assigned to the original document; if not, the document is assigned a normal year label; if there are multiple words corresponding to multiple climate event level screening terms, the label assigned to the original document is the highest level climate event level.

5. The method for processing ancient document climate record data according to claim 1, characterized in that: After step 4, the following steps are also included: Perform word segmentation and entity recognition on the original document to identify the ancient place names contained in the original document; According to the correspondence between ancient place names and current place names, the ancient place names are replaced with the longitude and latitude of the current place names; By using GIS software, the original document is loaded onto the map as point data according to the longitude and latitude of the current place name, and attribute information is added to the point data, wherein the attribute information includes: climate event type.

6. A system for processing ancient document climate record data, characterized in that: include: A climate event processing module is used to process the original document in the form of classical Chinese using a method for processing ancient document climate record data to determine the climate event type of the original document; The climate event grading module is used to perform word segmentation processing on the original document to obtain multiple words. According to the climate event type of the original document, the corresponding pre-created climate event grade screening terms are used to match the multiple words obtained by the word segmentation processing. If any word corresponds to a certain climate event grade screening term, the corresponding climate event grade label is assigned to the original document; if not, the normal year label is assigned to the document; if multiple words correspond to multiple climate event grade screening terms, the label assigned to the original document is the highest climate event grade; The method for processing ancient document climate record data is the method for processing ancient document climate record data as described in any one of claims 1 to 3.

7. A system for processing ancient document climate record data, characterized in that: include: A climate event processing module is used to process the original document in the form of ancient document classical Chinese using a method for processing ancient document climate record data to determine the climate event type of the original document; The visualization module is used to perform word segmentation and entity recognition on the original document, identify the ancient place names contained in the original document, and replace the ancient place names with the longitude and latitude of the current place names according to the correspondence between the ancient place names and the current place names; By using GIS software, the original document is loaded onto the map as point data according to the latitude and longitude of the current place name, and attribute information is added to the point data, the attribute information including: climate event type; The method for processing ancient document climate record data is the method for processing ancient document climate record data as described in any one of claims 1 to 3.

Images