An intelligent method for generating landslide knowledge graph based on fine-tuning ChatGLM2

Abstract

The present invention discloses a landslide knowledge graph intelligent generation method based on fine-tuning ChatGLM2, which belongs to the field of knowledge graph technology, including: processing the collected first data information into a text format; preliminarily cleaning the first data information, dividing it into a plurality of sentence information, and deeply cleaning the sentence information based on a regular expression to obtain a second data information; inputting the second data information into ChatGLM2 130B, and performing a rough triple division and manual proofreading on the second data information based on a constructed question-answering template to obtain a triple data set; training the parameters of ChatGLM2 6B based on a model fine-tuning technology and loading it into a large language model, inputting the triple data set to generate a triple set; splitting the triple set to obtain an entity relationship table and performing deduplication, obtaining a landslide knowledge graph, and importing it into a Neo4j database. The present invention improves the efficiency of establishing a landslide knowledge graph.

Description

Technical Field

[0001] The present invention belongs to the technical field of knowledge graphs, and specifically relates to an intelligent generation method of landslide knowledge graphs based on fine-tuning ChatGLM2. Background Art

[0002] In the past few decades, with global climate change and environmental degradation, the frequency and complexity of natural disasters have continued to increase, bringing severe challenges to human society. In this context, disaster management and emergency response have become hot issues of concern worldwide. Traditional landslide monitoring methods have relatively single monitoring objects and data sources, and lack consideration of the interactions and impacts with other natural disaster types (such as earthquakes, floods, mudslides, etc.). Knowledge graphs, as a new data organization and knowledge management technology, can transform complex interaction processes into structured graphical representations that are easy to understand and operate. Therefore, it is particularly necessary to provide strong support for disaster prevention and response by constructing landslide knowledge graphs.

[0003] Commonly used knowledge graph construction methods include rule-based, machine learning-based, open information extraction-based and remote supervision-based methods. Similar prior art includes a Chinese patent application with publication number CN118332028A, which proposes a data mining and knowledge graph construction method, system, electronic device and medium. The method includes: using data mining technology to collect relevant data in the field of power engineering internal audit, preprocessing the collected relevant data in the field of power engineering internal audit, identifying and extracting entities and entity relationship information in the preprocessed relevant data in the field of power engineering internal audit, using a natural language model to extract feature words in the preprocessed relevant data in the field of power engineering internal audit, and classifying the feature words through a feature machine learning model to generate a knowledge graph in the field of power engineering internal audit, and controlling the quality of the generated knowledge graph in the field of power engineering internal audit, and performing gap filling and optimization.

[0004] However, the above-mentioned existing technologies classify feature words based on machine learning, and the method of generating knowledge graphs requires a large amount of labeled data for training, which is expensive and has significant defects in dealing with language complexity and diversity, as well as dependence on data annotation and knowledge base.

[0005] A similar prior art includes a Chinese patent application with publication number CN118520945A, which discloses a method for automatically constructing a knowledge graph based on prior knowledge and knowledge connections, the method comprising: saving relevant subject information for constructing the knowledge graph as a character string to obtain prompt data; retrieving article paragraphs from an external data source based on the prompt data and saving them; injecting prompt templates of four large language model agents, namely, tagging, reasoning, cognition, and association, respectively; inputting the injected prompt templates, article paragraphs, and specific task requirements into the agent to obtain prior knowledge; inputting the article paragraphs, prior knowledge, and predefined comparison prompt texts into the knowledge connection large language model to obtain valid data related to the knowledge graph topic; using the valid data related to the knowledge graph topic and the predefined input prompts as the input layer of the large language model automatic proxy framework to obtain entity relationship triples and complete the knowledge graph construction.

[0006] The above-mentioned existing technologies are based on the automatic construction method of knowledge graphs based on prior knowledge and knowledge connections. If the knowledge connection rules are not designed flexibly enough, they may not be able to adapt to new data formats or knowledge types, resulting in limited updating and expansion of knowledge graphs. Therefore, in order to build a knowledge graph with rich content and strong support for disaster prevention and control, it is necessary to use appropriate methods to accurately and comprehensively capture accurate entities and relationships from unstructured data. Summary of the invention

[0007] To solve the above problems, the present invention provides a landslide knowledge graph intelligent generation method based on fine-tuning ChatGLM2 to solve the problems in the prior art.

[0008] In order to achieve the above-mentioned purpose of the invention, the present invention proposes a landslide knowledge graph intelligent generation method based on fine-tuning ChatGLM2, comprising:

[0009] S1: collecting first data information, and batch processing the first data information into a text format based on a Python program, wherein the first data information includes scientific and technological literature, disaster events, and social media information related to landslides;

[0010] S2: Preliminarily clean the first data information converted into the text format, the Python program divides the first data information after preliminary cleaning into multiple statement information using a period as a separator, and deeply cleans the statement information based on a regular expression to obtain second data information;

[0011] S3: construct a question-answering template, input the second data information into the first version of the large language model, the first version of the large language model roughly divides the second data information into triples based on the question-answering template, and manually proofreads the roughly divided triples to obtain a triple data set;

[0012] S4: Train the parameters of the second version of the large language model based on the P-tuning v2 model fine-tuning technology;

[0013] S5: loading the trained parameters into the second version of the large language model, and inputting the triplet data set to generate a triplet set;

[0014] S6: split the triple set based on the Python program to obtain an entity relationship table, and remove duplicates from the entity relationship table by combining the minimum edit distance and Jaccard method;

[0015] Step S7: import the entity relationship tables after deduplication into the first database in batches, construct a landslide knowledge graph and display it visually.

[0016] Furthermore, batch processing the first data information into a text format includes the following steps:

[0017] The first data information in PDF format is read into text format using the fitz library, and the first data information in docx format is read into the text format using the python-docx package. If the first data information is the social media information, it is obtained using crawler technology, and the full-width characters in the text information obtained by the crawler are converted into half-width characters. All punctuation marks except common English and Chinese punctuation marks in the text information are removed based on regular expressions to obtain the final text format.

[0018] Furthermore, preliminarily cleaning the first data information includes the following steps:

[0019] If the first data information is journal data in the scientific literature, identify the literature citations in the journal data and write the corresponding regular expression to remove the literature citations. If the first data information is the social media information, identify the first data features in the social media information, the first data features include URLs, user names, topic tags and special characters, and write the corresponding regular expression to clean the first data features in the social media information.

[0020] Furthermore, before the deep cleaning of the statement information, the following steps are also included:

[0021] A stop word library is constructed, and stop words in the sentence information are filtered out based on the stop word library.

[0022] Furthermore, the first version is ChatGLM2 130B version.

[0023] Furthermore, before training the parameters of the large language model, the following steps are also included:

[0024] The triple data set is organized into a data format supported by the second version of the large language model.

[0025] Furthermore, the second version is ChatGLM2 6B version.

[0026] Furthermore, deduplication of the entity relationship table includes the following steps:

[0027] The first similarity between character string A and character string B is calculated based on a first formula, where the first formula is: Among them, E(A, B) is the minimum edit distance between the string A and the string B, J(A, B) is the Jaccard similarity between the string A and the string B, α and β are the first weight and the second weight, len(·) is the string length calculation formula, max(·) is the maximum value calculation formula, the entities and relationships in the entity relationship table are converted into string form, if the first similarity of the two strings is greater than the first threshold, it means that the two strings are repeated, and one of the strings is selected for removal.

[0028] Furthermore, the first threshold is 0.8.

[0029] Furthermore, the first database is a Neo4j database.

[0030] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0031] Starting from the unstructured characteristics of landslide disaster data, the present invention adopts multi-source data integration and uses Python automation programs to extract, clean and organize data; a small number of samples are manually labeled with the assistance of the ChatGLM2 130B model, and the ChatGLM2 model is fine-tuned using the P-tuning technology, so that the full input corpus generates high-quality knowledge graph triples; in addition, the present invention also includes using Python programs to construct accurate entity and relationship tables, and using Neo4j database technology to construct and store knowledge graphs for entity and relationship tables.

[0032] By integrating multi-source unstructured data, the present invention not only enhances the richness and coverage of data, but also utilizes the powerful prior knowledge of large language models to more effectively process and understand entities and semantic relationships related to landslides. This method solves the problems of small scale and imprecise relationship definition of existing landslide knowledge graphs, with simple steps and low hardware consumption. The present invention can significantly improve the efficiency and quality of establishing landslide disaster knowledge graphs, and provide powerful knowledge support and decision-making basis for the prevention, response and management of landslide disasters. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of the steps of a method for intelligently generating a landslide knowledge graph based on fine-tuning ChatGLM2 in the present invention;

[0034] Figure 2 Generate a flow chart for the knowledge graph of the present invention;

[0035] Figure 3 It is a partial entity graph of the knowledge graph of the present invention;

[0036] Figure 4 is a flow chart of the monitoring model of the present invention;

[0037] Figure 5 is a deformation diagram of the present invention;

[0038] Figure 6 It is a plan view of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0040] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0041] like Figure 1 As shown, a landslide knowledge graph intelligent generation method based on fine-tuning ChatGLM2 includes:

[0042] S1: Collect first data information, and batch process the first data information into a text format based on a Python program. The first data information includes scientific and technological literature, disaster events, and social media information related to landslides.

[0043] Specifically, Figure 2As shown in the figure, it is a flow chart for generating a knowledge graph, which screens and collects a large amount of first data information related to landslide phenomena from the Internet. The first data information includes but is not limited to scientific and technological literature, disaster events and social media information, integrates these multi-source data to build a comprehensive and rich corpus, and uses Python program to unify the first data information from different sources and in different formats into text format to facilitate subsequent integration and analysis. At the same time, the automatic batch processing of Python program improves the speed of data processing, so that information can be analyzed and utilized more quickly.

[0044] S2: Perform preliminary cleaning on the first data information converted into a text format. The Python program divides the first data information after preliminary cleaning into multiple statement information using a period as a separator, and performs in-depth cleaning on the statement information based on a regular expression to obtain the second data information.

[0045] Specifically, for the first data information from different sources, different regular matching expressions are needed to normalize the data format to achieve preliminary cleaning, and Python string operations are used to split the text information of the first data information into separate sentences by periods. Regular expressions are used to match and replace each sentence to remove unnecessary characters, punctuation marks, spaces, HTML tags, etc., to achieve deep cleaning.

[0046] S3: Construct a question-answering template, input the second data information into the large language model of the first version, the large language model of the first version roughly divides the sentence information in the second data information into triples based on the question-answering template, and manually proofreads the roughly divided triples to obtain a triple data set.

[0047] Specifically, the first version of the large language model is ChatGLM2 130B. When building question-answering templates, the powerful capabilities of version 130B can be used to design more complex and detailed templates. These templates can more accurately guide the model to extract key information from the text. For example, templates can be designed to specifically extract information such as the location, time, cause and impact of landslides, and then manually annotate a small number of samples to generate high-quality triples.

[0048] Assume that the starting instructions for the large language model to start generating triples are as follows: You are a network intelligence that helps humans track all knowledge triples of related people, things, and concepts, and associate these knowledge triples with the knowledge in the existing knowledge base and knowledge graph, and extract all knowledge triples from the text, including clauses of subject, predicate, and object. The subject is an entity, which can be a person, place, organization, concept, etc. The predicate is the attribute of the subject described, and the object is the value of the attribute.

[0049] After entering the above startup command, enter the text information of a certain paragraph in the first data information. Assume that the text information is:

[0050] Example 1: There is confusion between landslide "susceptibility" and landslide "risk". The two terms are often used synonymously, even though they express different concepts. Here, we consider landslide susceptibility as the probability of a landslide occurring in a certain area, based on the local topography and environmental conditions. Susceptibility measures the extent to which a certain terrain may be affected by a landslide in the future. In other words, it estimates the "places" where a landslide may occur. In mathematical terms, susceptibility can be defined as the spatial probability of slope failure given a set of geological and environmental conditions. This is called "landslide analysis".

[0051] The above text information is input into the ChatGLM2 130B large language model for coarse triple division, and the output is: (landslide susceptibility, definition, predicting the possibility of landslides in a certain area)<|>(landslide susceptibility, measurement standard, the extent to which the terrain may be affected by landslides in the future)<|>(landslide susceptibility, use, estimated locations where landslides may occur)<|>(landslide susceptibility, definition conditions, the spatial probability of slope instability given a set of geological environmental conditions)<|>(landslide analysis, definition, mathematical definition of landslide susceptibility), where “<|>” is used to separate different triplet information.

[0052] Example 2: Geological disasters are natural disasters caused mainly by geological dynamic activities or abnormal changes in the geological environment. Under the influence of the earth's internal dynamics, external dynamics or man-made geological dynamics, the earth releases abnormal energy, material movement, deformation and displacement of rock and soil bodies, and abnormal changes in the environment, which endanger human life and property, life and economic activities, or destroy the resources and environment on which human beings depend for survival and development.

[0053] Output: (geological disasters, main causes, geological dynamic activities or abnormal changes in the geological environment) <|> (geological disasters, causes, internal and external forces of the earth or human geological dynamic effects) <|> ((geological disasters, manifestations, abnormal energy release, material movement, deformation and displacement of rock and soil bodies, and abnormal changes in the environment, etc.) <|> (geological disasters, hazards, phenomena or processes that endanger human life and property, life and economic activities, or destroy the resources and environment on which human beings depend for survival and development).

[0054] 5086 sample data, i.e., sentence information, are collected. Each sample is a text string. The code is written using Python program to call the API for deploying the large language model and send each sample text to the API. The large language model generates roughly divided triples based on the question and answer template. Manual reviewers check the automatically generated triples to confirm whether the entities and relationships are correct. For incorrect triples, the reviewers correct or re-extract them. For similar or repeated triples, the reviewers merge or delete them. The manually proofread triples are constructed into the final finely divided set, i.e., the triple data set.

[0055] S4: Fine-tune and train the parameters of the second version of the large language model based on the P-tuning v2 model fine-tuning technology.

[0056] Specifically, the second version of the large language model is ChatGLM2 6B. Compared with the first version of the large language model ChatGLM2 130B, ChatGLM2 6B has improvements in performance, context length, reasoning efficiency and openness, making it more effective and practical in tasks such as dialogue generation and knowledge graph construction. The ChatGLM2 model is fine-tuned using P-tuning technology to generate high-quality knowledge graph triples from the full input corpus.

[0057] For the convenience of description, the second version of the large language model is defined as M here, and the input sequence of model M is set to x, P is prompt tokens, that is, the prompt information that guides the model to understand the task and generate correct output. The output y of model M can be calculated in the following way: y = M([P; x]), where [P; x] represents the concatenation of prompt tokens and the input sequence x. The prompt tokens and the input sequence are concatenated to form a new sequence, which is the direct input of the model. The concatenated sequence is input into the model, and the model performs forward propagation calculations through its network structure. Prompt tokens can be some special tags or pre-designed text templates, which are added to the front of the input sequence to guide the generation process of the large language model.

[0058] Step S5: Load the trained parameters into the second version of the large language model, and input the triplet data set to generate a triplet set.

[0059] Specifically, the parameters obtained during the training process (such as fine-tuned weights) are loaded into the model. The parameters are obtained by training on a specific task and can make the model better adapt to the task. All corpora, i.e., the triple data set after format conversion, are input to generate a triple set.

[0060] Step S6: Split the triple set based on the Python program to obtain the entity relationship table, and combine the minimum edit distance and Jaccard method to deduplicate the entity relationship table.

[0061] Specifically, two data structures are created using Python programs, one for storing entities and the other for storing relationships. The entity table may contain the unique identifier and related attributes of the entity, and the relationship table may contain entity pairs and the relationship types between them. The triple set is traversed, and for each triple, the subject and object are extracted as entities, and the predicate is extracted as a relationship. Finally, the entity relationship table is obtained. When extracting entities and relationships, ensure that they are unique. The minimum edit distance fusion Jaccard method can be used to deduplicate entities and relationships. The specific implementation steps will be explained later.

[0062] Step S7: Import the entity relationship tables after deduplication into the first database in batches, construct a landslide knowledge graph and perform visual display.

[0063] Specifically, the first database is a Neo4j database. The present invention uses a Python program to construct accurate entity and relationship tables, and uses Neo4j database technology to construct and store knowledge graphs for the entity and relationship tables, and visualizes the constructed knowledge graphs.

[0064] like Figure 3 As shown in the figure, it is a partial entity graph of the knowledge graph, which contains three types of entities, namely landslide entities, geographic entities and remote sensing entities. Among them, the brown part is the knowledge graph of some landslide entities, which is extracted from the unstructured corpus of social media. The landslide event entity connects the disaster location entity, the disaster time entity, the disaster extent entity and the loss entity. In addition to the entities in the figure, it can also connect the sliding direction entity, the accumulation and the landslide body entity. The green part of the loss entity connects the death toll entity, the affected number entity, the house damage entity, the property loss entity and other loss entities.

[0065] The blue part represents the geographic entity part, which includes natural feature entities, human activity entities or economic activity entities. Natural feature entities include: rainfall, steep slopes and earthquakes. Human activity entities include vegetation, drainage, water storage, blasting and excavation entities. Blasting and excavation may lead to economic activities. Since the landslide was caused by an earthquake, Figure 3 The geographic entity part of is earthquake, including measuring station and magnitude.

[0066] The pink part represents the remote sensing entity, which is connected to the remote sensing processing entity and the remote sensing index entity. Both the pre-disaster image entity and the post-disaster image entity are included in the remote sensing entity and the remote sensing processing entity. The remote sensing processing entity is further connected to the slope calculation entity, the vegetation index calculation entity, the filtering processing entity, and the radiation correction entity, which are associated and operated through the remote sensing processing entity. The remote sensing index entity connects the difference vegetation index entity, the terrain moisture index entity, the landslide hazard index entity, and the remote sensing hydrological model entity, which are specific index models and calculation methods associated with the remote sensing index entity. Specifically, the slope calculation entity contains three attributes: formula, type, and function. It is worth noting that the slope calculation entity and the vegetation index calculation entity belong to the same type and share similar attributes and methods, although they have significant differences in functionality. The slope calculation entity is mainly used to calculate the terrain slope and determine the terrain gradient, so as to better understand and analyze terrain changes and characteristics. In contrast, the function of the vegetation index calculation entity is to calculate various vegetation indices, which are used to assess the density and health of vegetation cover. These vegetation indices help monitor and manage ecosystems and assess the impact of environmental changes on vegetation. At the same time, the remote sensing index entity provides rich data and analysis tools for geographic information systems and environmental monitoring by connecting different index models.

[0067] Based on the above landslide knowledge graph, a monitoring model is established, and the landslide knowledge graph is updated through real-time monitoring data through the monitoring model. Figure 4 As shown in Figure 1, it is a flow chart of the monitoring model. The operation of the monitoring model includes three processes, namely, the landslide monitoring and analysis process, the process of embedding structured data into ontology to form triples, and the process of extracting triples from unstructured data. The landslide monitoring and analysis process includes the following: First, the monitoring model performs abnormal data detection to ensure the quality of the input data. If an abnormality is found, semantic query technology is used and external data sources such as geological databases and meteorological databases are retrieved to obtain more comprehensive information and conclusions. The conclusion on whether it is abnormal is uploaded to the Neo4j database to complete the analysis process.

[0068] The process of embedding structured data into the ontology to form triples includes the following: To ensure the integrity and consistency of the data, it may be necessary to MD5 sign the data. For the files in the mapping process, decide whether to discard or embed them into the ontology. Integrate the data into the Neo4j database using the py2neo library for storing and managing knowledge graph data.

[0069] The process of extracting triples from unstructured data includes the following: integrating the extracted data into the question-and-answer template, using ChatGLM2130B to build the question-and-answer template that forms the knowledge graph, and assisting in manual annotation of a small number of samples, fine-tuning and training the parameters of the ChatGLM2-6B model based on the P-tuning v2 model fine-tuning technology, and using the trained model to generate triples. These triples contain information such as entities, relationships, and attributes, and the generated triplets are stored in the Neo4j database.

[0070] like Figure 3 As shown in the figure, in the hypothetical case, after the landslide occurred on December 18, 2023, the monitoring model detected abnormal data, downloaded Sentinel-1 images that overlapped with the disaster site (35.70°N, 102.79°E) and were collected before the disaster, merged and imported the images, and started a separate thread to wait for the next image that overlapped with the disaster site and was collected after the disaster. It is worth noting that since most event entities are generated based on notifications, there is a certain delay. Therefore, when the model starts to obtain pre-disaster images, post-disaster images are collected. In this case, the model will automatically download two images before and after the disaster. Then the model queries the available remote sensing processing methods. Through the mesh structure, it can quickly query that the D-InSAR method may be needed under the landslide node. At this time, the model tries to call the D-InSAR method in the remote sensing processing method. The metadata of the data required by the method can be queried according to the nodes connected to the D-InSAR node, and these metadata can be used to query the data in the database, and these data will be transferred to the working directory. After waiting for the data transmission to be completed, the D-InSAR workflow will be carried out according to the preset parameters. The final results will be saved in the geographic database, and a new node will be added to the Neo4J database to save the location of the results.

[0071] The saved results were extracted and further plotted as follows Figure 5 The deformation diagram shown is from Figure 5 It can be seen that there is a large uplift between 102°44′ and 102°54′ north latitude and 35°42′ and 35°50′ east longitude, with the maximum uplift reaching 0.097 meters. The mountains in the southwest direction have a large unilateral settlement. In addition, there is a large local uplift in the northwest direction of the central uplift area. Generally speaking, most of the deformation areas are located between the mountains.

[0072] The terrain deformation map and the vector data of the core deformation area are input into the monitoring model, and the horizontal interval is set to 3 meters and the vertical interval is 4 meters. The monitoring model has integrated advanced profile analysis methods, which can automatically push relevant data when researchers query specific terrain events. Researchers only need to upload the vector file of the study area and the target image to the model to easily generate a profile map. Figure 6 As shown in the figure, the profile automatically generated by the monitoring model, the deformation of profile f is the most significant in the horizontal direction, the slope of the fitting line is as high as 1.186, and the average deformation reaches 7.5 cm; in the vertical direction, the deformation of profile a is the largest, and the slope of the fitting line is 1.032, indicating that the horizontal deformation is generally greater than the vertical deformation, which may indicate that the crustal stress is unevenly distributed in the horizontal direction, causing a more significant deformation response. Specifically, profiles e to f show an obvious trend of high in the west and low in the east, which may be related to underground tectonic activities or historical landslide activities, while the deformation of profile h is relatively uniform, suggesting that the geological conditions in this area are relatively stable, perhaps due to relatively uniform stratigraphic structure and lithological conditions. In addition, the model also automatically performs curve fitting and derivation, accurately obtains the deformation rate of each point, and selects the derivative value with the largest absolute value in each profile to store in the landslide knowledge map. The address index of the profile and its corresponding derivative value are also effectively associated and stored, providing rich secondary data support for subsequent research and reasoning.

[0073] The above results prove that the monitoring model based on the landslide knowledge graph can effectively realize the comprehensive organization and integration of landslide monitoring data, geological data and meteorological information. By utilizing the NLP technology of ChatGLM2, the "unstructured knowledge" of expert knowledge, media knowledge and simulation models is converted into "structured knowledge", thereby improving data processing efficiency and the processing capacity of large-scale data sets; the intelligent construction method based on ChatGLM2 makes full use of the natural language understanding ability of the large language model, fully and accurately extracts a variety of unstructured data, and realizes the extraction of multiple groups of associated triples; the knowledge graph ontology model not only promotes the organic link between data by defining the relationship and attributes between different data types, but also enhances the semantic understanding ability of data, improves data retrieval efficiency and analysis quality; through automated and intelligent data processing, the efficiency and accuracy of disaster monitoring are greatly improved, providing strong technical support for disaster prevention and mitigation.

[0074] Starting from the unstructured characteristics of landslide disaster data, the present invention adopts multi-source data integration and uses Python automation programs to extract, clean and organize data; a small number of samples are manually labeled with the assistance of the ChatGLM2 130B model, and the ChatGLM2 model is fine-tuned using the P-tuning technology, so that the full input corpus generates high-quality knowledge graph triples; in addition, the present invention also includes using Python programs to construct accurate entity and relationship tables, and using Neo4j database technology to construct and store knowledge graphs for entity and relationship tables.

[0075] By integrating multi-source unstructured data, the present invention not only enhances the richness and coverage of data, but also utilizes the powerful prior knowledge of large language models to more effectively process and understand entities and semantic relationships related to landslides. This method solves the problems of small scale and imprecise relationship definition of existing landslide knowledge graphs, with simple steps and low hardware consumption. The present invention can significantly improve the efficiency and quality of establishing landslide disaster knowledge graphs, and provide powerful knowledge support and decision-making basis for the prevention, response and management of landslide disasters.

[0076] As a preferred technical solution of the present invention, batch processing of the first data information into a text format includes the following steps:

[0077] The first data information in PDF format is read into text format using the fitz library, and the first data information in docx format is read into text format using the python-docx package. If the first data information is social media information, it is obtained using crawler technology, and the full-width characters in the text information obtained by the crawler are converted into half-width characters. All punctuation marks except common English and Chinese punctuation marks in the text information are removed based on regular expressions to obtain the final text format.

[0078] Specifically, the first data information in PDF format is read into text format using the fitz library. In the PDF document, line breaks and hyphens (such as -) may appear, and the PDF document generates other irrelevant special characters due to format errors or other encoding problems. The special characters marked in the present invention include: \ue5d2, \ue5cf, \ue5d2, \ue5cf, \ue5ce. These characters are usually part of the document format rather than the actual text content. The present invention uses the string replacement function replace to replace the line breaks, hyphens and these characteristic characters with empty characters.

[0079] The first data information in docx format is read using the python-docx package. First, use the Document class to create a document object in the docx module, traverse all paragraphs in the document object, use the paragraph.text attribute to get the text content of each paragraph, create an empty list, add the text content of each paragraph to the empty list, use the string's join() method to concatenate all the text contents in the list into a single string, and return the string.

[0080] For official reports and the first data information of major social media, crawler technology is used to obtain the designated website. The Python request library is used to send HTTP requests to obtain the web page content. The cookies and headers required for access are set in the request to simulate browser access. The data obtained from the website is usually in HTML format, and useful text information needs to be extracted from it. In order to facilitate data processing, the full-width characters in the text information need to be converted into half-width characters. Each character of the input text is traversed. Full-width and half-width characters have different code points in Unicode encoding. First, the characters are converted to their corresponding Unicode code points. Then, if it is a full-width space (code point 0x3000), it is converted to a half-width space (code point 0x0020); for other full-width characters (characters with code points between 0xFF01 and 0xFF5E), they are converted to corresponding half-width characters by subtracting 0xFEE0. Then, all punctuation marks except common English and Chinese punctuation marks in the text information are removed through regular expressions. The regular expressions used are as follows:

[0081]

[0082] This function relies on the re package in Python. The text filtered by the regular expression is then written to a txt file for later use. In text analysis and natural language processing, full-width characters may be regarded as noise. Converting them to half-width characters helps improve data quality.

[0083] Preliminary cleaning of the first data information includes the following steps:

[0084] If the first data information is journal data in scientific literature, identify the literature citations in the journal data and write a corresponding regular expression to remove the literature citations. If the first data information is social media information, identify the first data features in the social media information. The first data features include URLs, user names, topic tags and special characters, and write a corresponding regular expression to clean the first data features in the social media information.

[0085] Specifically, for journal data, the following regular expressions are adopted to remove literature citations:

[0086] (References|Literature Citation|Citation).*

[0087] The re.sub function is used to replace the matched parts with empty strings. For social media postures, corresponding regular expressions need to be written for different websites to identify the first data features in social media profiles and write corresponding regular expressions to clean the first data features. For example, to remove URLs, the regular expression: http\S+|www\S+ is used, and to remove usernames: @\w+. By using regular expressions, text data can be effectively cleaned and preprocessed to remove unnecessary information and retain valuable data.

[0088] Before deeply cleaning the statement information, the following steps are also included:

[0089] Construct a stop word library and filter out the stop words in the statement information based on the stop word library.

[0090] Specifically, stop words usually refer to those words that frequently appear in the text but contribute little to the meaning of the text, such as "de", "shi", "zai", etc. Define a stop word library and create a corresponding function to filter out these stop words. Filtering these texts helps with subsequent text analysis and processing.

[0091] Before training the parameters of the large language model, the following steps are also included:

[0092] Organize the triple dataset into the data format supported by the second version of the large language model.

[0093] Specifically, the second version of the large language model is the ChatGLM2 6B version. Organize the refined partition set obtained into the data format supported by the ChatGLM2 6B version: {"content": "", "summary": ""}.

[0094] The steps for deduplicating the entity relationship table include:

[0095] Calculate the first similarity between string A and string B based on the first formula. The first formula is: Wherein, E(A, B) is the minimum edit distance between string A and string B, I(A, B) is the Jaccard similarity between string A and string B, α and β are the first weight and the second weight, len(·) is the formula for calculating the length of the string, and max(·) is the formula for calculating the maximum value. The entities and relationships in the entity relationship table are converted into string form. If the first similarity of the two strings is greater than the first threshold, it means that the two strings are repeated, and one string is selected to be removed.

[0096] Specifically, the first threshold is 0.8, and the minimum edit distance E(A, B) is calculated as follows: define D(i, j) as the minimum edit distance between the first i characters of string A and the first j characters of string B, set D(0, j) = j, D(i, 0) = i, and recursively calculate for all i, j>0: D(i, j) = min(D(i-1, j) + 1, D(i, j-1) + 1, D(i-1, j-1) + cost), where cost = 0 if A[i] = B[j], otherwise cost = 1. After the above calculation, the minimum edit distance is E(A, B) = D(len(A), len(B)), and the Jaccard similarity calculation method is as follows: The minimum edit distance focuses on the minimum number of editing operations required to transform one string into another, while the Jaccard similarity focuses on the ratio of the intersection to the union of two sets. This method can evaluate the similarity of texts from different perspectives and provide a more comprehensive similarity evaluation.

[0097] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0098] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0099] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0101] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A landslide knowledge graph intelligent generation method based on fine-tuning ChatGLM2, characterized in that: The method comprises the following steps: S1: collecting first data information, and batch processing the first data information into a text format based on a Python program, wherein the first data information includes scientific and technological literature, disaster events, and social media information related to landslides; S2: Preliminarily clean the first data information converted into the text format, the Python program divides the first data information after preliminary cleaning into multiple statement information using a period as a separator, and deeply cleans the statement information based on a regular expression to obtain second data information; S3: construct a question-answering template, input the second data information into the first version of the large language model, the first version of the large language model performs a rough division of the second data information into triples based on the question-answering template, and manually proofreads the roughly divided triples to obtain a triple data set, wherein the first version is ChatGLM2 130B version; S4: training the parameters of the second version of the large language model based on the P-tuning v2 model fine-tuning technology, wherein the second version is ChatGLM2 6B version; S5: loading the trained parameters into the second version of the large language model, and inputting the triplet data set to generate a triplet set; S6: split the triple set based on the Python program to obtain an entity relationship table, and remove duplicates from the entity relationship table by combining the minimum edit distance and Jaccard method; Deduplication of the entity relationship table includes the following steps: The first similarity between character string A and character string B is calculated based on a first formula, where the first formula is: ,in, is the minimum edit distance between the string A and the string B, is the Jaccard similarity between the string A and the string B, and are the first weight and the second weight, is the length calculation formula of the string A, is the length calculation formula of the string B, is a maximum value calculation formula, converting entities and relationships in the entity relationship table into string form, if the first similarity of two strings is greater than a first threshold, it means that the two strings are repeated, and any one of the two strings is removed; Step S7: import the entity relationship tables after deduplication into the first database in batches, construct a landslide knowledge graph and display it visually.

2. The method according to claim 1, characterized in that The batch processing of the first data information into a text format comprises the following steps: The first data information in PDF format is read into text format using the fitz library, and the first data information in docx format is read into the text format using the python-docx package. If the first data information is the social media information, it is obtained using crawler technology, and the full-width characters in the text information obtained by the crawler are converted into half-width characters. All punctuation marks except common English and Chinese punctuation marks in the text information are removed based on regular expressions to obtain the final text format.

3. The method according to claim 1, characterized in that Preliminary cleaning of the first data information includes the following steps: If the first data information is journal data in the scientific literature, identify the literature citations in the journal data and write the corresponding regular expression to remove the literature citations. If the first data information is the social media information, identify the first data features in the social media information, the first data features include URLs, user names, topic tags and special characters, and write the corresponding regular expression to clean the first data features in the social media information.

4. The method according to claim 1, characterized in that Before deep cleaning the statement information, the following steps are also included: A stop word library is constructed, and stop words in the sentence information are filtered out based on the stop word library.

5. The method according to claim 1, characterized in that Before training the parameters of the large language model, the following steps are also included: The triple data set is organized into a data format supported by the second version of the large language model.

6. The method according to claim 1, characterized in that The first threshold is 0.

8.

7. The method according to claim 1, characterized in that The first database is a Neo4j database.

Images