Cooperative prevention and control knowledge integration management method and system for producing area cadmium and arsenic pollution

Abstract

The invention discloses a production area cadmium and arsenic pollution collaborative prevention and control knowledge integration management method and system, and relates to the technical field of data integration management, in the method, a multi-thread parallel acquisition technology is used for acquiring massive network literatures related to cadmium and arsenic pollution, and core text data in the network literatures are extracted; the method comprises the following steps of: preprocessing the network literature, identifying and screening named entities to obtain a key sentence set, determining the category of the corresponding network literature based on the key sentence set by adopting a random forest algorithm, and clustering all the network literatures to obtain a network literature clustering result; and finally, constructing a cadmium and arsenic pollution knowledge graph and visually displaying the cadmium and arsenic pollution knowledge graph. According to the method, cadmium and arsenic pollution related data can be efficiently and automatically collected and subjected to standardization processing, the consistency and comparability of the data are ensured, meanwhile, a knowledge graph mode is adopted, multi-angle and multi-dimensional data analysis and display functions can be provided, and the requirements of users in different fields and different levels are met.

Description

Technical Field

[0001] The present application relates to the technical field of data integration management, and in particular to a knowledge integration management method and system for collaborative prevention and control of cadmium and arsenic pollution at production sites. Background Art

[0002] With the acceleration of industrialization and the continuous development of agricultural production, the pollution problem of heavy metal elements such as cadmium (Cd) and arsenic (As) in the environment has become increasingly serious. Cadmium and arsenic are heavy metals that are significantly harmful to human health. They are widely distributed through media such as the atmosphere, water bodies and soil, and can be enriched in organisms. They enter the human body through the food chain and cause chronic poisoning and various diseases. In particular, cadmium and arsenic pollution in farmland soils such as rice fields has become an important factor affecting the safety of agricultural products and needs to be effectively controlled.

[0003] In this context, the business of integrated management of knowledge and literature on cadmium and arsenic pollution has emerged. It aims to provide comprehensive, accurate and timely knowledge and literature services to scientific research institutions, agricultural enterprises and the public by integrating academic research, paper literature, network knowledge, case analysis and other diversified information related to cadmium and arsenic pollution. Through the knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution in the production area, users can easily obtain the latest research results on cadmium and arsenic pollution, the latest progress in governance technology, real-time updates of monitoring data and other information, so as to better understand the current situation, causes and governance strategies of cadmium and arsenic pollution, and provide strong support for formulating scientific governance plans and promoting the process of pollution control.

[0004] However, the current cadmium and arsenic pollution knowledge and literature integration management business faces the following challenges: First, in terms of data collection, due to the diversity of data sources and formats, the efficiency and coverage of data collection are seriously challenged; second, in terms of information collection and mining, facing a large and diverse data set, it is impractical to manually extract the semantics of articles and classify them. With the iterative progress of digital technology and artificial intelligence technology, it has become possible to conduct intelligent research on cadmium and arsenic pollution using methods such as big data mining, text analysis, and machine learning. Therefore, how to use big data to design an efficient knowledge integration management system for the coordinated prevention and control of cadmium and arsenic pollution at the origin has become an urgent problem that needs to be solved. Summary of the invention

[0005] The purpose of this application is to provide a knowledge integration management method and system for collaborative prevention and control of cadmium and arsenic pollution at the production site, which can efficiently collect data and mine information based on massive cadmium and arsenic pollution knowledge literature, greatly enhancing the user's depth of understanding of cadmium and arsenic pollution.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In the first aspect, the present application provides a method for integrated management of knowledge on collaborative prevention and control of cadmium and arsenic pollution at the production site, comprising the following steps:

[0008] Using multi-threaded parallel collection technology, online literature related to cadmium and arsenic pollution is collected by searching keywords at preset sites; the preset sites are websites that are highly related to cadmium and arsenic pollution.

[0009] The core text data in online literature related to cadmium and arsenic pollution were extracted, and the core text data were classified using a preset classification system to mark the category information of the core text data.

[0010] After preprocessing the core text data, named entity recognition and screening are performed to obtain a key sentence set of the core text data.

[0011] A random forest algorithm is used to determine the category of online documents related to cadmium and arsenic pollution corresponding to the core text data based on a key sentence set of the core text data.

[0012] The k-means clustering method was used to cluster all network literature related to cadmium and arsenic pollution, and the network literature clustering results were obtained.

[0013] A cadmium and arsenic pollution knowledge graph is constructed based on the network literature clustering results, and the cadmium and arsenic pollution knowledge graph is visually displayed.

[0014] Optionally, multi-threaded parallel collection technology is used to collect online documents related to cadmium and arsenic pollution by searching keywords at a preset site, specifically including the following steps:

[0015] Websites highly related to cadmium and arsenic pollution are obtained as preset sites, and a priority is set for each preset site, and search keywords are set for subsequent searches.

[0016] The search results are sorted in descending order according to the priority of the preset site and the actual weight of the search keyword, and the first preset number of search results in the sort are obtained as network literature related to cadmium and arsenic pollution.

[0017] For each online document related to cadmium and arsenic pollution, content analysis and bibliometric methods are used to deeply understand the content and theme of the document and extract the corresponding core text data.

[0018] The core text data is classified using a preset classification system to mark the category information of the core text data.

[0019] Optionally, after preprocessing the core text data, named entity recognition and screening are performed to obtain a key sentence set of the core text data, which specifically includes the following steps:

[0020] The named entities are recognized after preprocessing the core text data; the preprocessing includes unified naming specifications and time and place format specifications.

[0021] Delete the high-frequency grammatical function words and low-frequency words in the identified named entities, and process the retained named entities as candidate words.

[0022] The candidate words are input into the BERT model to calculate the corresponding weights, and the candidate words with weights greater than the keyword weight threshold are selected as keywords; multiple keywords form a key sentence.

[0023] The weight of the key sentence is calculated according to the weight of each keyword in the key sentence, and the key sentences with weights less than the key sentence weight threshold are eliminated to obtain the key sentence set of the core text data.

[0024] Second invention, this application provides a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the production site, including:

[0025] The core text data collection module is used to use multi-threaded parallel collection technology to collect online documents related to cadmium and arsenic pollution by searching keywords at preset sites; it is also used to extract core text data from online documents related to cadmium and arsenic pollution, and classify the core text data using a preset classification system to mark the category information of the core text data; the preset sites are websites that are highly related to cadmium and arsenic pollution.

[0026] The core text data reorganization module is used to perform named entity recognition and screening on the core text data after preprocessing to obtain a set of key sentences of the core text data.

[0027] The core text data clustering and cleaning module is used to determine the category of cadmium and arsenic pollution-related online documents corresponding to the core text data based on the key sentence set of the core text data using the random forest algorithm; it is also used to cluster all cadmium and arsenic pollution-related online documents using the k-means clustering method to obtain the network document clustering results.

[0028] The knowledge graph construction and visualization module is used to construct a cadmium and arsenic pollution knowledge graph based on the results of online literature clustering, and to visualize the cadmium and arsenic pollution knowledge graph.

[0029] Optionally, the core text data collection module includes:

[0030] The cadmium-arsenic pollution highly correlated website marking unit is used to obtain websites that are highly correlated with cadmium-arsenic pollution as preset sites and set a priority for each preset site.

[0031] The search keyword setting unit is used to set the search keyword for the next search.

[0032] The search result sorting unit is used to sort the search results in descending order according to the priority of the preset site and the actual weight of the search keyword, and obtain the first preset number of search results in the sorting as network documents related to cadmium and arsenic pollution.

[0033] The core text data extraction unit is used to use content analysis and bibliometric methods to deeply understand the content and theme of each online document related to cadmium and arsenic pollution, and extract the corresponding core text data.

[0034] The core text data screening unit is used to classify the core text data using a preset classification system to mark the category information of the core text data.

[0035] Optionally, the core text data extracted by the core text data extraction unit will be saved as raw data in JSON format and transferred to the original database for effective storage and management; users can quickly access and obtain these raw data through simple search operations.

[0036] Optionally, the core text data reorganization module includes:

[0037] The named entity recognition unit is used to recognize named entities after preprocessing the core text data; the preprocessing includes unified naming specifications and time and place format specifications.

[0038] The named entity elimination unit is used to delete high-frequency grammatical function words and low-frequency words in the identified named entities, and process the retained named entities as candidate words.

[0039] The candidate vocabulary screening unit is used to input the candidate vocabulary into the BERT model to calculate the corresponding weight, and select the candidate vocabulary with a weight greater than the keyword weight threshold as the keyword; multiple keywords constitute a key sentence.

[0040] The key sentence screening unit is used to calculate the weight of the key sentence according to the weight of each keyword in the key sentence, and eliminate the key sentence whose weight is less than the key sentence weight threshold.

[0041] Optionally, the core text data clustering cleaning module includes the following units:

[0042] The network document category determination unit is used to determine the category of network documents related to cadmium and arsenic pollution corresponding to the core text data by using a random forest algorithm based on a key sentence set of the core text data.

[0043] The network document clustering unit is used to cluster all network documents related to cadmium and arsenic pollution using the k-means clustering method to obtain the initial network document clustering results.

[0044] The network document cleaning unit is used to calculate the similarity between any two network documents of the same category based on the initial network document clustering results, and to deduplicate the data whose similarity exceeds a threshold to eliminate duplicate records; it is also used to perform a secondary analysis on the outliers of the initial network document clustering results obtained by the k-means clustering method, and to delete the network documents corresponding to the outliers based on the results of the secondary analysis.

[0045] The network document clustering result output unit is used to obtain the network document clustering result according to the cleaning operation of the network document cleaning unit on the initial network document clustering result.

[0046] Optionally, based on the initial network document clustering results, the similarity between any two network documents of the same category is calculated, specifically including: based on the initial network document clustering results, taking two network documents of the same category whose similarity is to be calculated; based on the clustering results, obtaining the distance between any two key sentences between the two network documents as the similarity between the two key sentences, and calculating the average similarity between all key sentences as the similarity between the two network documents.

[0047] Optionally, the knowledge graph construction and visualization module adopts a dynamic semantic mapping framework to construct a cadmium and arsenic pollution knowledge graph; the dynamic semantic mapping framework maintains a three-level knowledge graph and runs an update mechanism; the three-level knowledge graph includes a basic layer, a relationship layer and a dynamic layer; the basic layer includes entities in each key sentence and the attributes of the entities, the relationship layer includes the relationships between entities, and the dynamic layer is used to describe the attributes of entities that change over time and the relationships between entities that change over time; the update mechanism is used to regularly obtain the results of network document clustering, and update the dynamic layer in the three-level knowledge graph according to the results of network document clustering.

[0048] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0049] The present application provides a method and system for integrated management of knowledge on collaborative prevention and control of cadmium and arsenic pollution at the production site. The core text data collection module uses multi-threaded parallel collection technology to collect a large amount of online documents related to cadmium and arsenic pollution by searching keywords at preset sites, extract the core text data in the online documents related to cadmium and arsenic pollution, and classify the core text data using a preset classification system to mark the category information of the core text data; the core text data is pre-processed by the core text data reorganization module to perform named entity recognition and screening to obtain a set of key sentences of the core text data, and then the core text data clustering and cleaning module uses a random forest algorithm based on the key sentence set of the core text data to determine the category of the online documents related to cadmium and arsenic pollution corresponding to the core text data, and the k-means clustering method is used to cluster all online documents related to cadmium and arsenic pollution to obtain the clustering results of the online documents; finally, the knowledge graph construction and visualization module is used to construct a knowledge graph of cadmium and arsenic pollution according to the clustering results of the online documents, and the knowledge graph of cadmium and arsenic pollution is visualized. Through the literature knowledge integration engine and multi-threaded parallel acquisition technology, the system can efficiently and automatically collect cadmium and arsenic pollution-related data from diverse data sources, and perform standardized processing to ensure the consistency and comparability of the data. At the same time, it uses a knowledge graph to display the clustering results of online literature related to cadmium and arsenic pollution, and can provide multi-angle and multi-dimensional data analysis and display functions to meet the needs of users in different fields and at different levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A schematic diagram of the functional modules of a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the production site provided in one embodiment of the present application.

[0052] Figure 2 A schematic diagram of the functional modules of a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the production site provided in one embodiment of the present application.

[0053] Figure 3 A schematic diagram of the functional modules of a core text data collection module in a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the origin provided in one embodiment of the present application.

[0054] Figure 4A schematic diagram of the functional modules of a core text data reorganization module in a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the origin provided in one embodiment of the present application.

[0055] Figure 5 A schematic diagram of the functional modules of a core text data clustering and cleaning module in a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the origin provided in one embodiment of the present application.

[0056] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0058] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0059] In an exemplary embodiment, Figure 1 The flowchart shown provides a knowledge integration management method for collaborative prevention and control of cadmium and arsenic pollution at the production site, including the following steps:

[0060] S1. Use multi-threaded parallel collection technology to collect online literature related to cadmium and arsenic pollution by searching keywords at preset sites; the preset sites are websites that are highly related to cadmium and arsenic pollution.

[0061] S2. Extract the core text data from online literature related to cadmium and arsenic pollution, and classify the core text data using a preset classification system to mark the category information of the core text data.

[0062] S3. After preprocessing the core text data, named entity recognition and screening are performed to obtain a set of key sentences of the core text data.

[0063] S4. Use the random forest algorithm based on the key sentence set of the core text data to determine the category of online literature related to cadmium and arsenic pollution corresponding to the core text data.

[0064] S5. Use the k-means clustering method to cluster all online literature related to cadmium and arsenic pollution to obtain the network literature clustering results.

[0065] S6. Construct a knowledge graph of cadmium and arsenic pollution based on the results of online literature clustering, and visualize the knowledge graph of cadmium and arsenic pollution.

[0066] In a specific embodiment, step S1 includes the following steps:

[0067] Websites highly related to cadmium and arsenic pollution are obtained as preset sites, and a priority is set for each preset site, and search keywords are set for subsequent searches.

[0068] The search results are sorted in descending order according to the priority of the preset site and the actual weight of the search keyword, and the first preset number of search results in the sort are obtained as network literature related to cadmium and arsenic pollution.

[0069] For each online document related to cadmium and arsenic pollution, content analysis and bibliometric methods are used to deeply understand the content and theme of the document and extract the corresponding core text data.

[0070] The core text data is classified using a preset classification system to mark the category information of the core text data.

[0071] In a specific embodiment, step S3 specifically includes the following steps:

[0072] The named entities are recognized after preprocessing the core text data; the preprocessing includes unified naming specifications and time and place format specifications.

[0073] Delete the high-frequency grammatical function words and low-frequency words in the identified named entities, and process the retained named entities as candidate words.

[0074] The candidate words are input into the BERT model to calculate the corresponding weights, and the candidate words with weights greater than the keyword weight threshold are selected as keywords; multiple keywords form a key sentence.

[0075] The weight of the key sentence is calculated according to the weight of each keyword in the key sentence, and the key sentences with weights less than the key sentence weight threshold are eliminated to obtain the key sentence set of the core text data.

[0076] In another exemplary embodiment, Figure 2 As shown, a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the production site is provided, including the following functional modules:

[0077] The core text data collection module is used to use multi-threaded parallel collection technology to collect online documents related to cadmium and arsenic pollution by searching keywords at preset sites; it is also used to extract core text data from online documents related to cadmium and arsenic pollution, and classify the core text data using a preset classification system to mark the category information of the core text data; the preset sites are websites that are highly related to cadmium and arsenic pollution.

[0078] The core text data reorganization module is used to perform named entity recognition and screening on the core text data after preprocessing to obtain a set of key sentences of the core text data.

[0079] The core text data clustering and cleaning module is used to determine the category of cadmium and arsenic pollution-related online documents corresponding to the core text data based on the key sentence set of the core text data using the random forest algorithm; it is also used to cluster all cadmium and arsenic pollution-related online documents using the k-means clustering method to obtain the network document clustering results.

[0080] The knowledge graph construction and visualization module is used to construct a cadmium and arsenic pollution knowledge graph based on the results of online literature clustering, and to visualize the cadmium and arsenic pollution knowledge graph.

[0081] Specifically, Figure 3 As shown in the structure, the core text data collection module includes the following units:

[0082] The cadmium-arsenic pollution highly correlated website marking unit is used to obtain websites that are highly correlated with cadmium-arsenic pollution as preset sites and set a priority for each preset site.

[0083] The search keyword setting unit is used to set the search keyword for the next search.

[0084] The search result sorting unit is used to sort the search results in descending order according to the priority of the preset site and the actual weight of the search keyword, and obtain the first preset number of search results in the sorting as network documents related to cadmium and arsenic pollution.

[0085] The core text data extraction unit is used to use content analysis and bibliometrics to deeply understand the content and theme of each online document related to cadmium and arsenic pollution, and extract the corresponding core text data. In this embodiment, the core text data extracted by the core text data extraction unit will be saved as raw data in JSON format and transferred to the original database for effective storage and management; users can quickly access and obtain these raw data through simple search operations, providing a convenient and reliable channel for obtaining literature knowledge in the academic research field.

[0086] The core text data screening unit is used to classify the core text data using a preset classification system to mark the category information of the core text data. Specifically, based on the extracted core text data and its content and subject, the core text data content is subject-classified and screened, and the network documents that meet the preset classification will be transferred to the database of the corresponding category for storage and management, and the classification information will be added to the core text data.

[0087] Specifically, Figure 4 As shown in the structure, the core text data reorganization module includes the following units:

[0088] The named entity recognition unit is used to recognize named entities after preprocessing the core text data; the preprocessing includes unified naming specifications and time and place format specifications.

[0089] The named entity elimination unit is used to delete high-frequency grammatical function words and low-frequency words in the identified named entities, and process the retained named entities as candidate words.

[0090] The candidate vocabulary screening unit is used to input the candidate vocabulary into the BERT model to calculate the corresponding weight, and select the candidate vocabulary with a weight greater than the keyword weight threshold as the keyword; multiple keywords constitute a key sentence.

[0091] The key sentence screening unit is used to calculate the weight of the key sentence according to the weight of each keyword in the key sentence, and eliminate the key sentence whose weight is less than the key sentence weight threshold.

[0092] Specifically, Figure 5 As shown in the structure, the core text data clustering and cleaning module includes the following units:

[0093] The network document category determination unit is used to determine the category of network documents related to cadmium and arsenic pollution corresponding to the core text data by using a random forest algorithm based on a key sentence set of the core text data.

[0094] The network document clustering unit is used to cluster all network documents related to cadmium and arsenic pollution using the k-means clustering method to obtain the initial network document clustering results.

[0095] The network document cleaning unit is used to calculate the similarity between any two network documents of the same category according to the initial network document clustering result, and to remove the data with similarity exceeding the threshold value to eliminate duplicate records; it is also used to perform a secondary analysis on the outliers of the initial network document clustering result obtained by the k-means clustering method, and delete the network documents corresponding to the outliers according to the result of the secondary analysis. In this embodiment, according to the initial network document clustering result, the similarity between any two network documents of the same category is calculated, specifically including: according to the initial network document clustering result, two network documents of the same category whose similarity is to be calculated are taken; according to the clustering result, the distance between any two key sentences between the two network documents is obtained as the similarity between the two key sentences, and the average of the similarities between all key sentences is calculated as the similarity between the two network documents.

[0096] The network document clustering result output unit is used to obtain the network document clustering result according to the cleaning operation of the network document cleaning unit on the initial network document clustering result.

[0097] As an exemplary scheme, the knowledge graph construction and visualization module adopts a dynamic semantic mapping framework to construct a cadmium and arsenic pollution knowledge graph; the dynamic semantic mapping framework maintains a three-level knowledge graph and runs an update mechanism; the three-level knowledge graph includes a basic layer, a relationship layer and a dynamic layer; the basic layer includes entities in each key sentence and the attributes of the entities, the relationship layer includes the relationships between entities, and the dynamic layer is used to describe the attributes of entities that change over time and the relationships between entities that change over time; the update mechanism is used to regularly obtain the results of network document clustering, and update the dynamic layer in the three-level knowledge graph according to the results of network document clustering.

[0098] The core idea of ​​the dynamic semantic mapping framework is to use a multi-level semantic network to represent knowledge entities and the relationships between them. Each level represents a different level of abstraction or perspective, allowing the system to understand and analyze data from multiple dimensions. The dynamic semantic mapping framework can dynamically improve the knowledge graph based on key sentences and the gradual input of external data to the system. The data represented by the dynamic semantic mapping framework conforms to the data model of the knowledge graph, making the knowledge graph more flexible and easy to expand. It is also conducive to cross-system sharing and integration, and the associated information of cadmium and arsenic pollution is displayed through graph visualization to help users gain a deeper understanding.

[0099] The above system of this application uses a literature knowledge integration engine and multi-threaded parallel acquisition technology. The system can efficiently and automatically collect data related to cadmium and arsenic pollution from a variety of data sources, and perform standardized processing to ensure the consistency and comparability of the data. At the same time, it uses a knowledge graph to display the clustering results of online literature related to cadmium and arsenic pollution, and can provide multi-angle and multi-dimensional data analysis and display functions to meet the needs of users in different fields and at different levels.

[0100] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it realizes the function of a knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at the production site provided in the above embodiment.

[0101] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0102] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0103] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0104] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0105] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0106] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0107] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A knowledge integration management method for collaborative prevention and control of cadmium and arsenic pollution at production sites, characterized in that: include: Using multi-threaded parallel collection technology, we collected online literature related to cadmium and arsenic pollution by searching keywords at preset sites; The preset sites are websites that are highly related to cadmium and arsenic pollution; Extract the core text data from online literature related to cadmium and arsenic pollution, and classify the core text data using a preset classification system to mark the category information of the core text data; After preprocessing the core text data, named entity recognition and screening are performed to obtain a key sentence set of the core text data; Using a random forest algorithm based on a key sentence set of the core text data, determining the category of the network literature related to cadmium and arsenic pollution corresponding to the core text data; The k-means clustering method was used to cluster all network literature related to cadmium and arsenic pollution, and the network literature clustering results were obtained; A cadmium and arsenic pollution knowledge graph is constructed based on the network literature clustering results, and the cadmium and arsenic pollution knowledge graph is visually displayed.

2. The method for integrated management of knowledge on collaborative prevention and control of cadmium and arsenic pollution at production sites according to claim 1, characterized in that: Using multi-threaded parallel collection technology, we collected online literature related to cadmium and arsenic pollution by searching keywords at preset sites, including: Obtain websites that are highly related to cadmium and arsenic pollution as preset sites, set a priority for each preset site, and set search keywords for subsequent searches; The search results are sorted in descending order according to the priority of the preset sites and the actual weight of the search keywords, and the first preset number of search results in the sorting are obtained as network documents related to cadmium and arsenic pollution; For each online document related to cadmium and arsenic pollution, content analysis and bibliometrics were used to deeply understand the content and theme of the document and extract the corresponding core text data; The core text data is classified using a preset classification system to mark the category information of the core text data.

3. The method for integrated management of knowledge on collaborative prevention and control of cadmium and arsenic pollution at production sites according to claim 1 is characterized in that: After preprocessing the core text data, named entity recognition and screening are performed to obtain a key sentence set of the core text data, specifically including: Recognize named entities after preprocessing the core text data; the preprocessing includes unified naming standards and time and place format standards; Delete the high-frequency grammatical function words and low-frequency words in the identified named entities, and process the retained named entities as candidate words; The candidate words are input into the BERT model to calculate the corresponding weights, and the candidate words with weights greater than the keyword weight threshold are selected as keywords; multiple keywords form a key sentence; The weight of the key sentence is calculated according to the weight of each keyword in the key sentence, and the key sentences with weights less than the key sentence weight threshold are eliminated to obtain the key sentence set of the core text data.

4. A knowledge integration management system for collaborative prevention and control of cadmium and arsenic pollution at production sites, characterized in that: include: The core text data collection module is used to collect online literature related to cadmium and arsenic pollution by searching keywords at preset sites using multi-threaded parallel collection technology; It is also used to extract core text data from online documents related to cadmium and arsenic pollution, and classify the core text data using a preset classification system to mark the category information of the core text data; the preset site is a website that is highly related to cadmium and arsenic pollution; A core text data reorganization module is used to perform named entity recognition and screening on the core text data after preprocessing to obtain a key sentence set of the core text data; The core text data clustering cleaning module is used to determine the category of the network documents related to cadmium and arsenic pollution corresponding to the core text data based on the key sentence set of the core text data by using the random forest algorithm; and is also used to cluster all network documents related to cadmium and arsenic pollution by using the k-means clustering method to obtain the network document clustering results; The knowledge graph construction and visualization module is used to construct a cadmium and arsenic pollution knowledge graph based on the network literature clustering results, and to visualize the cadmium and arsenic pollution knowledge graph.

5. The knowledge integrated management system for collaborative prevention and control of cadmium and arsenic pollution at the production site according to claim 4 is characterized in that: The core text data collection module includes: A cadmium-arsenic pollution highly correlated site marking unit, used to obtain sites that are highly correlated with cadmium-arsenic pollution as preset sites, and set a priority for each preset site; A search keyword setting unit, used to set search keywords for the next search; A search result sorting unit, used to sort the search results in descending order according to the priority of the preset site and the actual weight of the search keyword, and obtain the first preset number of search results in the sorting as network documents related to cadmium and arsenic pollution; The core text data extraction unit is used to use content analysis and bibliometrics to deeply understand the content and theme of each online document related to cadmium and arsenic pollution, and extract the corresponding core text data; The core text data screening unit is used to classify the core text data using a preset classification system to mark the category information of the core text data.

6. The knowledge integrated management system for collaborative prevention and control of cadmium and arsenic pollution at the production site according to claim 5 is characterized in that: The core text data extracted by the core text data extraction unit will be saved as original data in JSON format and transferred to the original database for effective storage and management; users can quickly access and obtain these original data through simple search operations.

7. The knowledge integrated management system for collaborative prevention and control of cadmium and arsenic pollution at the production site according to claim 4 is characterized in that: The core text data reorganization module includes: A named entity recognition unit, used for recognizing named entities after preprocessing the core text data; the preprocessing includes unified naming specifications and time and place format specifications; A named entity elimination unit is used to delete high-frequency grammatical function words and low-frequency words in the identified named entities, and process the retained named entities as candidate words; A candidate vocabulary screening unit is used to input the candidate vocabulary into the BERT model to calculate the corresponding weight, and select the candidate vocabulary with a weight greater than the keyword weight threshold as a keyword; multiple keywords form a key sentence; The key sentence screening unit is used to calculate the weight of the key sentence according to the weight of each keyword in the key sentence, and eliminate the key sentence whose weight is less than the key sentence weight threshold.

8. The knowledge integrated management system for collaborative prevention and control of cadmium and arsenic pollution at production sites according to claim 4 is characterized in that: The core text data clustering cleaning module includes: A network document category determination unit, used to determine the category of the network document related to cadmium and arsenic pollution corresponding to the core text data by using a random forest algorithm based on a key sentence set of the core text data; The network literature clustering unit is used to cluster all network literature related to cadmium and arsenic pollution using the k-means clustering method to obtain the initial network literature clustering results; A network document cleaning unit is used to calculate the similarity between any two network documents of the same category based on the initial network document clustering result, and to remove duplicate data with similarity exceeding a threshold value to eliminate duplicate records; it is also used to perform a secondary analysis on the outliers of the initial network document clustering result obtained by the k-means clustering method, and to delete the network documents corresponding to the outliers based on the results of the secondary analysis; The network document clustering result output unit is used to obtain the network document clustering result according to the cleaning operation of the network document cleaning unit on the initial network document clustering result.

9. The knowledge integrated management system for collaborative prevention and control of cadmium and arsenic pollution at production sites according to claim 4 is characterized in that: According to the initial network document clustering result, the similarity between any two network documents of the same category is calculated, specifically including: according to the initial network document clustering result, two network documents of the same category whose similarity is to be calculated are selected; according to the clustering result, the distance between any two key sentences between the two network documents is obtained as the similarity between the two key sentences, and the average similarity between all key sentences is calculated as the similarity between the two network documents.

10. The knowledge integrated management system for collaborative prevention and control of cadmium and arsenic pollution at production sites according to claim 4 is characterized in that: The knowledge graph construction and visualization module adopts a dynamic semantic mapping framework to construct the cadmium and arsenic pollution knowledge graph; the dynamic semantic mapping framework maintains a three-level knowledge graph and runs an update mechanism; the three-level knowledge graph includes a basic layer, a relationship layer and a dynamic layer; the basic layer includes entities in each key sentence and the attributes of the entities, the relationship layer includes the relationships between entities, and the dynamic layer is used to describe the attributes of entities that change over time and the relationships between entities that change over time; the update mechanism is used to regularly obtain network document clustering results, and update the dynamic layer in the three-level knowledge graph according to the network document clustering results.

Images