Data visualization processing method and system
Through Python network crawler and hybrid database storage technology, combined with two-way long and short-term memory network and conditional random field model, the problem of cultural relics data collection and visualization is solved, efficient management and in-depth analysis of cultural relics data is achieved, and the quality of the knowledge graph is improved.
Patent Information
- Application Number
- CN202510375351.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-27
- Publication Date
- 2025-08-01
AI Technical Summary
The existing technology is difficult to efficiently collect and integrate cultural relics data from multiple sources and formats, resulting in missing or incomplete data, poor visual display effect, lack of interactiveness and in-depth analysis capabilities, insufficient accuracy in identifying named entities, affecting the comprehensive analysis and research of cultural relics data.
Python network crawling technology is used to collect data, combine NoSQL and relational database storage, and use two-way long and short-term memory networks and conditional random field models to clean and standardize data to build a high-quality knowledge graph.
It realizes comprehensive coverage and efficient management of cultural relics data, improves the integrity and consistency of data, improves the accuracy of knowledge extraction and visual interaction, and supports the construction of high-quality cultural relics knowledge graphs.
Smart Images
Figure CN120407666A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data visualization, and particularly relates to a data visualization processing method and system. Background Art
[0002] In data visualization technology, the basic idea is to represent each data item in a database as a single graphic element. A large dataset constitutes a data image, and at the same time, the attribute values of the data are represented in the form of multi-dimensional data, enabling data to be observed from different dimensions, thereby allowing for a more in-depth observation and analysis of the data. In the continuous development of visualization technology, the visualization of large amounts of data is optimized to solve the problem of unoptimizable effects on web pages. First, the data visualization is optimized and analyzed, and then query display is performed. At the same time, the technology of interactive visualization in the process of interactive information processing has thus attracted much attention.
[0003] Some specific data in the prior art, including historical and cultural relic data, etc., have irreplaceable value for inheriting national culture, studying historical processes, and promoting cultural development. The existing cultural relic and historical data have the following problems:
[0004] 1. Difficult data collection and integration: The sources of cultural relic data are extensive, including museums, archives, literature materials, etc. The data formats are diverse, including both structured data, such as the basic information of cultural relics in the museum collection database, and unstructured data, such as the historical background introduction of cultural relics and archaeological excavation reports. Traditional data collection methods are difficult to comprehensively and efficiently obtain this data, and face many challenges in integrating data from different sources and formats, easily resulting in data loss or incompleteness. The data sharing and communication between different cultural relic collection units are limited, forming data islands, which hinder the comprehensive analysis and in-depth study of cultural relic data.
[0005] 2. Poor visualization display effect: The existing data visualization processing methods for cultural relic data usually can only display simple data charts, lacking in-depth exploration of the complex relationships and deep information among cultural relics. Lack of interactivity and dynamics, users cannot flexibly explore cultural relic data according to their own needs. When faced with a large amount of cultural relic data, users are difficult to quickly locate the cultural relic information they are interested in, nor can they deeply understand the stories and related knowledge behind the cultural relics.
[0006] 3. Imperfect knowledge extraction and graph construction: The accuracy of named entity recognition in cultural relic text information needs to be improved. Cultural relic texts contain rich historical and cultural knowledge, but due to their professionalism and complexity, the existing named entity recognition methods are prone to missed recognition or misrecognition, affecting the accurate understanding of cultural relic information and the construction quality of knowledge graphs.
[0007] Traditional data analysis tools often struggle to handle large-scale unstructured historical data and have limitations in data cleaning, standardization, and visual display. Summary of the Invention
[0008] In view of the problems existing in the prior art mentioned in the background art, the present invention proposes a data visualization processing method and system.
[0009] Technical Solution: To solve the above technical problems, the technical solution adopted by the present invention is as follows:
[0010] A data visualization processing method, comprising the following steps:
[0011] S1: Data collection: Collect the required resource data to establish a data set;
[0012] S2: Data preprocessing: Clean and standardize the collected data;
[0013] S3: Data storage: Store the preprocessed data;
[0014] S4: Model construction: Establish a model for knowledge extraction;
[0015] S5: Visual display analysis: Extract and display the collected data based on the established model.
[0016] Preferably, in S1, the Python web crawler technology is used. For the structured data published on government websites, it is crawled by analyzing Ajax requests and using corresponding programming means; for unstructured data, a combination of Request and regular expressions is used for extraction.
[0017] Preferably, in S2, an automated cleaning tool is used to handle missing values, outliers, and duplicate data; a standardization tool is used to unify the formats of key information such as time, location, and people.
[0018] Preferably, in S3, according to the characteristics of the collected data, the corresponding table structure and field names are set up to establish a data resource database;
[0019] Adopt a hybrid storage method of NoSQL and relational databases, store the structured data in the Mysql database; store the knowledge graph data in the Neo4j graph database.
[0020] Preferably, in S4, it specifically includes:
[0021] S41: Bidirectional long short-term memory network;
[0022] The bidirectional long short-term memory network includes an input layer, a forward layer LSTM, a backward layer LSTM network, and an output layer. The specific calculation is as follows:
[0023]
[0024] The forward layer processes the input vectors from the start time to the nth time in the input order to calculate the forward hidden state ; the backward layer LSTM network processes the input vectors from the start time to the nth time in the same order to calculate the backward hidden state ; after the calculation and processing of the forward layer and the backward layer, the input vector has two vectors in different directions at any time and , and finally the two vectors are concatenated to obtain ;
[0025] S42: Conditional random field model;
[0026] S43: Classify the processed data.
[0027] Preferably, in S42, the specific content of the conditional random field model is:
[0028] The conditional random field model outputs another set of random sequences under the influence of the sequence of X by inputting a set of known and definite random sequences P(Y|X).
[0029] A data visualization processing system, adopting any of the above data visualization processing methods, includes:
[0030] Data acquisition module: Collect the required resource data to establish a data set;
[0031] Data preprocessing module: Clean and standardize the collected data;
[0032] Data storage module: Store the preprocessed data;
[0033] Data modeling module: Establish a model for knowledge extraction;
[0034] Visualization analysis module: Extract and display the collected data based on the established model.
[0035] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.
[0036] A computer-readable storage medium stores a computer program thereon, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0037] Beneficial effects: Compared with the prior art, the present invention has the following advantages:
[0038] (1) Through Python web crawler technology, the present invention adopts specific means to collect data of different structures, which can comprehensively cover various sources of cultural relic data. At the same time, a hybrid storage method of NoSQL and relational databases is adopted, giving full play to the advantages of both databases. Relational databases (such as Mysql) are suitable for storing structured data, facilitating complex queries, data consistency maintenance, and transaction processing. In the present invention, it can ensure the efficient retrieval and management of data. For example, it can quickly query cultural relic information of a specific era, category, or collection location. Graph databases (such as Neo4j) are good at processing complex relational data. Storing knowledge-based graph data in Neo4j can intuitively display the association relationships between cultural relics and between cultural relics and entities such as historical figures and locations.
[0039] (2) The application of the automated cleaning tool of the present invention can effectively process missing values, outliers, and duplicate data in the data. For missing values, the system can select appropriate filling methods according to the distribution characteristics of the data and business requirements, such as mean, median, mode, or interpolation method, etc., to ensure the integrity of the data and avoid analysis result deviation caused by data missing. Outlier processing can identify and handle errors or abnormal situations in the data to ensure the rationality of the data. Standardization improves data consistency.
[0040] (3) With the assistance of bidirectional long short-term memory networks and conditional random field models, the present invention can continuously improve the accuracy and efficiency of knowledge extraction as data accumulates when processing a large amount of cultural relic and historical data, providing strong support for constructing a high-quality cultural relic knowledge graph. Description of the Drawings
[0041] Figure 1 is the technical flow chart of the present invention. Detailed Embodiments
[0042] The following further clarifies the present invention in conjunction with specific embodiments. The embodiments are implemented on the premise of the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention.
[0043] As Figure 1 shown, the data visualization processing method provided in this embodiment includes the following steps:
[0044] S1: Data collection: Collect the required resource data to establish a data set;
[0045] Data collection: Using Python web crawler technology, specific collection methods are adopted for data from different sources.
[0046] For the structured data published on government websites (such as the Chinese Museum Dataset and the Chinese Cultural Relics in Collection Dataset), by analyzing Ajax requests and adopting corresponding programming means (such as multi-threaded programming) for scraping; for the unstructured data on Baidu Encyclopedia, a combination of Request and regular expressions is used for extraction.
[0047] The collected data covers a wide range, including the basic attributes of cultural relics (such as name, age, category, collection location, etc.), relevant historical backgrounds, unearthed information, and historical figures involved, etc., to ensure the comprehensiveness and richness of the data.
[0048] S2: Data preprocessing: Clean and standardize the collected data;
[0049] Adopt automated cleaning tools to handle missing values, outliers, and duplicate data; use standardization tools to unify the formats of key information such as time, location, and people.
[0050] S3: Data storage: Store the preprocessed data;
[0051] Design a reasonable table structure and field names according to the characteristics of the collected data, and establish a cultural relic resource database. For example, design a table for museum data containing fields such as collection unit, province where located, number of collections, number of visitors, etc.; design a table for cultural relics in collection data containing fields such as census registration number, name, category, age, collection museum, etc.; design a table for cultural relics prohibited from going abroad data containing fields such as number, name, era, unearthed time, collection unit, etc.
[0052] Adopt a mixed storage method of NoSQL and relational databases, store the structured data in a Mysql database to facilitate data query, management, and analysis; store the knowledge graph data in a Neo4j graph database, and utilize its advantage of efficiently processing complex relationship data to facilitate the display of the association relationships between cultural relics.
[0053] S4: Build a model: Establish a model for knowledge extraction;
[0054] S41: Bidirectional long short-term memory network;
[0055] Bidirectional Long Short Term Memory Neural Network (Long Short Term Memory, LSTM): LSTM is a special type of RNN. This model introduces the mechanism of "gates" and adds special memory units in the hidden layer. By selectively discarding and retaining the input data, it can transmit information over long distances to the current moment, thereby better discovering the interdependencies in the data over a long period of time.
[0056] Ordinary recurrent neural networks only consider the influence of the previous moment and do not consider the influence of the next moment. Therefore, a bidirectional recurrent neural network model needs to be introduced. However, LSTM extracts information based on the context information of the current word, and in the application of entities, the context information of the current word is equally important. The Bidirectional Long Short Term Memory (Bi-LSTM) network model precisely makes up for this deficiency and can include text context information for entity extraction.
[0057] The bidirectional long short term memory network consists of four parts: an input layer, a forward LSTM layer, a backward LSTM layer, and an output layer. The specific calculations are as follows:
[0058]
[0059] The forward layer processes the input vectors from the start time to the nth time in the input order to calculate the forward hidden state . The backward LSTM layer processes the input vectors from the start time to the nth time in the same order to calculate the backward hidden state . After the calculations of the forward layer and the backward layer, the input vector has two vectors in different directions at any time and . Finally, the two vectors are concatenated to obtain .
[0060] S42: Conditional Random Field Model;
[0061] The Conditional Random Field (CRF) model outputs the probability P(Y|X) of another set of random sequences under the influence of the sequence of X by inputting a set of known and definite random sequences . It has been widely used in natural language processing. For sequence labeling problems such as named entity recognition, the linear conditional random field (linear-CRF) is usually adopted.
[0062] Conditional random fields usually solve three types of problems: evaluation, decoding, and learning. In solving sequence labeling problems, CRF is commonly used for decoding implementation. Among them, sequence X represents the input observation sequence, and sequence Y represents the sequence that has been labeled. When conducting the evaluation problem, for known random input variables, by solving the conditional probability of the sequence, the output sequence with the maximum probability is obtained. When solving the learning problem, through training on the dataset, and under certain regularization conditions, the maximum likelihood estimation is used to derive the conditional probability P(Y|X).
[0063] S43: Classify the processed data;
[0064] Based on the K-means clustering algorithm and the Pearson correlation coefficient for data classification, the K-means clustering algorithm is one of the classic algorithms in partitioning-based clustering algorithms. It is an iterative process, and the specific steps are mainly as follows: First, randomly select k data objects as the initial centers; Second, for each remaining object, according to its distance from each cluster center, assign it to the cluster corresponding to the cluster center that is most similar to it; Then recalculate the average value of all objects in each cluster as the new cluster center. Continuously repeat the above process until the criterion function converges, that is, the cluster centers do not change significantly. Usually, the mean square error is used as the criterion function, that is, to minimize the sum of the squares of the distances from each point to the nearest cluster center.
[0065] The Pearson correlation coefficient, its definition formula is:
[0066]
[0067] Among them, and respectively represent the expectations of vectors X and Y, and Y i respectively represent the components of vectors X and Y. The absolute value of r ranges from 0 to 1, and n represents the lengths of vectors X and Y.
[0068] S5: Visualization display analysis: Extract and display the collected data based on the established model.
[0069] A data visualization processing system adopts the above data visualization processing method, specifically including: a data acquisition module, a data preprocessing module, a data storage module, a data modeling module, and a visualization analysis module.
[0070] Data acquisition module: Acquire the required resource data to establish a dataset.
[0071] Data preprocessing module: An automated cleaning tool is used to process missing values, outliers, and duplicate data; standardization tools are utilized to unify the formats of key information such as time, location, and people.
[0072] Data storage module: Stores the preprocessed data.
[0073] Data modeling module: Establishes a model for knowledge extraction.
[0074] Visualization analysis module: Extracts and displays the collected data based on the established model.
[0075] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the above method are implemented.
[0076] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the above method are implemented.
[0077] Through Python web crawler technology, the present invention uses specific means to collect data with different structures, which can comprehensively cover various sources of cultural relic data; at the same time, a hybrid storage method of NoSQL and relational databases is adopted to give full play to the advantages of both databases; the use of automated cleaning tools can effectively process missing values, outliers, and duplicate data in the data, ensuring the rationality of the data. Standardization improves data consistency; with the assistance of bidirectional long short-term memory networks and conditional random field models, when dealing with a large amount of cultural relics and historical data, the accuracy and efficiency of knowledge extraction can be continuously improved with the accumulation of data, providing strong support for constructing a high-quality cultural relic knowledge graph.
[0078] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.
Claims
1. A data visualization processing method, characterized in that: It includes the following steps: S1: Data collection: Collect the required resource data to establish a data set; S2: Data preprocessing: Clean and standardize the collected data; S3: Data storage: Store the preprocessed data; S4: Model construction: Establish a model for knowledge extraction; S5: Visualization display and analysis: Extract and display the collected data based on the established model.
2. The data visualization processing method according to claim 1, wherein: In S1, using Python web crawler technology, for the structured data published on government websites, it is crawled by analyzing Ajax requests and using corresponding programming means; for unstructured data, a combination of Request and regular expressions is used for extraction.
3. The data visualization processing method according to claim 1, wherein: In S2, an automated cleaning tool is used to handle missing values, outliers, and duplicate data; a standardization tool is used to unify the formats of key information such as time, location, and person.
4. The data visualization processing method according to claim 1, characterized in that: In S3, corresponding table structures and field names are set according to the characteristics of the collected data to establish a data resource database; Adopting a hybrid storage method of NoSQL and relational databases, the structured data is stored in the Mysql database; the knowledge graph data is stored in the Neo4j graph database.
5. The data visualization processing method according to claim 1, wherein: In S4, it specifically includes: S41: Bidirectional long short-term memory network; The bidirectional long short-term memory network includes an input layer, a forward layer LSTM, a backward layer LSTM network, and an output layer. The specific calculation is as follows: ； The forward layer processes the input vectors from the start time to the nth time in the input order and calculates the forward hidden state ; the backward layer LSTM network processes the input vectors from the start time to the nth time in the same order and calculates the backward hidden state ; after the calculation and processing of the forward layer and the backward layer, the input vector has two vectors in different directions at any time and , and finally the two vectors are concatenated to obtain ; S42: Conditional random field model; S43: Classify the processed data.
6. The data visualization processing method according to claim 5, wherein: In S42, the specific content of the conditional random field model is: The conditional random field model takes a set of known and definite random sequences as input and, under the influence of the sequence of X, outputs another set of random sequences P(Y|X).
7. A data visualization processing system, which is based on the data visualization processing method described in any one of the above claims 1-6, and is characterized in that: It includes: Data collection module: Collect the required resource data to establish a data set; Data preprocessing module: Clean and standardize the collected data; Data storage module: Store the preprocessed data; Data modeling module: Establish a model for knowledge extraction; Visualization analysis module: Extract and display the collected data based on the established model.
8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the above method when executing the program.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program implements the steps of the above method when executed by the processor.