Data management system for scientific and technological achievement transformation

By introducing data acquisition, analysis and display modules into the data management system and combining with the distributed storage architecture, the data management system's incomplete collection, insufficient analysis and low storage efficiency in the transformation of scientific and technological achievements has been solved, and the data is high Quality and intuitive display support high concurrency demand and promote the effective transformation of scientific and technological achievements.

CN118916641AInactive Publication Date: 2024-11-08SHANDONG IND RESEARCH BOZHENG INNOVATION CONSULTING CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202411091769.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-11-08
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing data management system has problems such as incomplete and inaccurate data collection, insufficient data analysis depth, low data storage efficiency, and unintuitive data display in the transformation of scientific and technological achievements, which is difficult to meet the in-depth understanding and effective utilization of scientific and technological achievements.

Method used

A data management system including data acquisition, data analysis, data storage and data display modules was designed. By automatically collecting multi-source data, using the reduced candidate set generation and optimization Apriori algorithm to discover potential associations, adopting distributed architecture to efficiently store, adopting Multiple verification rules and interactive verification ensure data quality and present analysis results with intuitive graphs and reports.

Benefits of technology

It realizes the accuracy, completeness and in-depth analysis of data, improves data storage efficiency and intuitive display, supports high concurrency needs, and promotes the effective transformation and application of scientific and technological achievements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916641A_ABST
    Figure CN118916641A_ABST
Patent Text Reader

Abstract

The invention relates to the field of data management systems, in particular to a data management system for scientific and technological achievement transformation. Comprising data acquisition, analysis, storage and display modules. The data acquisition module automatically acquires data from multiple sources, ensures accuracy and completeness and removes repeated invalid information; the data analysis module discovers potential association between results by using an optimized Apriori algorithm; the data storage module adopts a distributed architecture to efficiently store data, supports rapid retrieval and access, and meets the high-concurrency requirement; and the data display module displays an analysis result in a visual form to facilitate user understanding. The data quality of data acquisition is ensured through various verification and detection mechanisms; the data storage utilizes a distributed architecture and backup to ensure the safety and reliability of the data; the data display has the functions of interaction, multi-dimensional display and report generation. The system can effectively solve the problems in the aspects of data acquisition, analysis, storage, display and the like, and promotes conversion and application of scientific and technological achievements.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data management systems, and in particular to a data management system for transformation of scientific and technological achievements. Background Art

[0002] In today's era of rapid technological development, the transformation of scientific and technological achievements is of great significance to promoting economic growth and social progress. However, the existing data management system has some limitations in the transformation of scientific and technological achievements. In terms of data collection, it is often impossible to ensure the accuracy and completeness of data, and it is difficult to remove duplicate and invalid information, resulting in low data quality. In terms of data analysis, there is a lack of in-depth analysis capabilities, and it is difficult to use advanced data mining to extract valuable information and patterns, and it is impossible to discover the potential correlation between achievements, which affects the in-depth understanding and effective use of scientific and technological achievements. In addition, data storage also faces challenges. Traditional architectures are difficult to efficiently store large-scale data, cannot support fast retrieval and access, and are difficult to meet high concurrency requirements, which affects the availability and timeliness of data. In terms of data display, it is not intuitive and clear enough, and it is impossible to provide users with a comprehensive and accurate data view in the form of charts, reports, and graphics, which makes it difficult for users to quickly understand the data, affecting the efficiency and accuracy of decision-making.

[0003] To sum up, the existing data management systems cannot meet the needs of transformation of scientific and technological achievements. There is an urgent need for a new data management system that can automatically collect accurate and complete data from multiple sources, use advanced technology for in-depth analysis, use an efficient distributed architecture to store data, and display the analysis results in an intuitive and clear manner to promote the effective transformation and application of scientific and technological achievements. Summary of the invention

[0004] In view of the technical problems existing in the background, the present invention proposes a data management system for the transformation of scientific and technological achievements, aiming to solve the problems of incomplete and inaccurate data collection, insufficient depth of data analysis, low data storage efficiency and non-intuitive data display.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a data management system for the transformation of scientific and technological achievements, including a data acquisition module, a data analysis module, a data storage module and a data display module, wherein the data acquisition module is responsible for automatically collecting data related to scientific and technological achievements from multiple sources, ensuring that the data is accurate and complete, and removing duplicate and invalid information; the data analysis module uses the Apriori algorithm for optimizing the generation of reduced candidate item sets to discover potential associations between achievements; the data storage module uses a distributed architecture to efficiently store large-scale data, supports fast retrieval and access, ensures that data is available at any time, and meets high concurrency requirements; the data display module presents the analysis results with intuitive and clear charts, reports and graphics, and provides users with a comprehensive and accurate data view, which is convenient for users to quickly understand;

[0006] In the data analysis module, the Apriori algorithm is used to optimize the generation of candidate item sets and discover the potential associations between different scientific and technological achievements through association rule mining. The following are the implementation steps:

[0007] Step 1: First, set the minimum support threshold according to the application scenario;

[0008] Step 2: Then scan the transaction database, count the support of each single item, and generate a frequent 1-item set L 1 ;

[0009] Step 3: Then use hashing technology to reduce the number of candidate item sets, hash all possible k item sets into different buckets, count the frequency of item sets in each bucket, and only retain the buckets and their corresponding item sets whose frequency exceeds the minimum support;

[0010] Step 4: Next, reduce the size of the transaction database to avoid repeatedly scanning a large number of transactions in subsequent steps. Delete transactions that do not contain frequent itemsets after each iteration because these transactions cannot generate new frequent itemsets. Merge transactions with the same frequent itemsets to reduce redundancy.

[0011] Step 5: Then prune. For each candidate k-item set, check whether all its k-1 subsets are frequent item sets. If any k-1 subset of a candidate item set is not a frequent item set, then the candidate item set is not frequent and is deleted directly.

[0012] Step 6: For each remaining candidate k-item set, scan the transaction database, count its support, and generate the frequent k-item set L k ;

[0013] Step 7. Finally, repeat steps 4 to 6 until no new frequent item sets can be generated. Generate association rules from the frequent item sets, calculate the confidence of each rule, and filter out the rules that meet the minimum confidence threshold.

[0014] Preferably, the data collection part adopts the following practices to ensure the integrity of the data: first, a variety of data verification rules and logical checks are used to verify the format and range of the input data to ensure that it meets the predetermined specifications; secondly, a duplicate data detection mechanism is set up to promptly discover and remove duplicate data records during the collection process to avoid data redundancy; then, real-time data interaction verification is performed with the data source, and when acquiring data, the accuracy of the data is immediately confirmed. If there is any abnormality, timely feedback is given and re-collection is carried out; finally, a data quality assessment indicator system is established, and the quality of the collected data is regularly assessed. If problems are found, they are corrected and supplemented in a timely manner. When collecting funding data for scientific research projects, verification rules for numerical ranges are set to avoid unreasonable funding values; for duplicate funding records, only the latest and accurate one is retained.

[0015] Preferably, the data storage module adopts a distributed architecture, and the implementation steps are:

[0016] Step 1: First, divide the storage system into multiple nodes, each node independently stores a part of the data; then divide the large-scale data into multiple small blocks, each shard is stored on a different node, and the sharding formula of the data item key is shard_id = hash (key) mod N, where N is the total number of storage nodes and key is the key value of the data item;

[0017] Step 2: To improve data reliability and availability, the data of each shard will be replicated to multiple nodes, which ensures that even if a node fails, the data can still be obtained from other nodes;

[0018] Step 3: Use gzip data compression technology to reduce storage space usage and improve transmission efficiency; design efficient retrieval and access mechanisms;

[0019] Step 4: Then design an efficient retrieval and access mechanism. When reading data, locate the node where the data is located according to the sharding strategy, and then read the data from the node. If the node is unavailable, read from the replica node. When writing data, write the data to the corresponding node according to the sharding strategy and synchronize it to the replica node.

[0020] Step 5. Finally, set up regular automatic backup to ensure that data can be quickly restored in the event of a failure. The backup data is stored in an independent backup node or cloud storage.

[0021] Preferably, the data display module has an interactive function, and the user can dynamically adjust and view data details through operations such as clicking, zooming, and dragging.

[0022] Preferably, the data display module supports multi-dimensional data display, and the user can view data analysis results in different dimensions by filtering and switching dimensions.

[0023] Preferably, the data display module has a report generation function, and the user can generate reports in PDF, Excel and other formats based on the analysis results to facilitate data sharing and archiving.

[0024] Compared with the prior art, the advantages and positive effects of the present invention are:

[0025] 1. The use of multiple data validation rules, logical checks, duplicate data detection mechanisms, real-time data interactive verification, and data quality assessment indicator systems ensures the accuracy and completeness of data, removes duplicate and invalid information, and improves data quality.

[0026] 2. By using the Apriori algorithm to optimize the generation of reduced candidate item sets, we can discover the potential correlation between different scientific and technological achievements, providing support for the in-depth understanding and effective use of scientific and technological achievements.

[0027] 3. The distributed architecture and automatic backup and recovery functions can efficiently store large-scale data, support fast retrieval and access, ensure that data is available at any time, meet high concurrency requirements, and at the same time ensure data security and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 This is a schematic diagram of the structure of the scientific and technological achievement data management system; DETAILED DESCRIPTION

[0030] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0031] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways than those described herein. Therefore, the present invention is not limited to the specific embodiments of the following disclosure.

[0032] In the practical application of scientific and technological achievements transformation, a large amount of relevant data needs to be effectively managed. However, the existing data management methods are difficult to meet the requirements of accuracy and efficiency. In order to better promote the transformation of scientific and technological achievements, a complete data management system needs to be built. Figure 1As shown, it includes a data acquisition module, a data analysis module, a data storage module and a data display module.

[0033] To ensure the integrity of the data, the data collection part first adopts a variety of data validation rules and logical checks to verify the format and range of the input data to ensure that it meets the predetermined specifications; secondly, a duplicate data detection mechanism is set up to promptly discover and remove duplicate data records during the collection process to avoid data redundancy; then real-time data interaction verification is carried out with the data source, and the accuracy of the data is immediately confirmed when the data is acquired. If there is any abnormality, timely feedback is given and the data is re-collected; finally, a data quality assessment indicator system is established, and the quality of the collected data is regularly assessed. Problems are promptly corrected and supplemented when they are discovered. When collecting funding data for scientific research projects, verification rules for numerical ranges are set to avoid unreasonable funding values; for duplicate funding records, only the latest and accurate one is retained.

[0034] The data analysis module uses the Apriori algorithm to reduce the generation of candidate item sets and use association rules to mine and discover the potential associations between different scientific and technological achievements. First, the minimum support threshold is set according to the application scenario; then the transaction database is scanned, the support of each single item is counted, and the frequent 1-item set L is generated. 1 ; Then use hashing technology to reduce the number of candidate item sets, hash all possible k-item sets into different buckets, count the frequency of item sets in each bucket, and only retain buckets and their corresponding item sets with frequencies exceeding the minimum support; Next, reduce the size of the transaction database to avoid repeated scanning of a large number of transactions in subsequent steps, delete transactions that do not contain frequent item sets after each iteration, because these transactions cannot generate new frequent item sets, merge transactions with the same frequent item sets to reduce redundancy; Then prune, for each candidate k-item set, check whether all its k-1 subsets are frequent item sets. If any k-1 subset of a candidate item set is not a frequent item set, then the candidate item set is not frequent and is deleted directly; For each remaining candidate k-item set, scan the transaction database and count its support Generate frequent k-itemset L k Finally, repeat the steps of reducing the size of the transaction database, pruning, and generating frequent item sets until no new frequent item sets can be generated, generate association rules from the frequent item sets, and calculate the confidence of each rule Filter out rules that meet a minimum confidence threshold.

[0035] The data storage module adopts a distributed architecture, which can efficiently store large-scale data, support fast retrieval and access, ensure that data is available at any time, and meet high concurrency requirements. First, the storage system is divided into multiple nodes, each node independently stores a part of the data; then the large-scale data is divided into multiple small blocks, each shard is stored on a different node, and the sharding formula of the data item key is shard_id = hash (key) mod N, where N is the total number of storage nodes and key is the key value of the data item; in order to improve the reliability and availability of data, the data of each shard will be copied to multiple nodes, which ensures that even if a node fails, the data can still be obtained from other nodes; use gzip data compression technology to reduce storage space occupancy and improve transmission efficiency; design efficient retrieval and access mechanisms Then design an efficient retrieval and access mechanism. When reading data, locate the node where the data is located according to the sharding strategy, and then read the data from the node. If the node is unavailable, read from the replica node. When writing data, write the data to the corresponding node according to the sharding strategy and synchronize it to the replica node. For the data item key, the node positioning formula for writing and reading is primary_node(key)=hash(key)modN. First, perform a hash operation on the key to obtain a hash value. Then, use the hash value to perform a modulo operation on the total number of nodes N to ensure that the result is between 0 and N-1, and locate a specific node. This node is the primary node that stores the data item key. replica_node(key)={(hash(key)+i)modNi=1,...,R-1}, where i is the offset of the replica node and R is the number of replicas of each data item. For each replica, use the hash value plus the offset i to perform a modulo operation on the total number of nodes N. By changing the value of i from 1 to R-1, different nodes can be located to obtain a set of multiple replica nodes. These replica nodes store copies of the same data item keys as the primary node. Finally, set up regular automatic backups to ensure that data can be quickly restored in the event of a failure. The backup data is stored in independent backup nodes or cloud storage.

[0036] The data display module has interactive functions, and users can dynamically adjust and view data details through operations such as clicking, zooming, and dragging; it supports multi-dimensional data display, and users can view data analysis results in different dimensions by filtering and switching dimensions; the report generation function allows users to generate reports in PDF, Excel and other formats based on the analysis results, which facilitates data sharing and archiving.

[0037] The above description is only a preferred embodiment of the present invention and does not limit the present invention in other forms. Any technician familiar with the profession may use the technical content disclosed above to change or modify it into an equivalent embodiment with equivalent changes and apply it to other fields. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A data management system for the transformation of scientific and technological achievements, characterized in that: It includes a data collection module, a data analysis module, a data storage module and a data display module. The data collection module is responsible for automatically collecting data related to scientific and technological achievements from multiple sources to ensure that the data is accurate and complete and to remove duplicate and invalid information. The data analysis module uses the Apriori algorithm to optimize the generation of reduced candidate item sets to discover potential associations between achievements. The data storage module uses a distributed architecture to efficiently store large-scale data, supports fast retrieval and access, ensures that data is available at any time, and meets high concurrency requirements. The data display module presents the analysis results with intuitive and clear charts, reports and graphics, providing users with a comprehensive and accurate data view to facilitate users to quickly understand. In the data analysis module, the Apriori algorithm is used to optimize the generation of candidate item sets and discover the potential associations between different scientific and technological achievements through association rule mining. The following are the implementation steps: Step 1: First, set the minimum support threshold according to the application scenario; Step 2: Then scan the transaction database, count the support of each single item, and generate the frequent 1-item set L1; Step 3: Then use hashing technology to reduce the number of candidate item sets, hash all possible k item sets into different buckets, count the frequency of item sets in each bucket, and only retain the buckets and their corresponding item sets whose frequency exceeds the minimum support; Step 4: Next, reduce the size of the transaction database to avoid repeatedly scanning a large number of transactions in subsequent steps. Delete transactions that do not contain frequent itemsets after each iteration because these transactions cannot generate new frequent itemsets. Merge transactions with the same frequent itemsets to reduce redundancy. Step 5: Then prune. For each candidate k-item set, check whether all its k-1 subsets are frequent item sets. If any k-1 subset of a candidate item set is not a frequent item set, then the candidate item set is not frequent and is deleted directly. Step 6: For each remaining candidate k-item set, scan the transaction database, count its support, and generate the frequent k-item set L k ; Step 7. Finally, repeat steps 4 to 6 until no new frequent item sets can be generated. Generate association rules from the frequent item sets, calculate the confidence of each rule, and filter out the rules that meet the minimum confidence threshold.

2. A data management system for transformation of scientific and technological achievements according to claim 1, characterized in that: The data collection part adopts the following practices to ensure the integrity of the data: first, a variety of data verification rules and logical checks are used to verify the input data format and range to ensure that they meet the predetermined specifications; secondly, a duplicate data detection mechanism is set up to promptly discover and remove duplicate data records during the collection process to avoid data redundancy; then real-time data interaction verification is carried out with the data source, and the accuracy of the data is immediately confirmed when the data is acquired. If there are any abnormalities, timely feedback and re-collection are carried out; finally, a data quality assessment indicator system is established, and the quality of the collected data is regularly assessed. Problems are promptly corrected and supplemented when they are discovered. When collecting funding data for scientific research projects, verification rules for numerical ranges are set to avoid unreasonable funding values; for duplicate funding records, only the latest and accurate one is retained.

3. A data management system for transformation of scientific and technological achievements according to claim 1, characterized in that: The data storage module adopts a distributed architecture, and the implementation steps are as follows: Step 1: First, divide the storage system into multiple nodes, each node independently stores a part of the data; then divide the large-scale data into multiple small blocks, each shard is stored on a different node, and the sharding formula of the data item key is shard_id = hash (key) mod N, where N is the total number of storage nodes and key is the key value of the data item; Step 2: To improve data reliability and availability, the data of each shard will be replicated to multiple nodes, which ensures that even if a node fails, the data can still be obtained from other nodes; Step 3: Use gzip data compression technology to reduce storage space usage and improve transmission efficiency; design efficient retrieval and access mechanisms; Step 4: Then design an efficient retrieval and access mechanism. When reading data, locate the node where the data is located according to the sharding strategy, and then read the data from the node. If the node is unavailable, read from the replica node. When writing data, write the data to the corresponding node according to the sharding strategy and synchronize it to the replica node. Step 5. Finally, set up regular automatic backup to ensure that data can be quickly restored in the event of a failure. The backup data is stored in an independent backup node or cloud storage.

4. A data management system for transformation of scientific and technological achievements according to claim 1, characterized in that: The data display module has interactive functions, and users can dynamically adjust and view data details through operations such as clicking, zooming, and dragging.

5. A data management system for transformation of scientific and technological achievements according to claim 1, characterized in that: The data display module supports multi-dimensional data display, and users can view data analysis results in different dimensions by filtering and switching dimensions.

6. A data management system for transformation of scientific and technological achievements according to claim 1, characterized in that: The data display module has a report generation function, and users can generate reports in PDF, Excel and other formats based on the analysis results to facilitate data sharing and archiving.

Citation Information

Patent Citations

  • Digital base operation management system

    CN117493412A