Data management system based on structural information

Abstract

The invention provides a data management system based on structural information, and belongs to the technical field of data management, and the system comprises a first determination module which obtains data element distribution of various types of data, and determines data features of the various types of data; the extraction module is used for constructing a data model of the data of each type based on the data features of the data of each type, and extracting classified data of each type from mass data; the second determination module is used for extracting basic metadata and scene service attributes of the various types of classified data and determining a data association structure between the basic metadata and the scene service attributes of the various types of classified data; and the storage management module is used for carrying out storage management on various types of classified data based on the data association structure. The problems that the flexibility of data is limited, corresponding types of data cannot be extracted in real time according to data characteristics, and meanwhile, the data management query efficiency is low and the reliability of a system is reduced due to the fact that traditional data management is common management of a heap of data and cannot determine association between the data are solved.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and in particular to a data management system based on structured information. Background Technology

[0002] With the explosive growth of data volume and the increasing complexity of business operations, the need for data management is becoming more and more apparent.

[0003] However, traditional data management systems limit data flexibility and cannot accurately extract the corresponding data type in real time based on data characteristics. At the same time, traditional data management involves managing a bunch of data together, making it impossible to determine the relationship between data, resulting in low efficiency in data management, retrieval and querying, and reducing the reliability of the system.

[0004] Therefore, this invention proposes a data management system based on structural information. Summary of the Invention

[0005] This invention provides a data management system based on structured information to address the limitations of existing technologies that restrict data flexibility, fail to accurately extract corresponding data types in real time based on data characteristics, and, in addition, traditional data management involves managing a large amount of data together without determining the relationships between data, resulting in low efficiency in data management, retrieval, and querying, and reduced system reliability.

[0006] On one hand, the present invention provides a data management system based on structured information, comprising:

[0007] The first determination module: obtains the data element distribution of each type of data, and determines the data characteristics of each type of data based on the data element distribution;

[0008] Extraction module: Constructs a data model for each type of data based on its data characteristics, and extracts categorized data of each type from massive datasets based on the data model;

[0009] The second determination module extracts the basic metadata and scenario business attributes of each type of classified data, and determines the data association structure between the basic metadata and scenario business data of each type of classified data based on the basic metadata and scenario business attributes.

[0010] Storage Management Module: Manages the storage of various types of categorized data based on data association structures.

[0011] According to the present invention, a data management system based on structured information includes a first determining module, comprising:

[0012] Preprocessing unit: Acquires multiple data from different data sources using web crawling technology, and preprocesses the multiple data;

[0013] Classification unit: Obtains the data structure of the preprocessed data and classifies the data by type according to the data structure;

[0014] Generation Unit: Determines the frequency of occurrence of each type of data using frequency analysis tools, generates a data element frequency table and graph based on the frequency of occurrence, and determines the distribution of each type of data based on the table and graph;

[0015] First determining unit: Determines the symmetric relationship between different types of data, and determines the pattern existing between the data based on the symmetric relationship;

[0016] First acquisition unit: Acquires the data element distribution of each type of data based on the distribution of each type of data and the patterns existing between the data;

[0017] The second determining unit: determines the data characteristics of each type of data based on the distribution of the data elements and a clustering algorithm.

[0018] According to the present invention, a data management system based on structured information includes an extraction module comprising:

[0019] Identification unit: Identifies data based on the data characteristics of each type of data, and determines the entity class and attributes of the data based on the identification results;

[0020] The third determining unit: defines the data type according to the entity class and its attributes of the data, and determines the uniqueness constraints and default values ​​according to the data type;

[0021] Setting unit: Defines the relationships between entities based on the application scenario and the defined relation schema, and sets constraints based on the relationships between the entities;

[0022] The first extraction unit constructs data models for various data types based on uniqueness constraints, default values, and constraints, and extracts various types of categorized data from massive amounts of data based on the data models.

[0023] According to the data management system based on structure information provided by the present invention, the setting unit further includes:

[0024] Analysis sub-unit: Analyze the application scenario to clarify the key information of the subject area and functional requirements involved in the entity;

[0025] Determine sub-units: Determine the responsibilities of each entity class based on the key information, and determine the sub-types of each entity based on the responsibilities;

[0026] Define sub-units: Define the relationships between entities based on the subtype of each entity and the defined relation schema.

[0027] According to a data management system based on structural information provided by the present invention, the second determining module includes:

[0028] The second acquisition unit analyzes the data of each type and obtains the structure and content of the data of each type based on the analysis results.

[0029] The third acquisition unit: determines the classification system of each type of classified data based on the structure and content, and obtains the classification level based on the classification system;

[0030] Traversal Unit: Based on the classification hierarchy and combined with the programming language, traverse each data item to obtain the common and dissimilar features of each type of classification data;

[0031] The second extraction unit extracts basic metadata and scenario business attributes for each type of classified data based on the common and differential features.

[0032] According to the data management system based on structural information provided by the present invention, the second determining module further includes:

[0033] The fourth determining unit: Determines the meaning of basic metadata and scenario business attributes based on data mining and natural language processing techniques;

[0034] Fifth determining unit: Based on the stated meaning and combined with the semantic understanding model, determine the relationship between basic metadata and scenario business attributes;

[0035] The sixth determining unit: Based on the relationship, the basic metadata is used as the backbone of the dataset, and the scene business attributes are used as the data subsets of the subordinate relationship. The data association structure between the basic metadata and the scene business data is determined by combining the backbone of the dataset and the data subsets of the subordinate relationship with the association rules.

[0036] According to the present invention, a data management system based on structured information includes a storage management module, comprising:

[0037] The fourth acquisition unit: Based on the data association structure, obtain the association types between various categories of data;

[0038] Seventh determining unit: Determine the relational and non-relational types of each type of categorized data based on the association type;

[0039] Storage Management Unit: Establishes a database schema based on the relational and non-relational combined association mapping technology, and performs storage management on various types of classified data based on the database schema and storage structure.

[0040] According to the data management system based on structured information provided by the present invention, after acquiring the data element distribution of various types of data, it further includes:

[0041] First acquisition module: Determine the real data elements and virtual data elements based on the distribution of data elements, and multiply the real data elements and virtual data elements to obtain the real-virtual product;

[0042] Building module: Determine the matrix row and column indices based on the real and imaginary product, and construct data unit matrices for each type of data based on the matrix row and column indices, as well as the real and imaginary data elements;

[0043] The third determination module: Determine the data volume range of each type of data based on the data unit matrix, and determine the effective data ratio of each type of data based on the data volume range;

[0044] First generation module: Determines the effective response ratio of each type of data element based on the effective data ratio, and generates a scatter plot of the effective response ratio of each type of data element based on the effective response ratio.

[0045] The second acquisition module determines the element response probability data structure for each type of data based on the scatter plot of the effective response ratio of the elements, and obtains the current iteration value of each data entry based on the element response probability data structure.

[0046] The third acquisition module: determines the reference iteration value of the element response entry for each type of data based on the current iteration value, and obtains the data field structure of the reference data of the reference iteration value;

[0047] The fourth module determines the data catalog for each data type by extracting common and unique domain data features based on the data domain structure.

[0048] The second generation module: obtains the mapping data description of the data directory for each type of data, and generates data integrity evaluation rules based on the mapping data description;

[0049] Evaluation module: It evaluates the completeness of various types of data through data integrity evaluation rules, filters out incomplete data based on the evaluation results, and issues reminders for data non-compliance and replacement.

[0050] Compared with the prior art, the beneficial effects of this application are as follows:

[0051] By extracting categorized data from data models of various data types, determining the data association structure between the basic metadata and scenario business attributes of each categorized data type, and storing the data, we can improve data flexibility. We can extract the corresponding type of data in real time and accurately based on data characteristics. At the same time, we can accurately determine the relationship between data, improve the efficiency of data management, retrieval and query, and enhance the reliability of the system. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of the structure of the data management system based on structural information provided in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of the first determining module provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] Example 1:

[0057] This invention provides a data management system based on structural information, such as... Figure 1 As shown, the system mainly includes the following modules:

[0058] The first determination module: obtains the distribution of each type of data, and determines the data characteristics of each type of data based on the distribution of data elements;

[0059] Extraction module: Constructs a data model for each type of data based on its data characteristics, and extracts categorized data of each type from massive datasets based on the data model;

[0060] The second determination module extracts the basic metadata and scenario business attributes of each type of classified data, and determines the data association structure between the basic metadata and scenario business data of each type of classified data based on the basic metadata and scenario business attributes.

[0061] Storage Management Module: Manages the storage of various types of categorized data based on data association structures.

[0062] In this embodiment, the data elements include: tangible data elements and intangible data elements.

[0063] In this embodiment, a data model refers to a conceptual model used to describe and define data, which describes the basic structure and relationships of the data.

[0064] In this embodiment, the data association structure refers to a way of establishing connections between different data elements. For example, each data object is represented as a node, and the connecting lines between nodes represent the relationship between them. The graph can be sparse or dense, and each vertex of the graph has a unique identifier and can be distinguished according to its name or other attributes.

[0065] The beneficial effects of the above technical solution are as follows: by extracting various types of classified data through data models of various types of data, determining the data association structure between the basic metadata and scenario business attributes of each type of classified data, and storing the data, the flexibility of the data can be improved. The corresponding type of data can be extracted in real time and accurately according to the data characteristics. At the same time, the association between data can be accurately determined, improving the efficiency of data management, retrieval and query, and enhancing the reliability of the system.

[0066] Example 2:

[0067] Based on Embodiment 1, the first determining module of this embodiment of the invention, such as... Figure 2 As shown, it includes:

[0068] Preprocessing unit: Acquires multiple data from different data sources using web crawling technology, and preprocesses the multiple data;

[0069] Classification unit: Obtains the data structure of the preprocessed data and classifies the data by type according to the data structure;

[0070] Generation Unit: Determines the frequency of occurrence of each type of data using frequency analysis tools, generates a data element frequency table and graph based on the frequency of occurrence, and determines the distribution of each type of data based on the table and graph;

[0071] First determining unit: Determines the symmetric relationship between different types of data, and determines the pattern existing between the data based on the symmetric relationship;

[0072] First acquisition unit: Acquires the data element distribution of each type of data based on the distribution of each type of data and the patterns existing between the data;

[0073] The second determining unit: determines the data characteristics of each type of data based on the distribution of the data elements and a clustering algorithm.

[0074] In this embodiment, web crawling technology is an automated program or script that obtains web page information by simulating the behavior of a user browsing web pages.

[0075] In this embodiment, the frequency analysis tool is used to perform frequency analysis on a set of data. Frequency analysis is a method of descriptive statistics that aims to determine the frequency or number of times each value appears in the dataset.

[0076] In this embodiment, the data element frequency table is a table used to record the frequency of different elements in the dataset. In the data element frequency table, each row represents a data element, and each cell in the table represents the frequency of the element in the dataset.

[0077] In this embodiment, the symmetrical relationship between different types of data can be:

[0078] The symmetry between time series data and symmetry-axis time series data: Time series data is the sequential arrangement of events in a dataset, while symmetry-axis time series data is a new dataset obtained by flipping time series data along a certain time axis. The relationship between the two is symmetry, that is, if the original time series data is flipped along the symmetry axis, the new dataset obtained has the exact same event occurrence order and time order as the original dataset.

[0079] The symmetric relationship between geospatial data and inverse geospatial data: Geospatial data is a description of the geographic spatial location of a dataset, such as latitude, longitude, and elevation, while inverse geospatial data is obtained by inverting geospatial data, that is, inverting information such as latitude, longitude, and elevation back to the corresponding geographic entities.

[0080] In this embodiment, the clustering algorithm can be: K-means clustering algorithm, spectral clustering algorithm, or other density-based clustering algorithm.

[0081] The beneficial effects of the above technical solution are: to obtain the data element distribution of various types of data, to determine the data characteristics of each type of data based on the data element distribution and the clustering algorithm, and to determine the data characteristics of each type of data more quickly by using the clustering algorithm, thereby improving the analysis efficiency, while ensuring the accuracy and usability of the obtained data characteristics.

[0082] Example 3:

[0083] Based on Example 2, the extraction module of this embodiment includes:

[0084] Identification unit: Identifies data based on the data characteristics of each type of data, and determines the entity class and attributes of the data based on the identification results;

[0085] The third determining unit: defines the data type according to the entity class and its attributes of the data, and determines the uniqueness constraints and default values ​​according to the data type;

[0086] Setting unit: Defines the relationships between entities based on the application scenario and the defined relation schema, and sets constraints based on the relationships between the entities;

[0087] The first extraction unit constructs data models for various data types based on uniqueness constraints, default values, and constraints, and extracts various types of categorized data from massive amounts of data based on the data models.

[0088] In this embodiment, an entity class refers to a "thing" that has practical significance in the application scenario, while attributes are descriptions of entity classes, such as name, description, and value. For example, in a student management system, entity classes can be "student", "course" and "grade", and corresponding attributes can be "name", "gender", "major" and "credits".

[0089] In this embodiment, data types can be divided into two main categories: value types and reference types. Value types are mainly used to represent information such as numbers, strings, dates, and times.

[0090] In this embodiment, determining uniqueness constraints based on data type means defining the rules or conditions for the uniqueness of a variable based on its data type. For example, if a user's ID is an integer field, then all user IDs must be unique.

[0091] In this embodiment, constraints such as primary keys, foreign keys, and cascading can be set in the relation schema.

[0092] In this embodiment, a data model refers to a conceptual model used to describe and define data, which describes the basic structure and relationships of the data.

[0093] The beneficial effects of the above technical solution are: it can construct a data model for each type of data based on the data characteristics of each type of data, and extract the classification data of each type from the massive data based on the data model, which can ensure the accurate classification of the data types and greatly improve the efficiency of data mining.

[0094] Example 4:

[0095] Based on Embodiment 3, the setting unit of this embodiment further includes:

[0096] Analysis sub-unit: Analyze the application scenario to clarify the key information of the subject area and functional requirements involved in the entity;

[0097] Determine sub-units: Determine the responsibilities of each entity class based on the key information, and determine the sub-types of each entity based on the responsibilities;

[0098] Define sub-units: Define the relationships between entities based on the subtype of each entity and the defined relation schema.

[0099] In this embodiment, the responsibility of each entity class refers to defining the responsibilities that each entity class should bear in object-oriented programming. An entity class is an instantiation of a class that represents a specific entity and is responsible for managing the entity's state and behavior. The responsibility of the entity class is to determine which attributes and methods the entity should have, and how to use these attributes and methods to represent and manipulate the entity.

[0100] In this embodiment, a subtype of an entity is a specific form of the entity that has the same base type attributes but may have some additional or different attributes or methods. A subtype is an extension or variant of the base type and can be created through inheritance. For example, if there is an animal entity class, the subtypes may be: cat, dog, bird, or they may be: fur color, breed, health status.

[0101] In this embodiment, a relation schema is a normalized way of describing the table structure and data in a relational database.

[0102] The beneficial effects of the above technical solution are: by defining the relationships between entities based on the application scenario and the defined relational schema, the logical connections between different entities can be seen, thereby improving the availability and reusability of the system program. At the same time, it is possible to better understand the meaning between each entity.

[0103] Example 5:

[0104] Based on Embodiment 4, the second determining module of this embodiment includes:

[0105] The second acquisition unit analyzes the data of each type and obtains the structure and content of the data of each type based on the analysis results.

[0106] The third acquisition unit: determines the classification system of each type of classified data based on the structure and content, and obtains the classification level based on the classification system;

[0107] Traversal Unit: Based on the classification hierarchy and combined with the programming language, traverse each data item to obtain the common and dissimilar features of each type of classification data;

[0108] The second extraction unit extracts basic metadata and scenario business attributes for each type of classified data based on the common and differential features.

[0109] In this embodiment, the classification system can be: a tree structure or a list.

[0110] In this embodiment, classification hierarchy is a method of organizing concepts or objects according to their attributes and relationships, and classifying and organizing concepts or objects through a hierarchical classification structure.

[0111] In this embodiment, data structure refers to the way data is organized and stored in a computer program, such as linear lists, tree structures, graph structures, and stacks.

[0112] In this embodiment, common features may include: category name, parent node, and hierarchical relationship.

[0113] In this embodiment, the differentiating features may be: classification description, keywords, or special requirements.

[0114] The beneficial effects of the above technical solution are: it extracts the basic metadata and scenario business attributes of each type of classified data based on the common and different characteristics of each type of classified data, and can be customized according to data type and business needs, making data management and analysis simpler and more efficient, reducing the time and cost of data processing and analysis, while ensuring the consistency and accuracy of basic metadata and scenario business attributes.

[0115] Example 6:

[0116] Based on Embodiment 5, the second determining module of this embodiment further includes:

[0117] The fourth determining unit: Determines the meaning of basic metadata and scenario business attributes based on data mining and natural language processing techniques;

[0118] Fifth determining unit: Based on the stated meaning and combined with the semantic understanding model, determine the relationship between basic metadata and scenario business attributes;

[0119] The sixth determining unit: Based on the relationship, the basic metadata is used as the backbone of the dataset, and the scene business attributes are used as the data subsets of the subordinate relationship. The data association structure between the basic metadata and the scene business data is determined by combining the backbone of the dataset and the data subsets of the subordinate relationship with the association rules.

[0120] In this embodiment, data mining is a process that uses computer algorithms and techniques to analyze large amounts of data and extract useful information and knowledge from them.

[0121] In this embodiment, natural language processing enables computers to understand and process human natural language, including but not limited to: word segmentation, part-of-speech tagging, semantic analysis, sentiment analysis, and named entity recognition.

[0122] In this embodiment, the meanings of basic metadata and scenario business attributes are, for example, that for customer behavior data, basic metadata may include information such as timestamps, user IDs, and behavior categories, while scenario business attributes may include purchase intent, purchase channels, and consumption habits.

[0123] In this embodiment, the relationship between basic metadata and scenario business attributes can be as follows: purchase intent may be the root cause of purchase behavior, while purchase channel is the actual environment in which purchase behavior occurs.

[0124] The beneficial effects of the above technical solution are: the data association structure between the basic metadata and scenario business data of each type of classified data is determined according to the basic metadata and scenario business attributes. By clearly defining the association rules between the basic metadata and scenario business data of each type of classified data, it can be ensured that each data element has a unique identifier and the correct type, avoiding unnecessary data loss or duplication. At the same time, data sharing and collaboration can be easily realized, improving the reliability of the system.

[0125] Example 7:

[0126] Based on Embodiment 6, the storage management module of this embodiment includes:

[0127] The fourth acquisition unit: Based on the data association structure, obtain the association types between various categories of data;

[0128] Seventh determining unit: Determine the relational and non-relational types of each type of categorized data based on the association type;

[0129] Storage Management Unit: Establishes a database schema based on the relational and non-relational combined association mapping technology, and performs storage management on various types of classified data based on the database schema and storage structure.

[0130] In this embodiment, the data association structure refers to a way of establishing connections between different data elements. For example, each data object is represented as a node, and the connecting lines between nodes represent the relationship between them. The graph can be sparse or dense, and each vertex of the graph has a unique identifier and can be distinguished according to its name or other attributes.

[0131] In this embodiment, the types of associations between various categories of data can be: semantic association, temporal association, spatial association, and contextual association.

[0132] In this embodiment, the relational type between different types of categorized data can be:

[0133] Competition relationship: refers to a mutually exclusive relationship between a classification data item and other classification data items. Only one of them can be taken as the final result. For example, in a binary classification problem, one class is a positive sample and the other class is a negative sample.

[0134] Collaborative relationship: refers to a mutually beneficial relationship between a categorical data item and other categorical data items. They can achieve better classification results through joint decision-making. For example, in multi-class classification problems, each category can be jointly determined by multiple categorical data items.

[0135] Selection relationship: refers to the selective relationship between a categorical data item and other categorical data items.

[0136] In this embodiment, the non-relational type of each type of categorized data is:

[0137] Sequential relationship: This refers to the requirement that a categorical data item must appear before or after other categorical data items in order to form a correct classification result.

[0138] Contextual relationship: refers to the fact that a categorical data item needs to take into account its contextual information in the text in order to obtain the correct classification result.

[0139] Preference relationship: This refers to the fact that one category of data item has priority, while other category data items need to be adjusted according to its value.

[0140] In this embodiment, association mapping is a data modeling technique used to describe the relationships between multiple data. For example, in relational databases, the third normal form theory can be used to establish a database schema to ensure data consistency and maintainability.

[0141] In this embodiment, the database schema is a description of the structure and relationships of all data in the database, used to describe the various tables, fields, and relationships between them.

[0142] In this embodiment, storage structure refers to the way data elements in a computer system are arranged and organized in memory or external storage, describing how data is stored in memory, such as: linked storage, indexed storage, and balanced search tree.

[0143] The beneficial effects of the above technical solution are: determining the relational and non-relational types of data based on the association types between various types of data, thereby establishing a database schema for storing and managing the data of each type, can avoid storing too much redundant data in a single database, making data management more convenient. At the same time, data can be easily distributed across different databases, and databases can be added or removed at any time to adapt to needs, improving the scalability of the system.

[0144] Example 8:

[0145] Based on Example 7, after obtaining the data element distribution of each type of data, this embodiment of the invention further includes:

[0146] First acquisition module: Determine the real data elements and virtual data elements based on the distribution of data elements, and multiply the real data elements and virtual data elements to obtain the real-virtual product;

[0147] Building module: Determine the matrix row and column indices based on the real and imaginary product, and construct data unit matrices for each type of data based on the matrix row and column indices, as well as the real and imaginary data elements;

[0148] The third determination module: Determine the data volume range of each type of data based on the data unit matrix, and determine the effective data ratio of each type of data based on the data volume range;

[0149] First generation module: Determines the effective response ratio of each type of data element based on the effective data ratio, and generates a scatter plot of the effective response ratio of each type of data element based on the effective response ratio.

[0150] The second acquisition module determines the element response probability data structure for each type of data based on the scatter plot of the effective response ratio of the elements, and obtains the current iteration value of each data entry based on the element response probability data structure.

[0151] The third acquisition module: determines the reference iteration value of the element response entry for each type of data based on the current iteration value, and obtains the data field structure of the reference data of the reference iteration value;

[0152] The fourth module determines the data catalog for each data type by extracting common and unique domain data features based on the data domain structure.

[0153] The second generation module: obtains the mapping data description of the data directory for each type of data, and generates data integrity evaluation rules based on the mapping data description;

[0154] Evaluation module: It evaluates the completeness of various types of data through data integrity evaluation rules, filters out incomplete main clauses based on the evaluation results, and issues reminders for data non-compliance and replacement.

[0155] In this embodiment, a real data element refers to an actual data object stored in the database. It has independence and integrity. A real data element can contain multiple attributes, each of which has its corresponding data type, length and constraints. For example, in a student information table, each student is a real data element, which contains multiple attributes such as the student's name, student ID, gender, and age.

[0156] In this embodiment, a virtual data element refers to a collection of related data in a database. These data have some kind of relationship or business rule. A virtual data element is usually composed of multiple real data elements, and these real data elements have a certain correlation. For example, in a student information table, the virtual data element "all students" contains the real data elements of all students. These real data elements have a one-to-many relationship.

[0157] In this embodiment, multiplying real data elements and virtual data elements to obtain a real-virtual product refers to a method of combining real data elements and virtual data elements for processing in a specific context. For example, in a specific application or system design, real data elements and virtual data elements may be considered to have a certain degree of correlation. In this case, multiplying real data elements and virtual data elements is an attempt to simulate the logical relationship between entities or to calculate the correlation value under certain business rules.

[0158] In this embodiment, the data unit matrix for each type of data is a table used to describe the size and order of different data types.

[0159] In this embodiment, the data volume range for each type of data refers to the range of space required for different data types.

[0160] In this embodiment, the effective response ratio of each type of data element refers to the proportion of effective data successfully acquired and processed during the data processing process.

[0161] In this embodiment, the element effective response ratio scatter plot is a chart used to show the effective response relationship between different elements. The effective response of each element is represented as a scatter point, and the relationship between the elements can be understood by observing the distribution of the scatter points.

[0162] In this embodiment, the element response probability data structure for each type of data refers to a specific data structure used to store and manage various types of data and their response probabilities, such as: hash table, binary indexed tree, graph data structure.

[0163] In this embodiment, the reference iteration value of the element response entry for each type of data is an indicator used to describe the data response speed, which refers to the time interval from when the user submits a query to when the data is returned.

[0164] In this embodiment, the mapping data description of the data directory for each type of data means that for each type of data, it is necessary to define its data directory to describe the structure and attributes of the data type, and at the same time, it is also necessary to define the mapping rules of the data type to determine how to map the data access request submitted by the user to the corresponding data element in the database.

[0165] The beneficial effects of the above technical solution are: by using data integrity assessment rules to assess the integrity of various types of data, incomplete main clauses are selected based on the assessment results and reminders for data non-compliance and replacement are issued, which can ensure the reliability and validity of the data. At the same time, it can more accurately obtain the characteristics of the data and improve the efficiency of data management.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0167] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data management system based on structured information, characterized in that, include: The first determination module: obtains the data element distribution of each type of data, and determines the data characteristics of each type of data based on the data element distribution; Extraction module: Constructs a data model for each type of data based on its data characteristics, and extracts categorized data of each type from massive datasets based on the data model; The second determination module extracts the basic metadata and scenario business attributes of each type of classified data, and determines the data association structure between the basic metadata and scenario business data of each type of classified data based on the basic metadata and scenario business attributes. Storage Management Module: Manages the storage of various types of categorized data based on data association structures.

2. The data management system based on structured information according to claim 1, characterized in that, The first determining module includes: Preprocessing unit: Acquires multiple data from different data sources using web crawling technology, and preprocesses the multiple data; Classification unit: Obtains the data structure of the preprocessed data and classifies the data by type according to the data structure; Generation Unit: Determines the frequency of occurrence of each type of data using frequency analysis tools, generates a data element frequency table and graph based on the frequency of occurrence, and determines the distribution of each type of data based on the table and graph; First determining unit: Determines the symmetric relationship between different types of data, and determines the pattern existing between the data based on the symmetric relationship; First acquisition unit: Acquires the data element distribution of each type of data based on the distribution of each type of data and the patterns existing between the data; The second determining unit: determines the data characteristics of each type of data based on the distribution of the data elements and a clustering algorithm.

3. The data management system based on structured information according to claim 1, characterized in that, The extraction module includes: Identification unit: Identifies data based on the data characteristics of each type of data, and determines the entity class and attributes of the data based on the identification results; The third determining unit: defines the data type according to the entity class and its attributes of the data, and determines the uniqueness constraints and default values ​​according to the data type; Setting unit: Defines the relationships between entities based on the application scenario and the defined relation schema, and sets constraints based on the relationships between the entities; The first extraction unit constructs data models for various data types based on uniqueness constraints, default values, and constraints, and extracts various types of categorized data from massive amounts of data based on the data models.

4. The data management system based on structured information according to claim 1, characterized in that, The setting unit also includes: Analysis sub-unit: Analyze the application scenario to clarify the key information of the subject area and functional requirements involved in the entity; Determine sub-units: Determine the responsibilities of each entity class based on the key information, and determine the sub-types of each entity based on the responsibilities; Define sub-units: Define the relationships between entities based on the subtype of each entity and the defined relation schema.

5. The data management system based on structured information according to claim 1, characterized in that, The second determining module includes: The second acquisition unit analyzes the data of each type and obtains the structure and content of the data of each type based on the analysis results. The third acquisition unit: determines the classification system of each type of classified data based on the structure and content, and obtains the classification level based on the classification system; Traversal Unit: Based on the classification hierarchy and combined with the programming language, traverse each data item to obtain the common and dissimilar features of each type of classification data; The second extraction unit extracts basic metadata and scenario business attributes for each type of classified data based on the common and differential features.

6. The data management system based on structured information according to claim 1, characterized in that, The second determining module also includes: The fourth determining unit: Determines the meaning of basic metadata and scenario business attributes based on data mining and natural language processing techniques; Fifth determining unit: Based on the stated meaning and combined with the semantic understanding model, determine the relationship between basic metadata and scenario business attributes; The sixth determining unit: Based on the relationship, the basic metadata is used as the backbone of the dataset, and the scene business attributes are used as the data subsets of the subordinate relationship. The data association structure between the basic metadata and the scene business data is determined by combining the backbone of the dataset and the data subsets of the subordinate relationship with the association rules.

7. The data management system based on structured information according to claim 1, characterized in that, The storage management module includes: The fourth acquisition unit: Based on the data association structure, obtain the association types between various categories of data; Seventh determining unit: Determine the relational and non-relational types of each type of categorized data based on the association type; Storage Management Unit: Establishes a database schema based on the relational and non-relational combined association mapping technology, and performs storage management on various types of classified data based on the database schema and storage structure.

8. The data management system based on structured information according to claim 1, characterized in that, After obtaining the data element distribution for each data type, the process also includes: First acquisition module: Determine the real data elements and virtual data elements based on the distribution of data elements, and multiply the real data elements and virtual data elements to obtain the real-virtual product; Building module: Determine the matrix row and column indices based on the real and imaginary product, and construct data unit matrices for each type of data based on the matrix row and column indices, as well as the real and imaginary data elements; The third determination module: Determine the data volume range of each type of data based on the data unit matrix, and determine the effective data ratio of each type of data based on the data volume range; First generation module: Determines the effective response ratio of each type of data element based on the effective data ratio, and generates a scatter plot of the effective response ratio of each type of data element based on the effective response ratio. The second acquisition module determines the element response probability data structure for each type of data based on the scatter plot of the effective response ratio of the elements, and obtains the current iteration value of each data entry based on the element response probability data structure. The third acquisition module: determines the reference iteration value of the element response entry for each type of data based on the current iteration value, and obtains the data field structure of the reference data of the reference iteration value; The fourth module determines the data catalog for each data type by extracting common and unique domain data features based on the data domain structure. The second generation module: obtains the mapping data description of the data directory for each type of data, and generates data integrity evaluation rules based on the mapping data description; Evaluation module: It evaluates the completeness of various types of data through data integrity evaluation rules, filters out incomplete data based on the evaluation results, and issues reminders for data non-compliance and replacement.

Images