Data component model construction based on graph decomposition, data analysis method and device

Abstract

The present application proposes a data component model construction, data analysis method and device based on graph decomposition, which relates to the field of data processing technology. The data component model construction method based on graph decomposition includes: A10, preprocessing the acquired authorized data to be governed, and the data to be governed is text data; A20, constructing a graph of the data to be governed based on the preprocessed data to be governed; A30, using a graph decomposition algorithm to decompose the graph of the data to be governed into multiple sub-graphs of the data to be governed; A40, extracting data elements about the data to be governed from the sub-graphs of the data to be governed. The present application can more effectively express and simulate the real situation of the data to be governed through graph decomposition. In the complex data processing process, graph decomposition can better reveal the internal structure and laws of the data to be governed, and provide strong support for subsequent data analysis and simulation. Moreover, the present application improves the accuracy and efficiency of the simulation of the data to be governed.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data element model construction, data analysis method and device based on graph decomposition. Background Art

[0002] Existing data processing methods mainly focus on data analysis and data mining. Although these methods can analyze the hidden information in the data to a certain extent, their efficiency and effectiveness need to be improved for more complex and large-scale data processing needs. In addition, existing data models are often unable to effectively express and simulate the real situation of data when dealing with complex problems in the real world or when sensitive information exists. Summary of the invention

[0003] The technical problem to be solved by the present application is how to improve the processing efficiency and effect of complex and large-scale data. The present application proposes a data element model construction, data analysis method and device based on graph decomposition.

[0004] According to an embodiment of the present application, a method for constructing a data component model based on graph decomposition includes:

[0005] A10, preprocessing the acquired authorized data to be managed, wherein the data to be managed is text data;

[0006] A20, constructing a map of the data to be governed based on the preprocessed data to be governed, specifically including:

[0007] A21, extracting data fields from the preprocessed data to be managed as vertices in the graph of the data to be managed;

[0008] A22, establishing link relationships between vertices in the to-be-governed data graph according to the association relationships in the data fields, and calculating link weights between vertices with link relationships;

[0009] A30, using a graph decomposition algorithm, decomposing the to-be-governed data graph into a plurality of to-be-governed data sub-graphs, specifically including:

[0010] A31, setting a corresponding threshold for each vertex in the to-be-governed data graph;

[0011] A32, select any vertex and traverse to obtain the link weights between other vertices and the selected vertex;

[0012] A33, if the link weight is greater than a corresponding threshold, clustering the corresponding vertex and the selected vertex into the same subgraph;

[0013] A34, repeating steps A32 to A33 to decompose the to-be-managed data graph into a plurality of to-be-managed data sub-graphs;

[0014] A40, extracting data elements related to the data to be managed from the sub-graph of the data to be managed.

[0015] According to the data component model construction method based on graph decomposition according to the embodiment of the present application, through graph decomposition, the real situation of the data to be managed can be more effectively expressed and simulated. In the complex data processing process, graph decomposition can better reveal the internal structure and rules of the data to be managed, and provide strong support for the subsequent analysis and simulation of the data to be managed.

[0016] According to some embodiments of the present application, in step A10, the acquired data to be managed is preprocessed, including:

[0017] The acquired data to be managed is cleaned and sorted to remove abnormal values ​​and missing values ​​in the data to be managed.

[0018] In some embodiments of the present application, in step A40, the data fields in each decomposed data sub-graph to be governed are subjected to information condensation and redundancy removal by the principal component method to obtain the data elements.

[0019] The data analysis method according to an embodiment of the present invention includes:

[0020] B10, constructing data elements for the data to be managed using the data element model construction method based on graph decomposition as described above;

[0021] B20, performing data simulation analysis on the data to be governed through the data element.

[0022] According to the data analysis method of the embodiment of the present application, the data elements of the data to be managed are used for simulation, which improves the accuracy and efficiency of the simulation of the data to be managed. The extraction and analysis of data elements can deeply understand the core characteristics of the data, so as to better simulate the future trend of the data to be managed. Compared with traditional data processing methods, this model can more accurately predict the future trend and possible results of the data to be managed.

[0023] According to some embodiments of the present application, in step B20, the future trend of the data to be governed is predicted through data simulation analysis.

[0024] A data component model construction device based on graph decomposition according to an embodiment of the present invention includes:

[0025] A preprocessing module, used for preprocessing the acquired authorized data to be managed, wherein the data to be managed is text data;

[0026] The graph construction module is used to construct a graph of the data to be governed based on the preprocessed data to be governed, specifically including:

[0027] Extracting data fields from the data to be managed as vertices in the graph of the data to be managed;

[0028] According to the association relationship in each data field, a link relationship between vertices in the to-be-governed data graph is established, and the link weights between vertices with link relationships are calculated;

[0029] The graph decomposition module is used to decompose the to-be-governed data graph into multiple to-be-governed data sub-graphs using a graph decomposition algorithm, specifically including:

[0030] For each vertex in the to-be-governed data graph, set a corresponding threshold;

[0031] Select any vertex and traverse to obtain the links between other vertices and the selected vertex;

[0032] If the link weight is greater than a corresponding threshold, the corresponding vertex and the selected vertex are clustered into the same subgraph;

[0033] Repeat traversing other vertices to decompose the data graph into multiple data sub-graphs to be managed;

[0034] A data element construction module is used to extract data elements about the data to be governed from the sub-graph.

[0035] According to the data component model construction device based on graph decomposition according to the embodiment of the present application, through graph decomposition, the real situation of the data to be managed can be more effectively expressed and simulated. In the complex data processing process, graph decomposition can better reveal the internal structure and rules of the data to be managed, and provide strong support for the subsequent analysis and simulation of the data to be managed.

[0036] According to some embodiments of the present application, the preprocessing module preprocesses the acquired data to be managed, including:

[0037] The acquired data to be managed is cleaned and sorted, and abnormal values ​​and missing values ​​in the acquired data to be managed are removed.

[0038] In some embodiments of the present application, the data element construction module condenses information and removes redundancy on the data fields in each decomposed data sub-graph to be governed by using the principal component method to obtain the data element.

[0039] A data analysis device according to an embodiment of the present invention includes:

[0040] A data element model construction device, using the data element model construction device based on graph decomposition as described above, for constructing data elements about the data to be managed;

[0041] The analysis module is used to perform data simulation analysis on the data to be governed through the data element.

[0042] According to the data analysis device of the embodiment of the present application, the data elements of the data to be managed are used for simulation, which improves the accuracy and efficiency of the simulation of the data to be managed. The extraction and analysis of data elements can deeply understand the core characteristics of the data, so as to better simulate the future trend of the data to be managed. Compared with traditional data processing methods, this model can more accurately predict the future trend and possible results of the data to be managed.

[0043] According to some embodiments of the present application, the analysis module predicts the future trend of the data to be governed through data simulation analysis.

[0044] According to an embodiment of the present application, a data element model construction device based on graph decomposition includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the data element model construction method based on graph decomposition as described above are implemented.

[0045] According to the storage medium of the embodiment of the present application, a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the data element model construction method based on graph decomposition as described above are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a flow chart of a method for constructing a data component model based on graph decomposition according to an embodiment of the present application;

[0047] Figure 2 A flow chart of a method for constructing a data graph according to an embodiment of the present application;

[0048] Figure 3 It is a flowchart of the spectrum decomposition according to an embodiment of the present application;

[0049] Figure 4 is a flow chart of a data analysis method according to an embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the composition of a data element model construction device based on graph decomposition according to an embodiment of the present application;

[0051] Figure 6 A schematic diagram of a data map according to an embodiment of the present application;

[0052] Figure 7 It is a schematic diagram of the spectrum decomposition according to an embodiment of the present application.

[0053] Reference numerals:

[0054] Data element model construction device 100,

[0055] Preprocessing module 10, graph construction module 20, graph decomposition module 30, data element construction module 40. DETAILED DESCRIPTION

[0056] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined purpose, the present application is described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0057] The description of the method flow in the specification of this application and the steps of the flowchart in the drawings of the specification of this application do not have to be strictly executed according to the step numbers, and the method steps can be executed in a different order. Moreover, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0058] The collection and acquisition of various data in this application complies with relevant laws and regulations and is authorized by the data provider. Any organization or individual that needs to obtain external data must obtain authorization in accordance with the law and ensure data security. It is not allowed to illegally collect, use, process, or transmit unauthorized or unprotected data, or to illegally buy, sell, provide, or disclose unauthorized or unprotected data.

[0059] like Figure 1 As shown, according to the data component model construction method based on graph decomposition according to an embodiment of the present application, the method includes:

[0060] A10, preprocessing the acquired authorized data to be managed, wherein the data to be managed is text data;

[0061] According to some embodiments of the present application, in step A10, the acquired data to be managed is preprocessed, including:

[0062] Clean and organize the acquired data to be managed, and remove abnormal values ​​and missing values ​​in the data to be managed, so as to ensure the quality and integrity of the data.

[0063] A20, based on the preprocessed data to be governed, constructs a data map to be governed, such as Figure 2 As shown, step A20 specifically includes:

[0064] A21, extracting data fields from the preprocessed data to be managed as vertices in the graph of the data to be managed;

[0065] A22, establishing link relationships between vertices in the to-be-governed data graph according to the association relationships in each data field, and calculating link weights between vertices with link relationships;

[0066] A30, using the graph decomposition algorithm, decomposes the to-be-governed data graph into multiple to-be-governed data sub-graphs, such as Figure 3 As shown, step A30 specifically includes:

[0067] A31, set corresponding thresholds for each vertex in the data graph to be governed;

[0068] A32, select any vertex and traverse to obtain the link weights between other vertices and the selected vertex;

[0069] A33, if the link weight is greater than the corresponding threshold, the corresponding vertex and the selected vertex are clustered into the same subgraph;

[0070] A34, repeating steps A32 to A33 to decompose the to-be-managed data graph into multiple to-be-managed data sub-graphs;

[0071] A40, extracting data elements about the data to be managed from the sub-graph of the data to be managed.

[0072] In some embodiments of the present application, in step A40, the data fields in each decomposed data sub-graph to be governed are subjected to information condensation and redundancy removal by the principal component method to obtain data elements.

[0073] According to the data component model construction method based on graph decomposition according to the embodiment of the present application, through graph decomposition, the real situation of the data to be managed can be more effectively expressed and simulated. In the complex data processing process, graph decomposition can better reveal the internal structure and rules of the data to be managed, and provide strong support for the subsequent analysis and simulation of the data to be managed.

[0074] like Figure 4 As shown, the data analysis method according to an embodiment of the present invention includes:

[0075] B10, constructing data elements about the data to be managed using the above data element model construction method based on graph decomposition;

[0076] B20, performs data simulation analysis on the governance data through data components.

[0077] According to some embodiments of the present application, in step B20, future trends of the data to be governed are predicted through data simulation analysis.

[0078] According to the data analysis method of the embodiment of the present application, the data elements of the data to be managed are used for simulation, which improves the accuracy and efficiency of the simulation of the data to be managed. The extraction and analysis of data elements can deeply understand the core characteristics of the data, so as to better simulate the future trend of the data to be managed. Compared with traditional data processing methods, this model can more accurately predict the future trend and possible results of the data to be managed.

[0079] like Figure 5 As shown, the data component model construction device 100 based on graph decomposition according to an embodiment of the present invention includes: a preprocessing module 10, a graph construction module 20, a graph decomposition module 30 and a data component construction module 40.

[0080] The preprocessing module 10 is used to preprocess the acquired authorized data to be managed, where the data to be managed is text data;

[0081] According to some embodiments of the present application, the preprocessing module 10 preprocesses the acquired data to be managed, including:

[0082] Clean and organize the acquired data to be managed, and remove outliers and missing values ​​in the acquired data to be managed.

[0083] The graph construction module 20 is used to construct a graph of the data to be governed based on the preprocessed data to be governed, which specifically includes:

[0084] Extract data fields from the data to be managed as vertices in the graph of the data to be managed;

[0085] According to the association relationship in each data field, establish the link relationship between the vertices in the data graph to be governed, and calculate the link weights between the vertices with link relationships;

[0086] The graph decomposition module 30 is used to decompose the to-be-governed data graph into multiple to-be-governed data sub-graphs using a graph decomposition algorithm, specifically including:

[0087] Set corresponding thresholds for each vertex in the data graph to be governed;

[0088] Select any vertex and traverse to obtain the links between other vertices and the selected vertex;

[0089] If the link weight is greater than the corresponding threshold, the corresponding vertex and the selected vertex are clustered into the same subgraph;

[0090] Repeat the traversal of other vertices to decompose the data graph into multiple sub-graphs of data to be managed;

[0091] The data element construction module 40 is used to extract data elements related to the data to be governed from the sub-graph.

[0092] In some embodiments of the present application, the data element construction module 40 performs information condensation and redundancy removal on the data fields in each decomposed data sub-graph to be governed by using the principal component method to obtain data elements.

[0093] According to the data component model construction device based on graph decomposition according to the embodiment of the present application, through graph decomposition, the real situation of the data to be managed can be more effectively expressed and simulated. In the complex data processing process, graph decomposition can better reveal the internal structure and rules of the data to be managed, and provide strong support for the subsequent analysis and simulation of the data to be managed.

[0094] The data analysis device according to the embodiment of the present invention includes: a data element model construction device 100 and an analysis module.

[0095] The data element model construction device 100 adopts the data element model construction device 100 based on graph decomposition as described above, which is used to construct data elements about the data to be governed;

[0096] The analysis module is used to perform data simulation analysis on the governance data through data elements.

[0097] According to some embodiments of the present application, the analysis module predicts the future trend of the data to be governed through data simulation analysis.

[0098] According to the data analysis device of the embodiment of the present application, the data elements of the data to be managed are used for simulation, which improves the accuracy and efficiency of the simulation of the data to be managed. The extraction and analysis of data elements can deeply understand the core characteristics of the data, so as to better simulate the future trend of the data to be managed. Compared with traditional data processing methods, this model can more accurately predict the future trend and possible results of the data to be managed.

[0099] According to an embodiment of the present application, a data element model construction device based on graph decomposition includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the data element model construction method based on graph decomposition are implemented as described above.

[0100] According to the storage medium of the embodiment of the present application, a computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the above-mentioned method for constructing a data element model based on graph decomposition are implemented.

[0101] The following describes in detail the data component model construction method based on graph decomposition of the present application with reference to the accompanying drawings in a specific embodiment. It should be understood that the following description is only an exemplary description and should not be construed as a specific limitation on the present application.

[0102] For example, the data component model construction method based on graph decomposition of the present application can be applied to a government affairs system, including the following steps:

[0103] (1) Data preprocessing: Clean and organize the authorized personal information data obtained, remove outliers and missing values, and ensure the quality and integrity of the data.

[0104] (2) Graph construction. According to the characteristics of the data, select an appropriate graph construction method and convert the data into a graph.

[0105] Each data field is a vertex of the graph. For example, the portrait information of an individual includes household registration, marriage, childbirth, adoption, professional qualifications, social security, income, age, investment and real estate, education and scientific research, public utilities, taxation, judicial credit, license information, health information, etc. It can be recorded as:

[0106] ;

[0107] In the formula, the parameters x1, x2, ..., x n Data fields include household registration, marriage, childbirth, adoption, health information, etc.

[0108] After obtaining the vertices of the graph through the data fields, the link weights between the vertices are calculated through the correlation between the data fields. For example, if income is positively correlated with age, the link weight between the income vertex and the age vertex is relatively large. Assuming that the correlation between income and age is 0.8, the link weight between the income vertex and the age vertex is 0.8.

[0109] like Figure 6 The figure shows a schematic diagram of the constructed graph, where the vertices A, B, C... in the graph are the corresponding data fields.

[0110] (3) Graph decomposition: Using the graph decomposition algorithm, the constructed graph is decomposed into multiple sub-graphs, each of which represents a part of the characteristics of the data.

[0111] For example, the following subgraph decomposition clustering graph decomposition method can be used to obtain complex nonlinear features: , the steps are as follows:

[0112] 1. Select a threshold T for vertex A. For example, when vertex A is a household registration identity, the threshold T is set to 0.8;

[0113] 2. Traverse the correlation between other fields in the graph and the vertex field, and compare the correlation between features with the threshold T.

[0114] For example, we can use the spectral clustering model, assuming that each data field is connected to a vertex by an undirected edge, and the entire sample space forms a graph. For this, we can get its neighborhood matrix , and the degree of each sample point , with the degree of all sample points as the value, forming a diagonal matrix of degree .

[0115] The neighborhood matrix w ij The correlation coefficient between fields can be calculated. If it is greater than the threshold T, it takes 1, and if it is less than the threshold T, it takes 0. According to the clustering index, clustering maximizes the distance between clusters and minimizes the distance within clusters. Partitioning makes the similarity of edges connecting different subgraphs as low as possible, and the similarity of edges within the same subgraph as high as possible.

[0116] 3. The final set of fields is the sub-graph vertex after segmentation. The segmented sub-graph is as follows: Figure 7 As shown, the graph is decomposed into three sub-graphs, and the vertex data fields in the three sub-graphs are: A, B, C, D, E, F, G, and H respectively.

[0117] (4) Data component extraction: Extract data components from the decomposed sub-graphs. These components can represent the core characteristics of the data.

[0118] After the subgraph is cut, the field set ABC can be used as a group of related variables, and the principal component method can be used to condense information and remove redundancy. The principal components extracted by the principal component analysis method are the data elements.

[0119] (5) Data simulation: Use the extracted data elements to simulate the data and predict the future trend of the data.

[0120] Compared with the original data, the extracted data elements contain most of the information of the original data, but the dimension is reduced and the computational complexity is reduced accordingly.

[0121] In summary, this application first uses graph decomposition, which can more effectively express and simulate the real situation of personal information data. In the complex data processing process, graph decomposition can better reveal the inherent structure and laws of the data, and provide strong support for subsequent personal information data analysis and simulation.

[0122] Secondly, the simulation using data elements about personal information improves the accuracy and efficiency of personal information data simulation. The extraction and analysis of data elements can deeply understand the core characteristics of the data, so as to better simulate the future trend of personal information of the data. Compared with traditional data processing methods, this model can more accurately predict the future trend and possible results of personal information.

[0123] The above-mentioned processing embodiments of personal information are only application examples of the present invention. The present invention can also be used in online trading platforms or other data platforms to generate data elements for complex and large-scale data and to perform corresponding data analysis using the generated data elements.

[0124] Through the description of the specific implementation methods, a deeper and more specific understanding of the technical means and effects adopted by the present application to achieve the predetermined purpose should be obtained. However, the accompanying drawings are only for reference and illustration purposes and are not used to limit the present application.

Claims

1. A method for constructing a data component model based on graph decomposition, characterized in that: include: A10, preprocessing the acquired authorized data to be managed, wherein the data to be managed is text data; A20, constructing a map of the data to be governed based on the preprocessed data to be governed, specifically including: A21, extracting data fields from the preprocessed data to be managed as vertices in the graph of the data to be managed; A22, establishing link relationships between vertices in the to-be-governed data graph according to the association relationships in the data fields, and calculating link weights between vertices with link relationships; A30, using a graph decomposition algorithm, decomposing the to-be-governed data graph into a plurality of to-be-governed data sub-graphs, specifically including: A31, setting a corresponding threshold for each vertex in the to-be-governed data graph; A32, select any vertex and traverse to obtain the link weights between other vertices and the selected vertex; A33, if the link weight is greater than a corresponding threshold, clustering the corresponding vertex and the selected vertex into the same subgraph; A34, repeating steps A32 to A33 to decompose the to-be-managed data graph into a plurality of to-be-managed data sub-graphs; A40, extracting data elements related to the data to be managed from the sub-graph of the data to be managed.

2. The method for constructing a data component model based on graph decomposition according to claim 1, characterized in that: In step A10, the acquired data to be managed is preprocessed, including: The acquired data to be managed is cleaned and sorted to remove abnormal values ​​and missing values ​​in the data to be managed.

3. The method for constructing a data component model based on graph decomposition according to claim 1, characterized in that: In step A40, the data fields in each decomposed data sub-graph to be managed are subjected to information condensation and redundancy removal by using the principal component method to obtain the data elements.

4. A data analysis method, characterized in that: include: B10, constructing data elements about the data to be governed by using the data element model construction method based on graph decomposition as described in any one of claims 1 to 3; B20, performing data simulation analysis on the data to be governed through the data element.

5. The data analysis method according to claim 4, characterized in that: In step B20, the future trend of the data to be governed is predicted through data simulation analysis.

6. A data component model construction device based on graph decomposition, characterized in that: include: A preprocessing module, used for preprocessing the acquired authorized data to be managed, wherein the data to be managed is text data; The graph construction module is used to construct a graph of the data to be governed based on the preprocessed data to be governed, specifically including: Extracting data fields from the data to be managed as vertices in the graph of the data to be managed; According to the association relationship in each data field, a link relationship between vertices in the to-be-governed data graph is established, and the link weights between vertices with link relationships are calculated; The graph decomposition module is used to decompose the to-be-governed data graph into multiple to-be-governed data sub-graphs using a graph decomposition algorithm, specifically including: For each vertex in the to-be-governed data graph, set a corresponding threshold; Select any vertex and traverse to obtain the links between other vertices and the selected vertex; If the link weight is greater than a corresponding threshold, the corresponding vertex and the selected vertex are clustered into the same subgraph; Repeat traversing other vertices to decompose the data graph into multiple data sub-graphs to be managed; A data element construction module is used to extract data elements about the data to be governed from the sub-graph.

7. The data component model construction device based on graph decomposition according to claim 6, characterized in that: The preprocessing module preprocesses the acquired data to be managed, including: The acquired data to be managed is cleaned and sorted, and abnormal values ​​and missing values ​​in the acquired data to be managed are removed.

8. The data component model construction device based on graph decomposition according to claim 6, characterized in that: The data element construction module condenses information and removes redundancy on the data fields in each decomposed data sub-graph to be governed by using the principal component method to obtain the data element.

9. A data analysis device, characterized in that: include: A data element model construction device, using the data element model construction device based on graph decomposition as described in any one of claims 6 to 8, for constructing data elements about the data to be governed; The analysis module is used to perform data simulation analysis on the data to be governed through the data element.

10. The data analysis device according to claim 9, characterized in that: The analysis module predicts the future trend of the data to be governed through data simulation analysis.

Images