Data analysis method based on artificial intelligence

Abstract

The invention provides an artificial intelligence-based data analysis method, which comprises the following steps of: integrating and cleaning collected multi-source data, storing the data into a relational database, designing an entity-relationship-based data model to construct nodes and relationships, and storing the nodes and the relationships into the relational database; analyzing and calculating the knowledge graph through a community discovery algorithm, a graph neural network and quantitative analysis to capture complex features and potential abnormal conditions of the knowledge graph; and fitting three results of the community discovery algorithm, the graph neural network and the quantitative analysis through an integration algorithm, and generating final suspicious score judgment in a multi-party weighted voting mode. According to the method, suspected problem clues are extracted from thousands of or even hundreds of thousands of data, a possible clue direction is provided for case investigation, problems can be found more timely, supervision is more accurate and efficient, and case handling efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular, to a data analysis method based on artificial intelligence. Background Art

[0002] Data analysis is a systematic and comprehensive work, covering the process from data collection, data cleaning, data organization, data analysis, data mining to data visualization. This process not only requires the professional knowledge and experience of the discipline inspection and supervision departments, but also requires the use of modern information technology to conduct in-depth mining and analysis of data to discover the implicit information in the data.

[0003] The existing methods for data analysis and their defects are as follows:

[0004] The Chinese patent "CN115618007 Supervision Data Analysis System Based on Knowledge Graph" provides a framework diagram of a supervision data analysis system, which mainly includes: knowledge modeling unit, knowledge management unit, knowledge calculation unit, knowledge application unit, knowledge storage unit and data warehouse unit. The data warehouse unit stores source data; the knowledge modeling unit obtains source data, creates a domain map of the supervision object, and creates a character portrait of the supervision object; the knowledge management unit accesses the information data of the person under investigation or extracts the corresponding source data to form semantic knowledge that can be understood by the computer, and maps and associates it with the character portrait of the supervision object to form a supervision map; the knowledge calculation unit obtains the supervision map, performs normalized fusion processing, and updates the calculation results to the supervision map; the knowledge application unit searches for qualified entity data, performs secondary data processing, obtains the relationship diagram of the map entity and displays it.

[0005] The technical solution described in the Chinese patent "CN115618007 Supervision Data Analysis System Based on Knowledge Graph" is to perform entity calculations on the constructed graph through multiple independent computing units when conducting data mining on the knowledge graph, and then normalize and fuse these entities. However, there is no effective collaboration and resource sharing between these computing units, which may lead to idle computing resources and low efficiency. If there are errors or errors in a computing unit, these errors may have a negative impact on the results of normalization and fusion.

[0006] The Chinese patent "CN116777634 A financial data analysis system and method based on artificial intelligence" obtains financial data from a pre-configured financial data source, processes the financial data to obtain triple data, and writes the triple data into a pre-built knowledge graph database. In the knowledge graph network model, the target node is used as the center to determine its associated nodes, obtain the historical data of the target node and the associated nodes, and generate a training data set for the multivariate time series model based on this historical data. The multivariate time series model of the target node and the associated nodes is trained with the training data set, and the model is used to predict the future data of the target node, thereby improving the accuracy of financial time series prediction.

[0007] The technical solution described in the Chinese patent "CN116777634 A financial data analysis system and method based on artificial intelligence" is mainly aimed at analyzing and processing data in the financial field. The data attributes of the Commission for Discipline Inspection and Supervision are different from financial data. Its data comes from multiple aspects and also implies more information. The model needs to be rebuilt and trained to achieve better performance. Summary of the invention

[0008] According to the technical problems raised above, a data analysis method based on artificial intelligence is provided. The present invention extracts clues of suspected problems from tens of thousands or even hundreds of thousands of data, provides possible clue directions, makes problem discovery more timely, makes supervision more accurate and efficient, and improves efficiency.

[0009] The technical means adopted by the present invention are as follows:

[0010] A data analysis method based on artificial intelligence, comprising:

[0011] S1. Integrate and clean the collected data from multiple sources and store them in a relational database. Design an entity-relationship-based data model to construct nodes and relationships and store them in a relational database.

[0012] S2. Analyze and calculate the knowledge graph through community discovery algorithms, graph neural networks, and quantitative analysis to capture the complex characteristics and potential anomalies of the knowledge graph;

[0013] S3. The three results of community discovery algorithm, graph neural network and quantitative analysis are fitted through an integrated algorithm, and the final suspicious score judgment is generated by multi-party weighted voting.

[0014] Furthermore, step S1 specifically includes:

[0015] S11. Integrate the collected data from different sources into a consistent data format through data integration and data conversion, perform data cleaning operations, and store the converted data in a relational database;

[0016] S12. In order to adapt to specific application scenarios, a data model based on entity-relationship is designed. By defining entities (nodes) and relationships (edges), a structured data model is constructed. The data model can fully reflect the subject person, transaction and real estate information involved in the relational database;

[0017] S13. On the basis of constructing the data model, corresponding nodes and relationships are created, wherein the nodes represent entities in the data model, and the relationships represent associations and connections between entities;

[0018] S14. Store the created nodes and relationships in the Neo4j graph database.

[0019] Furthermore, in step S11:

[0020] Data cleaning operations are used to ensure the accuracy and completeness of data, including: processing missing values, removing duplicate data, and resolving data conflicts.

[0021] Furthermore, in step S11:

[0022] The relational databases include: Neo4j, MySQL, Oracle, PostgreSQL, and Renmin University of China Golden Warehouse.

[0023] Further, in step S14:

[0024] The Neo4j graph database is a high-performance graph database management system that is specifically used to store and process graph data, and is used to provide powerful data query, graph analysis, and graph algorithm functions, thereby supporting more in-depth data analysis and query needs; through Neo4j, relevant data can be quickly retrieved and analyzed, and valuable information can be obtained from it.

[0025] Further, step S2 includes:

[0026] S21. Use Louvain algorithm to discover community structure in graphs.

[0027] S22. Use the trained graph neural network to classify the nodes and edges in the graph to identify suspicious nodes and edges;

[0028] S23. Detect suspicious entities and relationships based on requirements and business rules to discover potential violations or anomalies.

[0029] Furthermore, in step S21, the Louvain algorithm is used to discover the community structure in the graph, which mainly includes two stages:

[0030] Modularity optimization stage:

[0031] In the modularity optimization phase, each node is used as its own community label. Then, each node traverses all its neighboring nodes and tries to update its own community label to the community label of the neighboring nodes. The Louvain algorithm selects the community label with the largest modularity increment as the new community label of the node. This process uses the greedy idea and repeats this process until all nodes cannot increase their modularity by changing their community labels.

[0032] Network cohesion stage:

[0033] In the network cohesion stage, each community is merged into a new super node. The edge weight of this super node is the sum of the edge weights of all nodes in the original community. The nodes in the original network are condensed into larger super nodes to form a new network.

[0034] Further, step S23 specifically includes:

[0035] S231. In order to make full use of objective factual data, statistical, distance and density-based anomaly detection algorithms are used to detect the degree, aggregation and centrality statistical indicators of entities and relationships, and relatively low or high values ​​are regarded as outliers;

[0036] S232. Statistical anomaly detection algorithms use statistical characteristics and distribution assumptions of data to identify data points that are significantly different from normal behavior and detect entities or relationships that deviate significantly from normal data distribution;

[0037] S233, a distance-based anomaly detection algorithm, which determines whether an anomaly exists by calculating the distance or similarity between entities or relationships;

[0038] S234, if an entity or relationship is far away from or has low similarity with other entities or relationships, it is considered to be likely to be abnormal;

[0039] S235, density-based anomaly detection algorithm, further determines whether it is abnormal by calculating density distribution.

[0040] Furthermore, in step S3, the integrated algorithm combines the results of multiple algorithms to make a final suspicious score determination in a multi-party weighted voting manner, specifically including:

[0041] S31. For each entity or relationship, each algorithm will give a suspicious score, representing the degree of abnormality;

[0042] S32. Obtain a final suspicious score by performing a weighted sum or averaging operation on the suspicious scores given by each algorithm.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1. The present invention provides an artificial intelligence-based data analysis method, which utilizes the collected case data to convert it into structured data through data conversion and cleaning, and then uses a graph model to establish a visual map. By analyzing and mining detailed information such as the social network relationships of suspicious persons, clues to the case are sought, suspicious features are quickly captured, and analysis of case-related data is realized.

[0045] 2. The present invention provides an artificial intelligence-based data analysis method, which provides an interactive page front-end interface, a query and mining function back-end interface, and a suspicious score analysis model, providing a powerful tool and platform for case handlers.

[0046] 3. The artificial intelligence-based data analysis method provided by the present invention can enhance the case handlers' understanding and in-depth analysis of the case, assist them in discovering potential clues, and improve the case-solving rate.

[0047] Based on the above reasons, the present invention can be widely promoted in fields such as data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0049] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0052] like Figure 1 As shown, the present invention provides a data analysis method based on artificial intelligence, comprising:

[0053] S1. Integrate and clean the collected data from multiple sources and store them in a relational database. Design an entity-relationship-based data model to construct nodes and relationships and store them in a relational database.

[0054] S2. Analyze and calculate the knowledge graph through community discovery algorithms, graph neural networks, and quantitative analysis to capture the complex characteristics and potential anomalies of the knowledge graph;

[0055] S3. The three results of community discovery algorithm, graph neural network and quantitative analysis are fitted through an integrated algorithm, and the final suspicious score judgment is generated by multi-party weighted voting.

[0056] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:

[0057] S11. Integrate the collected data from different sources into a consistent data format through data integration and data conversion, perform data cleaning operations, and store the converted data in a relational database;

[0058] S12. In order to adapt to specific application scenarios, a data model based on entity-relationship is designed. By defining entities (nodes) and relationships (edges), a structured data model is constructed to better organize and manage data. The data model can fully reflect the subject person, transaction and real estate information involved in the relational database;

[0059] S13. On the basis of building the data model, corresponding nodes and relationships are created, where nodes represent entities in the data model, such as people, transactions, and real estate. Relationships represent associations and connections between entities, such as the relationship between people, the relationship between transactions and people, and the relationship between real estate and people. Such definitions of nodes and relationships will help to better describe and understand the complex relationships between data.

[0060] S14. Store the created nodes and relationships in the Neo4j graph database.

[0061] In this embodiment, the collected raw data is preprocessed, a suitable data model is designed to construct entities and relationships, and it is stored in the Neo4j graph database to provide a solid foundation for subsequent data query and analysis. This process can help better understand and utilize data information and provide targeted solutions for specific application scenarios.

[0062] In specific implementation, as a preferred embodiment of the present invention, in step S11:

[0063] Data cleaning operations are used to ensure the accuracy and completeness of data, including: processing missing values, removing duplicate data, and resolving data conflicts.

[0064] In specific implementation, as a preferred embodiment of the present invention, in step S11:

[0065] The relational databases include: Neo4j, MySQL, Oracle, PostgreSQL, and Renmin University of China Golden Warehouse.

[0066] In specific implementation, as a preferred embodiment of the present invention, in step S14:

[0067] The Neo4j graph database is a high-performance graph database management system that is specifically used to store and process graph data, and is used to provide powerful data query, graph analysis, and graph algorithm functions, thereby supporting more in-depth data analysis and query needs; Neo4j can be used to quickly retrieve and analyze relevant data and obtain valuable information from it.

[0068] In specific implementation, as a preferred embodiment of the present invention, step S2 includes:

[0069] S21. Use Louvain algorithm to discover community structure in graphs.

[0070] In this embodiment, in order to extract local features of the graph, the community discovery algorithm provided by Neo4j is used to find communities related to suspicious persons. Neo4j provides a variety of community discovery algorithms, including Louvain algorithm, LabelPropagation algorithm, Triangle Count algorithm, etc. In this module, the Louvain algorithm is selected to discover the community structure in the graph. The Louvain algorithm is a modularity-based community discovery algorithm that aims to group nodes into groups with close internal connections and relatively distant ones.

[0071] S22. Use the trained graph neural network to classify the nodes and edges in the graph to identify suspicious nodes and edges;

[0072] In this embodiment, in order to capture the global features of knowledge graph data, the use of graph neural network (GNN) is an effective method. GNN has the ability to learn complex relationships between nodes and can generate vector representations of nodes. By using graph data as input and training the GNN model, the feature representations of nodes and edges can be learned. During the training process, supervised learning can be performed using the label information of nodes and edges, or unsupervised learning methods such as graph autoencoders can be used. Supervised learning uses known label information of nodes and edges to guide the model to learn the feature representations of nodes and edges, so that the model can better classify. The unsupervised learning method learns the representations of nodes and edges in the graph through the reconstruction loss of the model itself or the properties of other graph structures without relying on external label information. Through the trained GNN model, the nodes and edges in the graph are classified to identify suspicious nodes and edges. For example, in an investigation, a GNN model can be used to identify potential suspicious people or abnormal behaviors. The prediction results can be presented through a visualization tool, showing the nodes and edges in the graph in a graphical form, highlighting the nodes and edges identified as suspicious, thereby helping users discover hidden patterns, potential associations, and abnormal situations.

[0073] S23. Detect suspicious entities and relationships based on requirements and business rules to discover potential violations or anomalies.

[0074] In specific implementation, as a preferred embodiment of the present invention, in step S21, the Louvain algorithm is used to discover the community structure in the graph, which mainly includes two stages:

[0075] Modularity optimization stage:

[0076] In the modularity optimization phase, each node is used as its own community label. Then, each node traverses all its neighboring nodes and tries to update its own community label to the community label of the neighboring nodes. The Louvain algorithm selects the community label with the largest modularity increment as the new community label of the node. This process uses the greedy idea and repeats this process until all nodes cannot increase their modularity by changing their community labels.

[0077] Network cohesion stage:

[0078] In the network cohesion stage, each community is merged into a new super node. The edge weight of this super node is the sum of the edge weights of all nodes in the original community. The nodes in the original network are condensed into larger super nodes to form a new network.

[0079] In this embodiment, by using the Louvain algorithm, the community structure in the graph can be found and related nodes can be clustered together. Applying it to the communities associated with suspicious persons can help discover potential related persons, thereby providing more comprehensive analysis and inspection.

[0080] In specific implementation, as a preferred embodiment of the present invention, step S23 specifically includes:

[0081] S231. In order to make full use of objective factual data, statistical, distance and density-based anomaly detection algorithms are used to detect the degree, aggregation and centrality statistical indicators of entities and relationships, and relatively low or high values ​​are regarded as outliers;

[0082] S232. Statistical anomaly detection algorithms use statistical characteristics and distribution assumptions of data to identify data points that are significantly different from normal behavior and detect entities or relationships that deviate significantly from normal data distribution;

[0083] S233, a distance-based anomaly detection algorithm, which determines whether an anomaly exists by calculating the distance or similarity between entities or relationships;

[0084] S234, if an entity or relationship is far away from or has low similarity with other entities or relationships, it is considered to be likely to be abnormal;

[0085] S235, density-based anomaly detection algorithm, further determining whether it is abnormal by calculating density distribution;

[0086] In this embodiment, if the density of the surrounding neighbors of an entity or relationship is low or low, then it may be an outlier. Multiple rules are formulated to detect suspicious entities and relationships to discover potential violations or abnormal situations. These rules are formulated based on requirements and business rules, focusing on 24 features, and extracting these 24 feature data in turn in combination with Cypher statements, and generating a feature matrix for each subject.

[0087] In specific implementation, as a preferred embodiment of the present invention, in step S3, the integrated algorithm combines the results of multiple algorithms to perform the final suspicious score determination in a multi-party weighted voting manner, specifically including:

[0088] S31. For each entity or relationship, each algorithm will give a suspicious score, representing the degree of abnormality;

[0089] S32. Obtain a final suspicious score by performing a weighted sum or averaging operation on the suspicious scores given by each algorithm.

[0090] In this embodiment, the above three algorithms play different roles in the knowledge graph analysis, and calculate and analyze the data from the perspectives of local features, global features and quantitative analysis. However, using a single algorithm alone may not be able to fully capture the complex features and potential anomalies of the knowledge graph. Therefore, integrating these algorithms is an effective strategy. The advantage of the integrated method is that it can comprehensively consider the advantages of different algorithms and alleviate the defects and limitations of a single algorithm. Different algorithms may have different sensitivities and expressiveness for different types of anomalies. Through integration, the results of these algorithms can be comprehensively utilized to provide more comprehensive and diverse judgment results. For an entity or relationship, if multiple algorithms give a high suspicious score, it can be more confident that it is an outlier. On the contrary, if only one algorithm gives a high suspicious score and the scores of other algorithms are low, it needs to be further analyzed to determine its abnormality. By integrating the results of different algorithms, a more powerful integrated model is constructed. This model can comprehensively consider the information of local features, global features and quantitative analysis, and improve the performance and effect of identifying suspicious entities and relationships. At the same time, the integrated model also has good flexibility and can be adjusted and optimized according to actual conditions to meet the needs of specific application scenarios.

[0091] In summary, the present invention provides a data analysis method based on artificial intelligence. First, the collected multi-source data is integrated, cleaned and stored in a relational database, and a data model based on entity-relationship is designed to construct nodes and relationships, which are stored in a Neo4j graph database. Then, the knowledge graph is analyzed and calculated through three algorithms: community discovery algorithm, graph neural network and quantitative analysis, so as to capture the complex characteristics and potential abnormalities of the knowledge graph. Finally, the above three results are fitted through an integrated algorithm, and the final suspicious score judgment is generated by multi-party weighted voting.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data analysis method based on artificial intelligence, characterized in that: include: S1. Integrate and clean the collected data from multiple sources and store them in a relational database. Design an entity-relationship-based data model to construct nodes and relationships and store them in a relational database. S2. Analyze and calculate the knowledge graph through community discovery algorithms, graph neural networks, and quantitative analysis to capture the complex characteristics and potential anomalies of the knowledge graph; S3. The three results of community discovery algorithm, graph neural network and quantitative analysis are fitted through an integrated algorithm, and the final suspicious score judgment is generated by multi-party weighted voting.

2. The data analysis method based on artificial intelligence according to claim 1, characterized in that: Step S1 specifically includes: S11. Integrate the collected data from different sources into a consistent data format through data integration and data conversion, perform data cleaning operations, and store the converted data in a relational database; S12. In order to adapt to specific application scenarios, a data model based on entity-relationship is designed. By defining entities (nodes) and relationships (edges), a structured data model is constructed. The data model can fully reflect the subject person, transaction and real estate information involved in the relational database; S13. On the basis of constructing the data model, corresponding nodes and relationships are created, wherein the nodes represent entities in the data model, and the relationships represent associations and connections between entities; S14. Store the created nodes and relationships in the Neo4j graph database.

3. The data analysis method based on artificial intelligence according to claim 2, characterized in that: In step S11: Data cleaning operations are used to ensure the accuracy and completeness of data, including: processing missing values, removing duplicate data, and resolving data conflicts.

4. The data analysis method based on artificial intelligence according to claim 2, characterized in that: In step S11: The relational databases include: Neo4j, MySQL, Oracle, PostgreSQL, and Renmin University of China Golden Warehouse.

5. The data analysis method based on artificial intelligence according to claim 2, characterized in that: In step S14: The Neo4j graph database is a high-performance graph database management system that is specifically used to store and process graph data, and is used to provide powerful data query, graph analysis, and graph algorithm functions, thereby supporting more in-depth data analysis and query needs; Neo4j can be used to quickly retrieve and analyze relevant data and obtain valuable information from it.

6. The data analysis method based on artificial intelligence according to claim 1, characterized in that: Step S2 comprises: S21. Use Louvain algorithm to discover community structure in graphs. S22. Use the trained graph neural network to classify the nodes and edges in the graph to identify suspicious nodes and edges; S23. Detect suspicious entities and relationships based on requirements and business rules to discover potential violations or anomalies.

7. The data analysis method based on artificial intelligence according to claim 6, characterized in that: In step S21, the Louvain algorithm is used to discover the community structure in the graph, which mainly includes two stages: Modularity optimization stage: In the modularity optimization phase, each node is used as its own community label. Then, each node traverses all its neighboring nodes and tries to update its own community label to the community label of the neighboring nodes. The Louvain algorithm selects the community label with the largest modularity increment as the new community label of the node. This process uses the greedy idea and repeats this process until all nodes cannot increase their modularity by changing their community labels. Network cohesion stage: In the network cohesion stage, each community is merged into a new super node. The edge weight of this super node is the sum of the edge weights of all nodes in the original community. The nodes in the original network are condensed into larger super nodes to form a new network.

8. The data analysis method based on artificial intelligence according to claim 6, characterized in that: Step S23 specifically includes: S231. In order to make full use of objective factual data, statistical, distance and density-based anomaly detection algorithms are used to detect the degree, aggregation and centrality statistical indicators of entities and relationships, and relatively low or high values ​​are regarded as outliers; S232. Statistical anomaly detection algorithms use statistical characteristics and distribution assumptions of data to identify data points that are significantly different from normal behavior and detect entities or relationships that deviate significantly from normal data distribution; S233, a distance-based anomaly detection algorithm, which determines whether an anomaly exists by calculating the distance or similarity between entities or relationships; S234, if an entity or relationship is far away from or has low similarity with other entities or relationships, it is considered to be likely to be abnormal; S235, density-based anomaly detection algorithm, further determines whether it is abnormal by calculating density distribution.

9. The data analysis method based on artificial intelligence according to claim 1, characterized in that: In step S3, the integrated algorithm combines the results of multiple algorithms and performs the final suspicious score determination in a multi-party weighted voting manner, specifically including: S31. For each entity or relationship, each algorithm will give a suspicious score, representing the degree of abnormality; S32. Obtain a final suspicious score by performing a weighted sum or averaging operation on the suspicious scores given by each algorithm.

Images