Data analysis method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of internet service data analysis, in particular to a data analysis method and device, electronic equipment and a storage medium, and aims to establish a directed weighted graph corresponding to service data of a target service, identify all data links on the basis of the directed weighted graph, and further perform service potential safety hazard analysis. And efficient capturing and processing of complex dependency relationships and dynamic changes are realized. According to the method, the interdependence and the data flow direction between the data entities are visually displayed, the importance degree of the business is accurately quantified through the weight of the edge, and a comprehensive and deep view angle is provided for business potential safety hazard analysis. Compared with the prior art, the method has the advantages that potential safety risk points can be identified more accurately, early warning can be performed in time, and measures can be taken, so that the safety and stability of service data are effectively improved, and the technical problem that a traditional method is difficult to deal with complex dependence and dynamic change is solved.

Description

Technical Field

[0001] The present application relates to the technical field of Internet business data analysis, and specifically to a data analysis method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of Internet technology, especially the widespread application of cloud computing, big data, artificial intelligence and other technologies, the business scale and complexity of Internet companies have increased dramatically, and user data has shown an explosive growth trend. These data are not only huge in volume, but also highly correlated, forming an intricate graph network structure, in which various nodes (such as services, applications, data tables, etc.) are closely connected through complex dependencies and interaction logic.

[0003] In this context, traditional operation, maintenance, monitoring and management methods face unprecedented challenges. Traditional monitoring systems focus more on the performance indicators of a single service or component, while ignoring the interactions and impacts between these components or services. When a problem occurs in the system, this monitoring method often fails to quickly locate the root cause of the problem, increasing the difficulty and cost of troubleshooting. Traditional operation, maintenance and management methods are difficult to automatically discover and visualize these implicit dependencies, resulting in huge challenges for managers in understanding the system structure and formulating operation and maintenance strategies. With the rapid development of business and the continuous iteration of technology, the system structure and dependencies will change frequently. Traditional operation, maintenance and management methods often find it difficult to adapt to such rapid changes, resulting in many difficulties in system upgrades, expansions or reconstructions.

[0004] It can be seen that the existing technology has the problem of being difficult to capture and handle complex dependencies and dynamic changes. Summary of the invention

[0005] The purpose of the present application is to provide a data analysis method, device, electronic device and storage medium to solve the problem in the prior art that it is difficult to capture and process complex dependencies and dynamic changes.

[0006] In order to achieve the above purpose, the technical solution adopted in this application is as follows:

[0007] According to one aspect of an embodiment of the present application, a data analysis method is provided, including: obtaining a directed weighted graph corresponding to business data of a target business, wherein the directed weighted graph includes a plurality of nodes corresponding to data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency relationship and data flow between the data of the two nodes connected by the directed weighted edge; identifying all data links from the directed weighted graph; performing business security risk analysis on all data links to obtain analysis results.

[0008] According to the above technical means, by constructing a directed weighted graph corresponding to the business data of the target business, and identifying all data links on this basis, and then conducting business security risk analysis, it is possible to efficiently capture and process complex dependencies and dynamic changes. This method not only intuitively displays the interdependence between data entities and the direction of data flow, but also accurately quantifies the importance of the business through the weight of the edge, providing a comprehensive and in-depth perspective for business security risk analysis. Compared with existing technologies, it can more accurately identify potential security risk points, issue warnings in a timely manner and take measures, thereby effectively improving the security and stability of business data, and solving the technical problem that traditional methods are difficult to deal with complex dependencies and dynamic changes.

[0009] Furthermore, a directed weighted graph corresponding to the target business data of the target business is obtained, including: constructing a directed graph based on the business data, wherein the directed graph includes multiple nodes and directed edges between the nodes; assigning a node weight to each node in the directed graph according to a preset weight assignment rule; assigning an edge weight to the directed edge according to the node weights of the nodes connected by the directed edge in the directed graph, thereby obtaining a directed weighted graph.

[0010] Based on the above technical means, a directed weighted graph corresponding to the target business data is constructed through a systematic method, achieving a deep understanding and structured representation of the business data. By constructing a directed graph and assigning weights to nodes and edges, not only the correlation and dependency between data entities are clearly displayed, but also the importance of these relationships is quantified. This method significantly improves the ability to capture the complexity and dynamic changes of business data, provides a solid foundation for subsequent business security risk analysis, and helps to more accurately identify potential risks, thereby enhancing the security of business data and the robustness of business operations.

[0011] Furthermore, based on the business data, a directed graph is constructed, including: extracting multiple data entities from the business data, generating a node for each data entity, wherein the node is a data node or a job node; according to the dependency relationship between the data and the data flow direction, establishing directed edges for multiple nodes corresponding to the multiple data entities, wherein the directed edges are input edges or output edges.

[0012] According to the above technical means, by extracting data entities from business data and constructing directed graphs, accurate mapping of complex dependencies between data and data flows is achieved. By generating nodes for each data entity and establishing directed edges based on the dependencies and data flows between them, this method not only clearly reveals the logical chain of data processing, but also facilitates subsequent analysis and optimization.

[0013] Furthermore, edge weights are assigned to the directed edges according to the node weights of the nodes connected by the directed edges in the directed graph to obtain a directed weighted graph, including: obtaining the target node weight of the node pointed to by the target directed edge, wherein the target directed edge is any directed edge with undetermined edge weight; assigning the target node weight to the target directed edge to obtain the edge weight of the target directed edge.

[0014] According to the above technical means, a directed weighted graph is constructed quickly and easily by directly assigning the weight of a node in a directed graph to the directed edge connecting the node. This method not only maintains the business importance information represented by the node weight, but also intuitively reflects the key path and the strength of the dependency relationship in the data flow process through the edge weight. This technical means simplifies the construction process of the weighted graph and improves efficiency, while ensuring that the weighted graph can accurately reflect the complexity and dynamics of business data, providing more accurate and comprehensive data support for subsequent business security risk analysis.

[0015] Furthermore, all data links are analyzed for business security risks to obtain analysis results, including: monitoring multiple target indicators of nodes in a directed weighted graph, and when any target indicator of the node is abnormal, the node is determined to be a faulty node, wherein the multiple target indicators include: data flow indicators and business execution status indicators; calling a core node identification algorithm to identify the core node in each data link, wherein the core node is the node with the largest connection weight value in the data link, and the connection weight is used to characterize the importance of the node in the data link; calling a weak edge identification algorithm to identify all weak edges in each data link, wherein a weak edge refers to an edge that is no longer connected to the data link after being deleted.

[0016] According to the above technical means, by comprehensively monitoring node indicators and accurately identifying core nodes and weak edges, in-depth mining and analysis of potential business security risks in data links are achieved. By real-time monitoring of node data traffic and business execution status, faulty nodes can be discovered and located in a timely manner to avoid the spread of security risks. At the same time, through the core node identification algorithm, key nodes in the data link are accurately locked. These nodes are crucial to business continuity and provide a clear direction for priority protection and optimization. In addition, the identification of weak edges reveals the vulnerable links in the data link, which helps to take measures in advance to strengthen it and prevent potential risks. This series of technical means work together to significantly improve the security and stability of business data links, providing a strong guarantee for the safe operation of the business.

[0017] Furthermore, after analyzing all data links and obtaining the analysis results, it includes: when it is determined that there are multiple faulty nodes in the directed weighted graph, all nodes in the data link where each faulty node is located are sorted to determine the order in which the faulty nodes are restored.

[0018] According to the above technical means, after comprehensively analyzing all data links and identifying multiple faulty nodes, technical means are used to scientifically sort all nodes in the data links associated with each faulty node, so as to accurately determine the priority of fault recovery. This method can significantly improve the efficiency and accuracy of fault handling, ensure the rapid recovery of key business links, minimize service interruption time and the impact on overall system performance, and thus enhance system stability and user satisfaction.

[0019] Furthermore, after all data links are identified from the directed weighted graph, it also includes: analyzing the out-degree and in-degree of each node in each data link to obtain a target node whose out-degree is higher than a first preset value and / or whose in-degree is higher than a second preset value; and reallocating the business data corresponding to the target node.

[0020] According to the above technical means, after exhaustively identifying all data links from the directed weighted graph, the out-degree and in-degree of each node are further analyzed to identify those key target nodes whose out-degree is higher than the first preset value or whose in-degree is higher than the second preset value, and the business data of these nodes are intelligently redistributed accordingly. This technical means can optimize the distribution of network traffic, reduce the pressure of high-load nodes, and prevent the formation of potential bottlenecks, thereby enhancing the load balancing ability and response speed of the entire system and ensuring efficient and stable transmission of business data.

[0021] According to another aspect of the embodiment of the present application, a data analysis device is also provided, including: a data acquisition module, used to acquire a directed weighted graph corresponding to the business data of the target business, wherein the directed weighted graph includes multiple nodes corresponding to the data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency and data flow between the data of the two nodes connected by the directed weighted edge; a link identification module, used to identify all data links from the directed weighted graph; a link analysis module, used to perform business security risk analysis on all data links to obtain analysis results.

[0022] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the steps of the data analysis method in any of the above embodiments by running the computer program stored in the memory.

[0023] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of the data analysis method in any of the above embodiments when run.

[0024] Beneficial effects of this application:

[0025] This application constructs a directed weighted graph corresponding to the business data of the target business, identifies all data links on this basis, and then conducts business security risk analysis, thereby achieving efficient capture and processing of complex dependencies and dynamic changes. This method not only intuitively displays the interdependence between data entities and data flows, but also accurately quantifies the importance of the business through the weight of the edges, providing a comprehensive and in-depth perspective for business security risk analysis. Compared with existing technologies, it can more accurately identify potential security risk points, issue warnings in a timely manner and take measures, thereby effectively improving the security and stability of business data, and solving the technical problem that traditional methods are difficult to deal with complex dependencies and dynamic changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 is a schematic diagram of a hardware environment of an optional data analysis method provided in an embodiment of the present application;

[0029] Figure 2 It is a flowchart of an optional data analysis method provided in an embodiment of the present application;

[0030] Figure 3 is a structural block diagram of an optional data analysis device provided in an embodiment of the present application;

[0031] Figure 4 It is a structural block diagram of an optional electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] According to one aspect of an embodiment of the present application, a data analysis method is provided. Optionally, in this embodiment, the above data analysis method can be applied to a hardware environment consisting of a terminal and a server. The server is connected to the terminal via a network, and can be used to provide services for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server.

[0035] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal may not be limited to a PC, a mobile phone, a tablet computer, etc.

[0036] The data analysis method of the embodiment of the present application can be executed by a server, a terminal, or a server and a terminal together. The terminal can also execute the data analysis method of the embodiment of the present application by a client installed thereon.

[0037] Taking the data analysis method in this embodiment executed by the server as an example, please refer to Figure 1 , Figure 1 is a schematic diagram of a hardware environment of an optional data analysis method provided in an embodiment of the present application, such as Figure 1As shown, the hardware environment of the data analysis method includes: a terminal 102, and a server 104 connected to the terminal 102 through a network. The server 104 is used to execute the data analysis method of the embodiment of the present application to analyze the business data of the target business; the terminal 102 is used to display the analysis result, wherein the analysis result can be obtained by the server 104 executing the data analysis method of the embodiment of the present application.

[0038] The data analysis method in this embodiment can be applied to the scenario of analyzing the business data of Internet companies. With the rapid development of Internet technology, especially the widespread application of technologies such as cloud computing, big data, and artificial intelligence, the business scale and complexity of Internet companies have increased dramatically, and user data has shown an explosive growth trend. These data are not only huge in volume, but also highly correlated, forming an intricate graph network structure, in which each node (such as service, application, data table, etc.) is closely connected through complex dependencies and interaction logic. Traditional monitoring systems focus more on the performance indicators of a single service or component, while ignoring the interactions and influences between these components or services. When a problem occurs in the system, this monitoring method is often difficult to quickly locate the root cause of the problem, increasing the difficulty and cost of troubleshooting. Traditional operation and maintenance and management methods are difficult to automatically discover and visualize these implicit dependencies, resulting in huge challenges for managers in understanding the system structure and formulating operation and maintenance strategies. With the rapid development of business and the continuous iteration of technology, the system structure and dependencies will change frequently. Traditional operation and maintenance and management methods are often difficult to adapt to such rapid changes, resulting in many difficulties in system upgrades, expansions, or reconstructions.

[0039] It can be seen that the existing technology has the problem of being difficult to capture and process complex dependencies and dynamic changes. To solve the above problems, this embodiment provides a data analysis method running on the above server, please refer to Figure 2 , Figure 2 is a flow chart of an optional data analysis method provided in an embodiment of the present application, such as Figure 2 As shown, the data analysis method of the embodiment of the present application specifically includes the following steps:

[0040] Step S201, obtaining a directed weighted graph corresponding to the business data of the target business, wherein the directed weighted graph includes a plurality of nodes corresponding to the data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency relationship and data flow direction between the data of the two nodes connected by the directed weighted edge;

[0041] Step S202, identifying all data links from the directed weighted graph;

[0042] Step S203: Perform business security risk analysis on all data links to obtain analysis results.

[0043] Through the above steps S201 to S203, by constructing a directed weighted graph corresponding to the business data of the target business, and identifying all data links on this basis, and then conducting business security risk analysis, efficient capture and processing of complex dependencies and dynamic changes are achieved. This method not only intuitively displays the interdependence and data flow between data entities, but also accurately quantifies the importance of the business through the weight of the edge, providing a comprehensive and in-depth perspective for business security risk analysis. Compared with the existing technology, it can more accurately identify potential security risk points, issue warnings in a timely manner and take measures, thereby effectively improving the security and stability of business data, and solving the technical problem that traditional methods are difficult to deal with complex dependencies and dynamic changes.

[0044] Combine the following Figure 2 The data analysis method in the examples of the present application is explained.

[0045] In the technical solution of step S201, a directed weighted graph corresponding to the business data of the target business is obtained. In this embodiment, the directed weighted graph includes a plurality of nodes corresponding to the data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency relationship and data flow between the data of the two nodes connected by the directed weighted edge.

[0046] In practice, the target business mainly refers to various processing jobs, such as Flink jobs, Spark jobs, etc. Business data refers to each job and the data required in each job, such as data tables (such as Hive tables) and Kafka topics. To obtain the directed weighted graph corresponding to the business data of the target business, you can directly obtain a pre-built directed weighted graph or directly build a directed weighted graph using the business data of the target business.

[0047] As an optional embodiment, obtaining a directed weighted graph corresponding to target business data of a target business includes: constructing a directed graph based on the business data, wherein the directed graph includes multiple nodes and directed edges between the nodes; assigning a node weight to each node in the directed graph according to a preset weight assignment rule; assigning an edge weight to the directed edge according to the node weights of the nodes connected by the directed edge in the directed graph, thereby obtaining a directed weighted graph.

[0048] In this embodiment, the business data of the target business is directly used to directly construct a directed weighted graph. When constructing the directed weighted graph, first, a directed graph is constructed according to the business data, and the directed graph includes multiple nodes and directed edges between the nodes; second, a node weight is assigned to each node in the directed graph according to a preset weight assignment rule; third, an edge weight is assigned to the directed edge according to the node weight of the node connected by the directed edge in the directed graph, thereby obtaining a directed weighted graph.

[0049] In specific practice, the data entity is assigned a level weight according to the business importance, that is, the node is assigned a node weight according to the business importance. Business importance indicates the importance of the business. In practical applications, the business importance is determined in advance according to the importance of the actual business. In this embodiment, the business importance is divided into 1-5 levels in order from high to low, and the corresponding relationship between each level and weight is: level 1 corresponds to a weight of 10, level 2 corresponds to a weight of 5, level 3 corresponds to a weight of 3, level 4 corresponds to a weight of 2, and level 5 corresponds to a weight of 1.

[0050] It is understandable that by constructing a directed weighted graph corresponding to the target business data in a systematic way, a deep understanding and structured representation of business data is achieved. By constructing a directed graph and assigning weights to nodes and edges, not only the correlation and dependency between data entities are clearly displayed, but also the importance of these relationships is quantified. This method significantly improves the ability to capture the complexity and dynamic changes of business data, provides a solid foundation for subsequent business security risk analysis, and helps to more accurately identify potential risks, thereby enhancing the security of business data and the robustness of business operations.

[0051] As an optional embodiment, a directed graph is constructed based on business data, including: extracting multiple data entities from the business data, generating a node for each data entity, wherein the node is a data node or a job node; and establishing directed edges for multiple nodes corresponding to the multiple data entities according to the dependency relationship between the data and the data flow direction, wherein the directed edges are input edges or output edges.

[0052] In this embodiment, when constructing a directed graph based on business data, first, multiple data entities are extracted from the business data, and a node is generated for each data entity. The nodes are divided into data nodes or job nodes according to type. The node attributes of the data node may include the data table name, owner, Kafka topic name, number of partitions, etc., and the node attributes of the job node may include the job type, job name, owner, etc. Secondly, according to the dependency relationship between the data and the data flow direction, directed edges are established for multiple nodes corresponding to the multiple data entities, and the directed edges are input edges or output edges. For the same node, it can have both input edges and output edges, or only input edges, or only output edges.

[0053] In order to better illustrate the scheme in this embodiment, the steps of constructing a directed graph according to business data in this embodiment are specifically described by taking an example. For example: Job a reads data table 1, performs processing, and outputs the result of the operation to data table 2. According to the above content, it can be obtained that the directed graph includes 3 nodes and 2 edges, wherein the 3 nodes are: job a, data table 1, and data table 2; the 2 edges are: the edge from data table 1 to job a, and the edge from job a to data table 2. According to the above directed graph, it can be determined that the three nodes in the above example are divided into data nodes (i.e., data table 1 and data table 2) and job nodes (i.e., job a) according to their types. The edge from data table 1 to job a and the edge from job a to data table 2 are both directed edges, representing the dependency relationship between data and the data flow direction, respectively. For data table 1, it only has an output edge (the edge from data table 1 to job a), and for data table 2, it only has an input edge (the edge from job a to data table 2). For job a, it has both an input edge (the edge from data table 1 to job a) and an output edge (the edge from job a to data table 2).

[0054] It is understandable that by extracting data entities from business data and constructing directed graphs, accurate mapping of complex dependencies between data and data flows is achieved. By generating nodes for each data entity and establishing directed edges based on the dependencies and data flows between them, this method not only clearly reveals the logical chain of data processing, but also facilitates subsequent analysis and optimization.

[0055] In order to better analyze the business security risks of the business data of the target business, weights are assigned to the directed edges in the directed graph based on the directed graph.

[0056] As an optional embodiment, edge weights are assigned to directed edges according to the node weights of the nodes connected by the directed edges in the directed graph to obtain a directed weighted graph, including: obtaining the target node weight of the node pointed to by the target directed edge, wherein the target directed edge is any directed edge with undetermined edge weight; assigning the target node weight to the target directed edge to obtain the edge weight of the target directed edge.

[0057] In this embodiment, first, the target node weight of the node pointed to by any target directed edge with undetermined edge weight is obtained, and then the target node weight is assigned to the target directed edge to obtain the edge weight of the target directed edge. For example, if there is a directed edge a→b in a directed graph, the node weight of node a is 5, and the node weight of node b is 3, then the edge weight of the directed edge a→b is 3.

[0058] It can be understood that by directly assigning the weight of the node in the directed graph to the directed edge connecting the node, a directed weighted graph can be constructed quickly and easily. This method not only maintains the business importance information represented by the node weight, but also intuitively reflects the key path and the strength of the dependency relationship in the data flow process through the edge weight. This technical means simplifies the construction process of the weighted graph and improves efficiency. At the same time, it ensures that the weighted graph can accurately reflect the complexity and dynamics of business data, providing more accurate and comprehensive data support for subsequent business security risk analysis.

[0059] In specific practice, since the target business and the required business data are dynamically changing, the directed weighted graph corresponding to the business data of the target business is also dynamically updated. Use mainstream graph databases, such as JanusGraph, to implement the steps of building a directed weighted graph based on the business data of the target business. The specific steps of using mainstream graph databases to implement the construction of a directed weighted graph are no longer limited in this embodiment.

[0060] In the technical solution of step S202, all data links are identified from the directed weighted graph. In this embodiment, the data link specifically refers to a node in the directed weighted graph, and a sub-directed weighted graph consisting of all nodes that can be connected to the node through directed edges. In specific practice, a community discovery algorithm is used to identify all data links from the directed weighted graph. The community discovery algorithm aims to discover the community structure in the network, that is, the tightly connected groups between nodes in the network. The basic idea of ​​the community discovery algorithm is to divide the nodes in the network into different communities in some way, so that the nodes in the same community are closely connected, while the nodes between different communities are sparsely connected. The specific steps for the community discovery algorithm to identify data links are not specifically limited in this embodiment.

[0061] As an optional embodiment, after identifying all data links from the directed weighted graph, it also includes: analyzing the out-degree and in-degree of each node in each data link to obtain a target node whose out-degree is higher than a first preset value and / or whose in-degree is higher than a second preset value; and reallocating the business data corresponding to the target node.

[0062] In this embodiment, the in-degree is the number of edges pointing to the node, and the out-degree is the number of edges starting from the node. When the out-degree or in-degree value of one or more nodes in the directed weighted graph is abnormal, it means that the load of the node is abnormal. Therefore, the business data of these nodes can be reallocated. Specifically, the business data is allocated to the business processing corresponding to the nodes of the same type. In this embodiment, the abnormal value of the out-degree or in-degree of the node specifically refers to the degree being higher than the first preset value and / or the in-degree being higher than the second preset value. The first preset value and the second preset value are set according to the actual situation and may be the same or different.

[0063] It is understandable that after exhaustively identifying all data links from the directed weighted graph, the out-degree and in-degree of each node are further analyzed to identify those key target nodes whose out-degree is higher than the first preset value or whose in-degree is higher than the second preset value, and the business data of these nodes are intelligently redistributed accordingly. This technical means can optimize the distribution of network traffic, reduce the pressure of high-load nodes, and prevent the formation of potential bottlenecks, thereby enhancing the load balancing capability and response speed of the entire system and ensuring efficient and stable transmission of business data.

[0064] In the technical solution of step S203, all data links are analyzed for business security risks to obtain analysis results. In this embodiment, business security risks specifically refer to: 1. Core nodes where failures have a larger impact on downstream areas; 2. Weak links in the links, i.e., weak edges with higher security risks. By using the corresponding algorithm, the core nodes and weak links in the directed weighted graph are analyzed to obtain analysis results, which are used to point out the core nodes and weak edges in the current directed weighted graph.

[0065] As an optional embodiment, a business security risk analysis is performed on all data links to obtain analysis results, including: monitoring multiple target indicators of nodes in a directed weighted graph, and when any target indicator of the node is abnormal, the node is determined to be a faulty node, wherein the multiple target indicators include: a data flow indicator and a business execution status indicator; calling a core node identification algorithm to identify the core node in each data link, wherein the core node is the node with the largest connection weight value in the data link, and the connection weight is used to characterize the importance of the node in the data link; calling a weak edge identification algorithm to identify all weak edges in each data link, wherein a weak edge refers to an edge that is no longer connected to the data link after being deleted.

[0066] In this embodiment, multiple target indicators of nodes in a directed weighted graph can be monitored to implement flow monitoring of a data table or abnormal monitoring of a processing operation. When any target indicator of a node is abnormal, it is determined that the node is faulty and the node is a faulty node. Multiple target indicators include: data flow indicators and business execution status indicators. The data flow indicators include the QPS of the data flow of the data table and the rate of change of the flow during the same period (the same day and time of each week). The business execution status indicator refers to the business execution status. The target indicator is abnormal, that is, the QPS exceeds the upper and lower limits, or the flow change rate exceeds the threshold, or the execution status is abnormal (for example: execution failed). In specific practice, for a directed weighted graph, the data node only monitors the data flow indicator, and the business node monitors the data flow indicator and the business execution status indicator.

[0067] In this embodiment, a core node identification algorithm can be called to identify the core nodes in each data link. The core node identification algorithm is implemented based on the PageRank algorithm. The connection weight that characterizes the importance of the node in the data link is determined by weighting the edge weight of the directed edge in the directed weighted graph. The node with the largest connection weight in each data link is the core node of the data link. The specific implementation steps of the core node identification algorithm are not limited in this embodiment.

[0068] In this embodiment, a weak edge identification algorithm may also be called to identify all weak edges in each data link. A weak edge refers to an edge whose data link is no longer connected after being deleted. The weak edge identification algorithm is implemented based on the Tarjan algorithm. The specific implementation steps of the weak edge identification algorithm are not limited in this embodiment.

[0069] In this embodiment, the identified core nodes can be monitored intensively to prevent them from failing. The identified weak edges can be reinforced, such as HA reinforcement. HA reinforcement mainly refers to improving the reliability and availability of the system or network through a series of technologies and strategies to ensure that the system can continue to provide services when some components fail, thereby reducing or avoiding service interruptions.

[0070] It is understandable that by comprehensively monitoring node indicators and accurately identifying core nodes and weak edges, in-depth mining and analysis of potential business security risks in data links have been achieved. By real-time monitoring of node data traffic and business execution status, faulty nodes can be discovered and located in a timely manner to avoid the spread of security risks. At the same time, through the core node identification algorithm, key nodes in the data link are accurately locked. These nodes are crucial to business continuity and provide a clear direction for priority protection and optimization. In addition, the identification of weak edges reveals the vulnerable links in the data link, which helps to take measures in advance to strengthen it and prevent potential risks. This series of technical means work together to significantly improve the security and stability of business data links, providing a strong guarantee for the safe operation of the business.

[0071] As an optional embodiment, after analyzing all data links and obtaining the analysis results, it includes: when it is determined that there are multiple faulty nodes in the directed weighted graph, all nodes in the data link where each faulty node is located are sorted to determine the order in which the faulty nodes are recovered.

[0072] In this embodiment, a topological sorting algorithm is used to sort all nodes in the data link where each faulty node is located to determine the order in which the faulty nodes are restored. When multiple nodes fail, the faulty nodes that are at the front of the sorting sequence are restored first, which minimizes the overall impact.

[0073] It is understandable that after comprehensively analyzing all data links and identifying multiple faulty nodes, technical means are used to scientifically sort all nodes in the data links associated with each faulty node, so as to accurately determine the priority of fault recovery. This method can significantly improve the efficiency and accuracy of fault handling, ensure the rapid recovery of key business links, minimize service interruption time and the impact on overall system performance, and thus enhance system stability and user satisfaction.

[0074] In actual operation and maintenance, the analysis methods in the above steps can be applied in combination to form a comprehensive system analysis and optimization plan. For example, first identify the weak links, then determine which nodes are core nodes, and finally plan the fault recovery strategy through topological sorting.

[0075] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0077] According to another aspect of the embodiment of the present application, a data analysis device for implementing the above data analysis method is also provided. Figure 3 , Figure 3 is a structural block diagram of an optional data analysis device provided in an embodiment of the present application, such as Figure 3 As shown, the device may include:

[0078] The data acquisition module 301 is used to acquire a directed weighted graph corresponding to the business data of the target business, wherein the directed weighted graph includes a plurality of nodes corresponding to the data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency relationship and data flow direction between the data of the two nodes connected by the directed weighted edge;

[0079] A link identification module 302, used to identify all data links from the directed weighted graph;

[0080] The link analysis module 303 is used to analyze the business security risks of all data links and obtain analysis results.

[0081] It should be noted that the data acquisition module 301 in this embodiment can be used to execute the above step S201, the link identification module 302 in this embodiment can be used to execute the above step S202, and the link analysis module 303 in this embodiment can be used to execute the above step S203.

[0082] Regarding the data analysis device in this embodiment, the specific manner in which its data acquisition module 301, link identification module 302 and link analysis module 303 execute the above-mentioned data analysis method has been described in detail in the embodiment of the data analysis method, and will not be elaborated here.

[0083] It can be understood that the technical solution provided by this embodiment, each module in the data analysis device, constructs a directed weighted graph corresponding to the business data of the target business, and identifies all data links on this basis, and then conducts business security risk analysis, thereby achieving efficient capture and processing of complex dependencies and dynamic changes. This method not only intuitively displays the interdependence and data flow between data entities, but also accurately quantifies the importance of the business through the weight of the edge, providing a comprehensive and in-depth perspective for business security risk analysis. Compared with the existing technology, it can more accurately identify potential security risk points, issue warnings and take measures in a timely manner, thereby effectively improving the security and stability of business data, and solving the technical problem that traditional methods are difficult to deal with complex dependencies and dynamic changes.

[0084] The device in this embodiment, in addition to the above modules, may also include a module for executing any method in any of the above data analysis method embodiments.

[0085] It should be noted that the above modules and corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can be run in the implementation of Figure 1The hardware environment of the method shown may be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0086] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned data analysis method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0087] According to another embodiment of the present application, an electronic device is also provided. Figure 4 , Figure 4 is a structural block diagram of an optional electronic device provided in an embodiment of the present application, such as Figure 4 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .

[0088] Memory 1503, used for storing computer programs;

[0089] The processor 1501 is used to implement the following steps when executing the program stored in the memory 1503:

[0090] Step S201, obtaining a directed weighted graph corresponding to the business data of the target business, wherein the directed weighted graph includes a plurality of nodes corresponding to the data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency relationship and data flow direction between the data of the two nodes connected by the directed weighted edge;

[0091] Step S202, identifying all data links from the directed weighted graph;

[0092] Step S203: Perform business security risk analysis on all data links to obtain analysis results.

[0093] It can be understood that the technical solution provided in this embodiment, the processor of the electronic device, by constructing a directed weighted graph corresponding to the business data of the target business, and on this basis identifying all data links, and then performing business security risk analysis, realizes efficient capture and processing of complex dependencies and dynamic changes. This method not only intuitively displays the interdependence and data flow between data entities, but also accurately quantifies the importance of the business through the weight of the edge, providing a comprehensive and in-depth perspective for business security risk analysis. Compared with the existing technology, it can more accurately identify potential security risk points, issue warnings in a timely manner and take measures, thereby effectively improving the security and stability of business data, and solving the technical problem that traditional methods are difficult to deal with complex dependencies and dynamic changes.

[0094] Optionally, in this embodiment, the above-mentioned communication bus can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0095] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0096] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0097] An embodiment of the present application further provides a computer-readable storage medium, the storage medium including a stored program, wherein the method steps of the above method embodiment are executed when the program is run.

[0098] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0099] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0100] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods of each embodiment of the present application.

[0101] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data analysis method, characterized in that: include: Obtaining a directed weighted graph corresponding to the business data of the target business, wherein the directed weighted graph includes a plurality of nodes corresponding to the data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency relationship and data flow direction between the data of the two nodes connected by the directed weighted edge; identifying all data links from the directed weighted graph; Performing a business security risk analysis on all the data links to obtain analysis results.

2. The data analysis method according to claim 1, characterized in that: Obtain a directed weighted graph corresponding to the target business data of the target business, including: Constructing a directed graph according to the business data, wherein the directed graph includes a plurality of nodes and directed edges between the nodes; Assigning a node weight to each node in the directed graph according to a preset weight assignment rule; According to the node weights of the nodes connected by the directed edges in the directed graph, edge weights are assigned to the directed edges to obtain the directed weighted graph.

3. The data analysis method according to claim 2, characterized in that: The step of constructing a directed graph according to the business data includes: Extracting multiple data entities from the business data, and generating a node for each data entity, wherein the node is a data node or a job node; According to the dependency relationship between the data and the data flow direction, directed edges are established for the multiple nodes corresponding to the multiple data entities, wherein the directed edges are input edges or output edges.

4. The data analysis method according to claim 2, characterized in that: The step of assigning edge weights to the directed edges according to the node weights of the nodes connected to the directed edges in the directed graph to obtain the directed weighted graph comprises: Obtaining a target node weight of the node pointed to by a target directed edge, wherein the target directed edge is any directed edge with an undetermined edge weight; The target node weight is assigned to the target directed edge to obtain the edge weight of the target directed edge.

5. The data analysis method according to claim 1, characterized in that: The performing of business security risk analysis on all the data links to obtain analysis results includes: Monitoring multiple target indicators of the node in the directed weighted graph, when any target indicator of the node is abnormal, determining the node as a faulty node, wherein the multiple target indicators include: a data flow indicator and a service execution status indicator; Calling a core node identification algorithm to identify the core node in each data link, wherein the core node is the node with the largest connection weight value in the data link, and the connection weight is used to characterize the importance of the node in the data link; A weak edge identification algorithm is called to identify all weak edges in each data link, wherein the weak edge refers to an edge whose data link is no longer connected after being deleted.

6. The data analysis method according to claim 1, characterized in that: After analyzing all the data links and obtaining the analysis results, the following steps are performed: When it is determined that there are multiple faulty nodes in the directed weighted graph, all nodes in the data link where each faulty node is located are sorted to determine the order in which the faulty nodes are restored.

7. The data analysis method according to claim 1, characterized in that: After all data links are identified from the directed weighted graph, the method further includes: Analyze the out-degree and in-degree of each node in each data link to obtain a target node whose out-degree is higher than a first preset value and / or whose in-degree is higher than a second preset value; The service data corresponding to the target node is reallocated.

8. A data analysis device, characterized in that: include: A data acquisition module, used to acquire a directed weighted graph corresponding to the business data of the target business, wherein the directed weighted graph includes a plurality of nodes corresponding to the data entities of the business data, and a directed weighted edge connecting two nodes, the weight of the directed weighted edge is used to characterize the importance of the target business, and the direction of the directed weighted edge is used to characterize the dependency relationship and data flow direction between the data of the two nodes connected by the directed weighted edge; A link identification module, used to identify all data links from the directed weighted graph; The link analysis module is used to analyze the business security risks of all the data links and obtain analysis results.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is used to execute the data analysis method according to any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the data analysis method according to any one of claims 1 to 7 when executed.

Images