Data flow link acquisition method, system and device, and medium

Abstract

The invention relates to a method, a system, equipment and a medium for obtaining a data flow link, and the method comprises the steps: obtaining a business flow link diagram and a deployment architecture diagram, confirming a micro-service tangency point of a business application, deploying a data collection probe in the form of an SDK assembly, and obtaining business data flow information; labeling a link label for a service operation corresponding to the service data flow information, and obtaining a data flow path by taking a service account as a dimension; creating a business domain knowledge base and a behavior judgment model; establishing a business behavior portrait, recognizing an abnormal operation by using the behavior judgment model and a preset illegal behavior recognition strategy, comparing the abnormal operation with the business behavior portrait corresponding to the business account, and performing risk assessment and early warning according to a comparison result; according to the method, multi-dimensional tangency point data acquisition is realized, a data flow link is completely restored and presented, abnormal events, illegal operation and other risks in a data processing flow are identified, early warning and management and control measures are effectively executed, and the security and compliance of data flow are improved.

Description

Technical Field

[0001] The present invention relates to the field of data flow links, and mainly to a method, system, device and medium for acquiring a data flow link. Background Art

[0002] The traditional data flow link restoration method can only restore the data flow path on the business system side and the data flow path on the database service side separately, and cannot associate the two. At the same time, the variety of business applications leads to a large number of data sources, which makes it difficult to uniformly process and associate them. The traditional user behavior analysis method builds a business behavior profile and compares the access behavior with the behavior characteristics to identify abnormal operations or illegal operations. It is not associated with the business scenario, resulting in inaccurate identification results and a high false alarm rate. In view of the multi-source, complex business logic aspects, difficult point-of-entry implantation, inaccurate data flow link restoration, incomplete security monitoring coverage, and untimely measures in the power business data fusion sharing and big data analysis applications, it is necessary to design and implement multi-dimensional point-of-entry data collection, completely restore and present the data flow link, intelligently identify abnormal events, illegal operations and other risks in the data processing process, effectively implement early warning and control measures, and improve the security and compliance of data flow. The data flow link acquisition method and device are very necessary.

[0003] For example, CN112035317A "A method, device, equipment and medium for monitoring a microservice link" discloses "a method, by creating a target probe and a target monitoring log format; based on the target probe and according to the target monitoring log format, collecting log data corresponding to the dubbo call of the target microservice link; obtaining the log data from the target probe through a data collector and sending the log data to a search server, so that the search server monitors and analyzes the target microservice link; this application creates a target probe, uses the target probe and collects log data corresponding to the target microservice link according to the target monitoring log format, and then the data collector sends the log data to the search server, and the search server monitors and analyzes the target microservice link", but the data collection scope of the method is limited, mainly relying on the target probe and the preset monitoring log format, resulting in insufficient comprehensive coverage of microservice calls, lack of in-depth analysis of business operations, and inability to effectively evaluate business flow paths and logic; in addition, the data collector of the method may reduce the real-time and accuracy of monitoring data due to the large amount of data. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present application provides a method, system, device and medium for obtaining a data flow link.

[0005] The technical solution of this application is as follows:

[0006] A method for acquiring a data flow link, the method comprising:

[0007] Obtain the business flow link diagram and deployment architecture diagram and confirm the microservice cut-off points of the business application. Deploy data collection probes in the form of SDK components at each microservice cut-off point to obtain business data flow information.

[0008] The business operations corresponding to the business data flow information are marked with link tags, which include tracking tags, sequence tags and hierarchical tags. The operation information with the same tracking tags is integrated based on the business account, and the calling sequence and hierarchical relationship of the data processing flow are restored according to the sequence tags and hierarchical tags to obtain the data flow path.

[0009] Creating a business domain knowledge base based on the professional knowledge of business personnel and the data flow path, and creating a behavior determination model, training and optimizing the behavior determination model based on the business domain knowledge base, and obtaining a trained behavior determination model;

[0010] A business behavior profile is established for each business account, and the business behavior profile is dynamically adjusted according to the access behavior information; the behavior determination model and the preset violation identification strategy are used to identify abnormal operations of the business account in business operations, and the abnormal operations are compared with the business behavior profile corresponding to the business account, and risk assessment and early warning are performed based on the comparison results.

[0011] Preferably, the method further comprises data cleaning, wherein the data cleaning comprises processing missing values, outliers and standardizing data formats.

[0012] Preferably, the business data flow information includes business application services, middleware services, component information, database services, interface URL addresses, access accounts, access device IP addresses and access time.

[0013] Preferably, a link label is marked on the business operation corresponding to the business data flow information through a custom link label generator, and the link label generator includes a TraceContext class, a TraceIdGenerator class, a SpanIdGenerator class, and a TraceContextManager class, wherein:

[0014] The TraceContext class is used to store the current tracing context information, including the tracing tag TraceID, the sequential tag Span ID and the hierarchical tag Parent Span ID; the TraceIdGenerator class is used to generate a unique string Trace ID based on a timestamp and a random number generation algorithm; the SpanIdGenerator class uses an incremental counter to independently generate a Span ID in each thread or request context; the TraceContextManager class calls TraceIdGenerator and SpanIdGenerator to generate a Parent Span ID at the beginning of each business operation, caches the Parent Span ID through the distributed cache Redis and attaches it to the corresponding TraceContext.

[0015] Preferably, the behavior determination model includes a feature extraction layer, a backbone network layer, a quantization selection layer and a recognition output layer, wherein:

[0016] The feature extraction layer includes two combined convolution blocks and a pooling layer, each of the convolution blocks includes a 1X3 convolution layer, a 3X1 convolution layer and a normalization layer, and the business data flow information is used as the input feature of the feature extraction layer, the output results of the convolution blocks are spliced, and the feature extraction results are output after global average pooling;

[0017] The backbone network layer includes a first branch and a second branch, wherein the first branch is used for processing business data flow features, including 4 shuffle blocks of different dimensions and 6 residual blocks of different dimensions, wherein each shuffle block includes a 1x1 upsampling group convolution layer, a 1x1 downsampling group convolution layer, 2 3x3 depth separation convolutions, 1 3x3 convolution layer, 3 normalization layers, 1 3x3 average pooling layer and a RELU activation function, and each residual block contains multiple convolution layers and normalization layers; the second branch is used for processing business operation timing features, including a bidirectional LSTM network and a connector, wherein the forward LSTM of the bidirectional LSTM network is used for forward processing of timing features, and the reverse LSTM is used for reverse processing of timing features, and the output results of the forward LSTM and the reverse LSTM are spliced ​​through the connector to obtain the final timing depth feature value;

[0018] The feedback optimization function y of the quantization selection layer is expressed as:

[0019]

[0020] In the formula, K is the number of network groups, k is the loop variable, σ is the number of backbone network layers, and E σ(k) is the optimization rate of the kth group of networks in the σth layer of the backbone network, E0 is the maximum optimization efficiency of the preset backbone network, ρ is the adaptive quantization parameter, B σ (k) is the loss function value of the k-th group of networks in the σ-th layer backbone network, μ1 and μ2 are adjustment parameters;

[0021] Adjust the update strategy of the behavior determination model according to the feedback optimization function;

[0022] The recognition output layer includes an average pooling layer and a fully connected layer, and the activation function is a Tanh activation function and a PReLU activation function, which are used to output the final judgment result.

[0023] Preferably, a business behavior profile is established for each business account, specifically:

[0024] Extract characteristic data of user characteristics and behavior habits, and establish a business behavior portrait for each access account based on the characteristic data and with the business access account as the dimension; use the BERT sentiment analysis model to perform sentiment classification on the characteristic data to obtain the access preference of the business access account; the access preference is divided into positive, negative and neutral; obtain multi-dimensional information of each business behavior portrait based on the characteristic data and access preference, and the multi-dimensional information includes business access frequency, number of visits, access preference and business feature label;

[0025] A MySQL database is designed, in which each table corresponds to a multi-dimensional information, and the multi-dimensional information of the business behavior portrait is input into the MySQL database to obtain a business behavior portrait database.

[0026] Preferably, the illegal behavior identification strategy includes identifying access frequency, detecting access devices, detecting operation time, and business behavior profile update times, specifically:

[0027] Collect all operation data of business accounts in real time, including time, location, equipment, data flow rate, business operation behavior and business operation path;

[0028] The preset access frequency threshold. If the number of accesses to the same resource by the current business account in a short period of time reaches this preset access frequency threshold, it is considered abnormal behavior.

[0029] Detect access devices and login locations. If the login device of the current business account is not the same as the previous login device, and the login location of the current business account is not the same as the previous login location, it is considered abnormal behavior.

[0030] Set a specific operation time window for a specific business operation. If the login time of the current business account is not within the specific operation time window, it will be considered as abnormal behavior.

[0031] A threshold for the number of business behavior profile updates is preset. If the number of times the current business account updates its business behavior profile exceeds the threshold for the number of business behavior profile updates within a preset time range, it is judged as abnormal behavior.

[0032] A data flow link acquisition system, the system includes a data acquisition module, a label marking module and an anomaly detection module:

[0033] The data acquisition module is used to obtain the business flow link diagram and deployment architecture diagram and confirm the microservice cut-off point of the business application, deploy data collection probes in the form of SDK components at each microservice cut-off point to obtain business data flow information; and transmit the business data flow information to the label marking module;

[0034] The label labeling module is used to label the business operations corresponding to the business data flow information with link labels, wherein the link labels include tracking labels, sequence labels and hierarchical labels. The operation information of the same tracking labels is integrated based on the business account, and the calling sequence and hierarchical relationship of the data processing flow are restored according to the sequence labels and hierarchical labels to obtain the data flow path;

[0035] The anomaly detection module is used to use the data flow path of the business account as the input of the trained behavior judgment model to obtain a judgment result; based on the judgment result and the preset violation behavior identification strategy, identify the abnormal operations of the business account in the business operation, compare the abnormal operations with the business behavior portrait corresponding to the business account, and perform risk assessment and early warning based on the comparison results.

[0036] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for acquiring a data flow link as described in any one of the embodiments is implemented.

[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements a method for obtaining a data flow link as described in any one of the embodiments.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1) The present invention provides a method, system, device and medium for acquiring a data flow link. By deploying a data collection probe in the form of an SDK component at each microservice cut-off point, it is possible to fully acquire business flow information, thereby improving the transparency of business processes, enhancing the optimization capability of the system architecture, and ensuring the stability and maintainability of the overall system.

[0040] 2) The present invention provides a method, system, device and medium for acquiring a data flow link, establishes a dynamic behavior profile of a business account, and adjusts it according to real-time access behavior, thereby improving the accuracy of the system in identifying risks; in addition, combined with the behavior judgment model obtained through training, it improves the ability to grasp the user's operation mode in real time, ensuring the accuracy of risk warning;

[0041] 3) The present invention provides a method, system, device and medium for acquiring data flow links. Combined with the professional knowledge of business personnel, a business domain knowledge base is created to achieve a comprehensive risk assessment of abnormal operations, improve the accuracy and comprehensiveness of risk assessment, enhance the understanding of business scenarios, and improve decision-making capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a method flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0044] The present invention provides the following technical solution: a method, system, device and medium for acquiring a data flow link.

[0045] Example 1

[0046] This embodiment provides a method for obtaining a data flow link, and the specific steps include:

[0047] S1. Obtain the business flow link diagram and deployment architecture diagram and confirm the microservice cut-points of the business application. Deploy data collection probes in the form of SDK components at each microservice cut-point to obtain business data flow information.

[0048] S11, the business data flow information includes business application services, middleware services, component information, database services, interface URL addresses, access accounts, access device IP addresses and access times, and the data flow rate is calculated by the amount of collected data and the time window to perform data flow monitoring and data decoupling analysis;

[0049] The data flow monitoring includes monitoring the speed, direction and amount of data flow, and calculating the data flow rate of a window within a certain period of time, which is expressed as follows:

[0050]

[0051] Where R represents the data transfer rate; V i represents the amount of data in the i-th data flow event; n represents the number of data flow events in the window within the ΔT time; ΔT represents the time interval;

[0052] The data decoupling analysis specifically analyzes the processing time of data flowing from the source to the destination, which can be expressed as:

[0053]

[0054] Where, T p Indicates processing time; Indicates the start time of the i-th data flow event; Indicates the start time of the i-th data flow event;

[0055] S12. SDK (Software Development Kit) components are a comprehensive tool set. Data collection probes are deployed in the form of SDK components. Probes can be added or removed at will, which is highly flexible and scalable. At the same time, SDK components usually have efficient data collection and processing capabilities, and can quickly transfer the collected data to the back-end server for storage and analysis. In addition, SDK components usually have standardized interfaces and configuration options, which can reduce maintenance costs, enhance business insights, and promote cross-departmental collaboration.

[0056] S2. Label the business operation corresponding to the business data flow information with a link label, wherein the link label includes a tracking label, a sequence label, and a level label;

[0057] S21. Tracking tags are used to record the entire process of data flow in the system. They contain unique identifiers and related metadata, and can track the source, destination, and change history of data.

[0058] S22. Sequence tags are used to identify the order of data processing or transmission, and contain serial numbers or timestamps, which can indicate the order in which data is processed;

[0059] S23. Hierarchical tags are used to define the hierarchical relationship or structure between data, including parent ID and child ID, which can enhance the readability and manageability of data;

[0060] S3. Label the business operations corresponding to the business data flow information with link labels through a custom link label generator, wherein the link label generator includes a TraceContext class, a TraceIdGenerator class, a SpanIdGenerator class, and a TraceContextManager class, wherein:

[0061] The TraceContext class is used to store the current tracing context information, including the tracing tag TraceID, the sequence tag Span ID and the hierarchical tag Parent Span ID; the TraceIdGenerator class is used to generate a unique string Trace ID according to the timestamp and random number generation algorithm; the SpanIdGenerator class uses an incremental counter to independently generate a Span ID in each thread or request context; the TraceContextManager class calls TraceIdGenerator and SpanIdGenerator to generate a Parent Span ID at the beginning of each business operation, caches the Parent Span ID through the distributed cache Redis and attaches it to the corresponding TraceContext;

[0062] S4. Use the SVM model to identify and complete the missing link labels; call the TraceContext class to obtain all historical link labels and perform data processing; extract the feature information of the historical link labels, the feature information includes node attributes, link attributes and timestamps, and mark the missing link labels; construct a link label data set based on the feature information and all historical link labels;

[0063] Construct an SVM model, which includes a linear kernel function, model parameters and an Adam optimizer; the linear kernel function f is expressed as follows:

[0064] f = x T y;

[0065] In the formula, x and y represent the data points in the link label dataset, x T y represents the product of the transpose of x and y;

[0066] Use the historical link label data set to train the SVM model to obtain a trained SVM model; obtain real-time link labels, extract feature information of the real-time link label data and input it into the trained SVM model, identify missing link labels, and output the prediction results of the missing link labels;

[0067] S5. The data collection probe uses the link label generator to add tracking labels, sequence labels, and hierarchical labels to the business operations of each microservice in the business application. Based on the business account, the operation information with the same tracking label is integrated, and the calling sequence and hierarchical relationship of the data processing flow are restored according to the sequence label and hierarchical label to obtain a complete data flow path.

[0068] S6. Create a business domain knowledge base based on the professional knowledge of business personnel and the data flow path. Build a business domain knowledge base based on the MySQL database. Business personnel analyze the data flow path, organize the professional knowledge of related business processes, operating specifications, and risk solutions, and obtain a professional knowledge set.

[0069] Classify the professional knowledge set according to preset types, each type corresponds to a table structure in the MySQL database; enter the professional knowledge set into the MySQL database according to the type, and generate a unique identifier for each piece of professional knowledge;

[0070] Business personnel collect the latest developments in the business field and regularly update the business field knowledge base;

[0071] S7, creating a behavior determination model, wherein the behavior determination model includes a feature extraction layer, a backbone network layer, a quantization selection layer, and a recognition output layer;

[0072] The feature extraction layer includes two combined convolution blocks and a pooling layer, each of the convolution blocks includes a 1X3 convolution layer, a 3X1 convolution layer and a normalization layer, and the business data flow information is used as the input feature of the feature extraction layer, the output results of the convolution blocks are spliced, and the feature extraction results are output after global average pooling;

[0073] The backbone network layer includes a first branch and a second branch, wherein the first branch is used for processing business data flow features, including 4 shuffle blocks of different dimensions and 6 residual blocks of different dimensions, wherein each shuffle block includes a 1x1 upsampling group convolution layer, a 1x1 downsampling group convolution layer, 2 3x3 depth separation convolutions, 1 3x3 convolution layer, 3 normalization layers, 1 3x3 average pooling layer and a RELU activation function, and each residual block contains multiple convolution layers and normalization layers; the second branch is used for processing business operation timing features, including a bidirectional LSTM network and a connector, wherein the forward LSTM of the bidirectional LSTM network is used for forward processing of timing features, and the reverse LSTM is used for reverse processing of timing features, and the output results of the forward LSTM and the reverse LSTM are spliced ​​through the connector to obtain the final timing depth feature value;

[0074] The feedback optimization function y of the quantization selection layer is expressed as:

[0075]

[0076] In the formula, K is the number of network groups, k is the loop variable, σ is the number of backbone network layers, and E σ (k) is the optimization rate of the kth group of networks in the σth layer of the backbone network, E0 is the maximum optimization efficiency of the preset backbone network, ρ is the adaptive quantization parameter, B σ(k) is the loss function value of the k-th group of networks in the σ-th layer backbone network, μ1 and μ2 are adjustment parameters;

[0077] Adjust the update strategy of the behavior determination model according to the feedback optimization function;

[0078] The recognition output layer includes an average pooling layer and a fully connected layer, and the activation function is a Tanh activation function and a PReLU activation function, which are used to output the final judgment result;

[0079] Acquire historical business behavior data as a business behavior data set, and use the business behavior data set to train the behavior determination model to obtain a trained behavior determination model;

[0080] S8. Establish a business behavior profile for each business account, and dynamically adjust the business behavior profile according to the access behavior information, extract feature data of user characteristics and behavior habits, and establish a business behavior profile for each access account based on the feature data and the business access account as a dimension; use the BERT sentiment analysis model to perform sentiment classification on the feature data to obtain the access preference of the business access account; the access preference is divided into positive, negative and neutral; based on the feature data and the access preference, obtain multi-dimensional information of each business behavior profile, the multi-dimensional information includes business access frequency, number of visits, access preference and business feature label;

[0081] Design a MySQL database, in which each table corresponds to a multi-dimensional information, input the multi-dimensional information of the business behavior portrait into the MySQL database, and obtain a business behavior portrait database;

[0082] Setting up a monitoring mechanism to dynamically adjust the business behavior profile based on access behavior information when user characteristics and behavior habits change significantly;

[0083] In this embodiment, the BERT sentiment analysis model and monitoring mechanism are prior art, so they are not described in detail.

[0084] S9. Using the trained behavior determination model and the preset illegal behavior identification strategy to identify abnormal operations of the business account in business operations, comparing the abnormal operations with the business behavior profile corresponding to the business account, and performing risk assessment and early warning based on the comparison results;

[0085] The violation identification strategy includes identifying access frequency, detecting access devices, detecting operation time and business behavior profile update times; real-time collection of all operation data of business accounts, including time, location, device, data flow rate, business operation behavior and business operation path;

[0086] The preset access frequency threshold. If the number of accesses to the same resource by the current business account in a short period of time reaches this preset access frequency threshold, it is considered abnormal behavior.

[0087] Detect access devices and login locations. If the login device of the current business account is not the same as the previous login device, and the login location of the current business account is not the same as the previous login location, it is considered abnormal behavior.

[0088] Set a specific operation time window for a specific business operation. If the login time of the current business account is not within the specific operation time window, it will be considered as abnormal behavior.

[0089] A threshold for the number of business behavior profile updates is preset. If the number of times the current business account updates its business behavior profile exceeds the threshold within a preset time range, it is considered abnormal behavior.

[0090] Compare the abnormal behavior with the business behavior profile corresponding to the business account, and classify it into different levels of risk based on the comparison results; trigger different response mechanisms based on the risk level, send warnings to the management, force secondary identity verification, or automatically freeze the account;

[0091] S10. In order to verify the effectiveness of this embodiment, scenario verification can be performed in combination with the business system, including a point-of-cut implantation technology verification solution, a data introspection technology verification solution, and a business behavior analysis technology verification solution;

[0092] S1001, the cutting point implantation technology verification scheme;

[0093] According to the business flow link diagram of the marketing 2.0 business system, select the core business cut-off points, which include the charging account integration interface microservice, pfcuc data microservice, pfc data microservice and payment microservice;

[0094] In each microservice, a data collection probe is installed in the form of an SDK component. The probe collects data about the program running process in the microservice from the memory level to obtain relevant operation information about electricity bill payment.

[0095] After the cut-points are implanted, the execution logic of the electricity bill ordering, payment completion, payment result notification, and account closing business scenarios is analyzed to form matching rules; the data collection probe collects data according to the rules;

[0096] S1002, the data introspection technology verification solution;

[0097] The data collection probe collects the time when the marketing 2.0 business account enters the business connection platform and the business application calls the business connection platform microservice, and adds a tracking tag, sequence tag, and level tag to this operation;

[0098] Taking the Marketing 2.0 business account as the dimension, according to the tracking tag of the electricity bill payment business scenario, retrieve the data flow information with the same tracking tag in the sub-business scenarios of electricity bill order placement, payment completion, payment result notification and account cancellation, and restore the data flow path according to the calling sequence and hierarchical tags;

[0099] S1003, the business behavior analysis technology verification solution;

[0100] Build a business domain knowledge base for Marketing 2.0 business applications, and have business personnel assist in configuring business account parameters;

[0101] Marketing 2.0 business personnel assist in configuring the violation identification strategy and identify the violations that occur in the electricity bill payment business scenario based on the associated information in the business domain knowledge base;

[0102] Based on the dimensions of the Marketing 2.0 business application account and the data processing flow information, the business account behavior characteristics and behavior patterns in the electricity bill ordering, payment completion, payment result notification and account closing business scenarios are converted into business behavior portraits;

[0103] Based on the calling sequence and hierarchical relationship of microservices in the electricity bill ordering, payment completion, payment result notification, and account settlement sub-business scenarios, each operation is compared.

[0104] Example 2

[0105] This embodiment provides a data flow link acquisition system, which includes a data acquisition module, a label marking module and an anomaly detection module:

[0106] The data acquisition module is used to obtain the business flow link diagram and deployment architecture diagram and confirm the microservice cut-off point of the business application, deploy data collection probes in the form of SDK components at each microservice cut-off point to obtain business data flow information; and transmit the business data flow information to the label marking module;

[0107] The label labeling module is used to label the business operations corresponding to the business data flow information with link labels, wherein the link labels include tracking labels, sequence labels and hierarchical labels. The operation information of the same tracking labels is integrated based on the business account, and the calling sequence and hierarchical relationship of the data processing flow are restored according to the sequence labels and hierarchical labels to obtain the data flow path;

[0108] The anomaly detection module is used to use the data flow path of the business account as the input of the trained behavior judgment model to obtain a judgment result; based on the judgment result and the preset violation behavior identification strategy, identify the abnormal operations of the business account in the business operation, compare the abnormal operations with the business behavior portrait corresponding to the business account, and perform risk assessment and early warning based on the comparison results.

[0109] Example 3

[0110] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for acquiring a data flow link as described in any one of Embodiment 1 is implemented.

[0111] Example 4

[0112] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, a method for obtaining a data flow link as described in any one of Embodiment 1 is implemented.

[0113] It is worth noting that the system, electronic device and computer-readable storage medium described in the present invention are based on the same principles as the method described in Example 1, and will not be described in detail here.

[0114] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for acquiring a data flow link, characterized in that: The method comprises: Obtain the business flow link diagram and deployment architecture diagram and confirm the microservice cut-off points of the business application. Deploy data collection probes in the form of SDK components at each microservice cut-off point to obtain business data flow information. The business operations corresponding to the business data flow information are marked with link tags, which include tracking tags, sequence tags and hierarchical tags. The operation information with the same tracking tags is integrated based on the business account, and the calling sequence and hierarchical relationship of the data processing flow are restored according to the sequence tags and hierarchical tags to obtain the data flow path. Creating a business domain knowledge base based on the professional knowledge of business personnel and the data flow path, and creating a behavior determination model, training and optimizing the behavior determination model based on the business domain knowledge base, and obtaining a trained behavior determination model; A business behavior profile is established for each business account, and the business behavior profile is dynamically adjusted according to the access behavior information. The trained behavior judgment model and the preset violation identification strategy are used to identify abnormal operations of the business account in business operations, and the abnormal operations are compared with the business behavior profile corresponding to the business account. Risk assessment and early warning are performed based on the comparison results.

2. A method for acquiring a data flow link according to claim 1, characterized in that: The method also includes data cleaning, which includes processing missing values, outliers and data format unification.

3. A method for acquiring a data flow link according to claim 2, characterized in that: Business data flow information includes business application services, middleware services, component information, database services, interface URL addresses, access accounts, access device IP addresses and access time.

4. A method for acquiring a data flow link according to claim 3, characterized in that: The business operations corresponding to the business data flow information are annotated with link labels through a custom link label generator, wherein the link label generator includes the TraceContext class, the TraceIdGenerator class, the SpanIdGenerator class, and the TraceContextManager class, wherein: The TraceContext class is used to store the current tracing context information, including the tracing tag Trace ID, the sequential tag Span ID and the hierarchical tag Parent Span ID; the TraceIdGenerator class is used to generate a unique string Trace ID based on a timestamp and a random number generation algorithm; the SpanIdGenerator class uses an incremental counter to independently generate a Span ID in each thread or request context; the TraceContextManager class calls TraceIdGenerator and SpanIdGenerator to generate a Parent Span ID at the beginning of each business operation, caches the Parent Span ID through the distributed cache Redis and attaches it to the corresponding TraceContext.

5. A method for acquiring a data flow link according to claim 4, characterized in that: The behavior determination model includes feature extraction layer, backbone network layer, quantization selection layer and recognition output layer, where: The feature extraction layer includes two combined convolution blocks and a pooling layer, each of the convolution blocks includes a 1X3 convolution layer, a 3X1 convolution layer and a normalization layer, and the business data flow information is used as the input feature of the feature extraction layer, the output results of the convolution blocks are spliced, and the feature extraction results are output after global average pooling; The backbone network layer includes a first branch and a second branch, wherein the first branch is used for processing business data flow features, including 4 shuffle blocks of different dimensions and 6 residual blocks of different dimensions, wherein each shuffle block includes a 1x1 upsampling group convolution layer, a 1x1 downsampling group convolution layer, 2 3x3 depth separation convolutions, 1 3x3 convolution layer, 3 normalization layers, 1 3x3 average pooling layer and a RELU activation function, and each residual block contains multiple convolution layers and normalization layers; the second branch is used for processing business operation timing features, including a bidirectional LSTM network and a connector, wherein the forward LSTM of the bidirectional LSTM network is used for forward processing of timing features, and the reverse LSTM is used for reverse processing of timing features, and the output results of the forward LSTM and the reverse LSTM are spliced ​​through the connector to obtain the final timing depth feature value; The feedback optimization function y of the quantization selection layer is expressed as: In the formula, K is the number of network groups, k is the loop variable, σ is the number of backbone network layers, and E σ (k) is the optimization rate of the kth group of networks in the σth layer of the backbone network, E0 is the maximum optimization efficiency of the preset backbone network, ρ is the adaptive quantization parameter, B σ (k) is the loss function value of the k-th group of networks in the σ-th layer backbone network, μ1 and μ2 are adjustment parameters; Adjust the update strategy of the behavior determination model according to the feedback optimization function; The recognition output layer includes an average pooling layer and a fully connected layer, and the activation function is a Tanh activation function and a PReLU activation function, which are used to output the final judgment result.

6. A method for acquiring a data flow link according to claim 5, characterized in that: Create a business behavior profile for each business account, specifically: Extract characteristic data of user characteristics and behavior habits, and build a business behavior profile for each access account based on the characteristic data and with the business access account as the dimension; use the BERT sentiment analysis model to perform sentiment classification on the characteristic data to obtain the access preference of the business access account; the access preference is divided into positive, negative and neutral; According to the characteristic data and access preferences, multi-dimensional information of each business behavior profile is obtained, wherein the multi-dimensional information includes business access frequency, number of visits, access preferences and business characteristic labels; A MySQL database is designed, in which each table corresponds to a multi-dimensional information, and the multi-dimensional information of the business behavior portrait is input into the MySQL database to obtain a business behavior portrait database.

7. A method for acquiring a data flow link according to claim 6, characterized in that: The violation identification strategy includes identifying access frequency, detecting access devices, detecting operation time, and the number of business behavior profile updates, specifically: Collect all operation data of business accounts in real time, including time, location, equipment, data flow rate, business operation behavior and business operation path; The preset access frequency threshold. If the number of accesses to the same resource by the current business account in a short period of time reaches this preset access frequency threshold, it is considered abnormal behavior. Detect access devices and login locations. If the login device of the current business account is not the same as the previous login device, and the login location of the current business account is not the same as the previous login location, it is considered abnormal behavior. Set a specific operation time window for a specific business operation. If the login time of the current business account is not within the specific operation time window, it will be considered as abnormal behavior. A threshold for the number of business behavior profile updates is preset. If the number of times the current business account updates its business behavior profile exceeds the threshold for the number of business behavior profile updates within a preset time range, it is judged as abnormal behavior.

8. A data flow link acquisition system, characterized in that: The system includes a data acquisition module, a label marking module and an anomaly detection module: The data acquisition module is used to obtain the business flow link diagram and deployment architecture diagram and confirm the microservice cut-points of the business application, deploy data collection probes in the form of SDK components at each microservice cut-point, and obtain business data flow information; Transmitting the business data flow information to the label marking module; The label labeling module is used to label the business operations corresponding to the business data flow information with link labels, wherein the link labels include tracking labels, sequence labels and hierarchical labels. The operation information of the same tracking labels is integrated based on the business account, and the calling sequence and hierarchical relationship of the data processing flow are restored according to the sequence labels and hierarchical labels to obtain the data flow path; The anomaly detection module is used to use the data flow path of the business account as the input of the trained behavior determination model to obtain a determination result; Based on the determination results and the preset violation identification strategy, abnormal operations of the business account in business operations are identified, the abnormal operations are compared with the business behavior portrait corresponding to the business account, and risk assessment and early warning are performed based on the comparison results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for obtaining a data flow link as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements a method for acquiring a data flow link as described in any one of claims 1 to 7.

Images