Data analysis method and device and storage medium

Abstract

The invention discloses a data analysis method and device and a storage medium, and is applied to the technical field of computers. According to the method, the data analysis model is constructed for the obtained to-be-analyzed data, then the strategy configuration interface is displayed for the use object to edit through the strategy configuration adding operation of the use object, and then the decision condition is determined according to the first edit data input by the use object on the strategy configuration interface; the decision condition is processed to determine the decision rule of the data analysis model, then the data analysis model is called to determine the score of the field of the to-be-analyzed data, so that the data analysis model outputs the field with the score meeting a preset threshold in the to-be-analyzed data, and the field is called an alarm field. By means of the method, the decision rule of data analysis is configured visually, the requirement of special rule logic can be met, special customization is not needed, and the cost is saved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular, to a data analysis method, apparatus, and storage medium. Background Art

[0002] With the continuous development of Internet technology, the physical asset devices such as computers in large systems such as enterprises are becoming increasingly large, and network security issues are attracting more and more attention from users. The advent of the big data era means that the degree of social informatization has been raised to a new level. As the support for big data applications, the security of databases is particularly important. Data analysis for databases is an important means to discover system vulnerabilities and security risks.

[0003] The current data analysis methods are mainly based on rule-based inference engines, which usually include rule parsers, rule matchers, and inference engines. The rule parser is responsible for parsing and validating the syntax and semantics of rules, the rule matcher is used to match the input data with the rules, and the inference engine executes the matched rules and generates corresponding output results. However, the rule-based inference engine may have certain limitations on the expression ability and flexibility of rules, and specific rule syntax and execution models may not meet the requirements of some complex rule logics, and additional customization or extension is required. Summary of the Invention

[0004] Embodiments of this application provide a data analysis method, apparatus, and storage medium, which can visually configure decision rules for data analysis, meet the requirements of special rule logics, do not require special customization, and save costs.

[0005] In view of this, on the one hand, this application provides a data analysis method, including:

[0006] Obtain data to be analyzed, and construct a data analysis model according to the data to be analyzed;

[0007] In response to a policy configuration addition operation, display a policy configuration interface to receive first edit data input by a user object based on the policy configuration interface;

[0008] Determine decision conditions according to the first edit data;

[0009] Determine decision rules for the data analysis model according to the decision conditions;

[0010] Call the data analysis model to determine the scores of fields in the data to be analyzed, and output warning fields, where the warning fields are fields in the data to be analyzed whose scores meet a preset threshold.

[0011] On the other hand, this application provides a data analysis apparatus, including:

[0012] A building unit, which is used to obtain data to be analyzed and build a data analysis model according to the data to be analyzed;

[0013] A display unit, which is used to respond to a policy configuration addition operation, display a policy configuration interface, and receive first edit data input by a user based on the policy configuration interface;

[0014] A determination unit, which is used to determine decision conditions according to the first edit data on the policy configuration interface; and determine decision rules of the data analysis model according to the decision conditions;

[0015] A scoring unit, which is used to call the data analysis model to determine the score of a field in the data to be analyzed, so as to output an alarm field, and the alarm field is a field in the data to be analyzed whose score meets a preset threshold.

[0016] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the data analysis device further includes a processing unit, and the processing unit is specifically used for:

[0017] Expand fields of the data to be analyzed according to an extension plugin of the data analysis model.

[0018] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the processing unit is further used for:

[0019] Perform abstraction processing on the data to be analyzed after field expansion according to the aggregation requirements of the data analysis model, so as to obtain a first field;

[0020] The scoring unit is specifically used for:

[0021] Call the data analysis model to determine the score of the first field, so as to output an alarm field.

[0022] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application,

[0023] The display unit is further used for:

[0024] Respond to a preprocessing addition operation, display a preprocessing interface, and receive second edit data input by a user based on the preprocessing interface;

[0025] The determination unit is further used for: determining an extension plugin according to the second edit data.

[0026] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application,

[0027] The display unit is further used for:

[0028] In response to the addition operation of abstract processing, an abstract processing interface is displayed to receive third editing data input by the user based on the abstract processing interface.

[0029] The determination unit is further configured to:

[0030] Determine the aggregation requirements according to the third editing data.

[0031] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the determination unit is specifically configured to:

[0032] Orchestrate the decision conditions according to the orchestration algorithm to determine the decision rules of the data analysis model, and the orchestration algorithm is the principle of the shortest rule set.

[0033] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the data analysis device further includes a splitting unit, and the splitting unit is specifically configured to:

[0034] Split and execute the decision rules in the data analysis model whose rule complexity exceeds the preset range.

[0035] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the data analysis device further includes an extraction unit, and the extraction unit is specifically configured to:

[0036] In response to the first selection operation on the target data source among multiple data sources, extract the data to be analyzed from the target data source, and the multiple data sources include at least two of a container database, an elastic search database, a distributed data warehouse, a distributed file system, offline data, a distributed message flow processing platform, and a script.

[0037] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the scoring unit is specifically configured to:

[0038] Call the data analysis model to determine the score for the fields that hit the decision rules according to the numerical interval or numerical comparison, and the numerical interval is the editable interval of the numerical value of the field of the data to be analyzed.

[0039] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the data analysis device further includes a verification unit, and the verification unit is specifically configured to:

[0040] Verify the effectiveness of the decision rules through custom data.

[0041] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the data analysis device further includes a trigger unit, and the trigger unit is specifically configured to:

[0042] Obtain the task cycle;

[0043] Trigger the construction unit in a loop according to the task cycle.

[0044] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the display unit is further configured to:

[0045] Display the alarm field.

[0046] In a possible implementation manner, in another implementation manner of another aspect of the embodiments of the present application, the data analysis device further includes an alarm unit, and the alarm unit is specifically configured to:

[0047] Notify the alarm system to issue an alarm.

[0048] Another aspect of the present application provides a computer device, including:

[0049] A memory, a transceiver, a processor, and a bus system;

[0050] Wherein, the memory is used to store programs;

[0051] The processor is used to execute the programs in the memory, including executing the methods of the above aspects;

[0052] The bus system is used to connect the memory and the processor, so that the memory and the processor can communicate.

[0053] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored, and when the instructions run on a computer, the computer is enabled to execute the methods of the above aspects.

[0054] Another aspect of the present application provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above aspects.

[0055] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0056] In the embodiments of the present application, a data analysis model is constructed for the obtained data to be analyzed, and then through the addition operation of the policy configuration of the user, the policy configuration interface is displayed for the user to edit. Then, the decision conditions are determined based on the first editing data input by the user on the policy configuration interface, and the decision rules of the data analysis model are determined by processing the decision conditions. Then, the data analysis model is called to determine the scores of the fields of the data to be analyzed, so that the data analysis model outputs the fields in the data to be analyzed whose scores meet the preset threshold, and this field is called an alarm field. By the above means, the decision rules of data analysis can be visually configured, which can meet the requirements of special rule logic, without the need for special customization, saving costs. Brief Description of the Drawings

[0057] Figure 1 It is a schematic block diagram of an exemplary system architecture in the embodiments of the present application;

[0058] Figure 2 It is a schematic flowchart of a data analysis method in the embodiments of the present application;

[0059] Figure 3 It is a schematic diagram of a decision representation in the embodiments of the present application;

[0060] Figure 4 It is a schematic diagram of a preprocessing interface in the embodiments of the present application;

[0061] Figure 5 It is a schematic diagram of an abstract processing interface in the embodiments of the present application;

[0062] Figure 6 It is a schematic flowchart of a policy execution in the embodiments of the present application;

[0063] Figure 7 It is a schematic diagram of a basic field configuration interface in the embodiments of the present application;

[0064] Figure 8 It is a schematic diagram of a scoring index in the embodiments of the present application;

[0065] Figure 9 It is a schematic diagram of a result display customization interface in the embodiments of the present application;

[0066] Figure 10 It is a schematic diagram of a result preview in the embodiments of the present application;

[0067] Figure 11 It is a schematic diagram of a unit test function in the embodiments of the present application;

[0068] Figure 12 It is a schematic diagram of a task editing interface in the embodiments of the present application;

[0069] Figure 13Schematic diagram of a data risk assessment architecture in an embodiment of the present application;

[0070] Figure 14 Schematic diagram of the architecture of a task scheduling center in an embodiment of the present application;

[0071] Figure 15 Schematic diagram of the structure of a data analysis device in an embodiment of the present application;

[0072] Figure 16 Schematic diagram of the structure of a computer device in an embodiment of the present application. Detailed implementation manners

[0073] The embodiments of the present application provide a data analysis method, device and storage medium, which are used to visually configure decision rules, can meet the requirements of special rule logics, do not need to be specially customized, and save costs.

[0074] In the description and claims of the present application and the above drawings, the terms "first", "second", "third", "fourth", etc. (if any) are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprising" and "corresponding to" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0075] The special term "exemplary" here means "serving as an example, embodiment or illustration". Any embodiment described as "exemplary" here does not have to be construed as superior to or better than other embodiments.

[0076] In the embodiments of the present application, the term "module" or "unit" refers to a computer program with a predetermined function or a part of a computer program, which works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of the overall module or unit that includes the function of the module or unit.

[0077] In addition, for a better illustration of the present application, numerous specific details are provided in the following specific embodiments. Those skilled in the art should understand that the present application can still be implemented without certain specific details. In some instances, methods, means, elements, and circuits well-known to those skilled in the art are not described in detail to highlight the gist of the present application.

[0078] With the continuous development of Internet technology, physical asset devices such as computers in large systems like enterprises are becoming increasingly large, and network security issues are attracting more and more attention from users.

[0079] The embodiment of the present application provides a data analysis method, which constructs a data analysis model for the to-be-analyzed data obtained, then adds operations through the policy configuration of the user, displays the policy configuration interface for the user to edit, then determines the decision conditions based on the first edited data input by the user on the policy configuration interface, processes the decision conditions to determine the decision rules of the data analysis model, and then calls the data analysis model to determine the field scores of the to-be-analyzed data, so that the data analysis model outputs the fields in the to-be-analyzed data whose scores meet the preset threshold, and this field is called the alarm field. By the above means, the decision rules of data analysis can be configured visually, which can meet the requirements of special rule logics, without the need for special customization, saving costs.

[0080] In the embodiment of the present application, the to-be-analyzed data can be input locally or obtained from a server. The server stores corresponding sample data based on cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or a local area network to achieve data calculation, storage, processing, and sharing. It is the general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model, which can form a resource pool, be used on demand, and be flexible and convenient. Cloud computing technology will become an important support. The background services of technical network systems require a large amount of computing and storage resources, such as video websites, picture websites, and more portal websites. With the highly developed application of the Internet industry, in the future, each item may have its own identification mark and needs to be transmitted to the background system for logical processing. Data at different levels will be processed separately, and various types of industry data require a powerful system support, which can only be achieved through cloud computing.

[0081] The embodiments of this application are applied to the field of artificial intelligence (AI). Artificial intelligence uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, and is a theory, method, technology, and application system that can perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable machines to have the functions of perception, reasoning, and decision-making.

[0082] Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models, also known as large models or foundation models, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0083] Machine learning is an interdisciplinary subject that involves multiple fields such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rote learning.

[0084] The data analysis method provided by the embodiments of this application can be implemented by various electronic devices. For example, it can be implemented by a terminal device alone, or by a server and a terminal device in cooperation. For example, the terminal device alone executes the data analysis method described below, or the terminal device and the server cooperate to execute the data analysis method described below. For example, the terminal device obtains the data to be analyzed, constructs a data analysis model for the data to be analyzed, then adds operations based on the policy configuration of the user, displays a policy configuration interface for the user to edit, the terminal device determines decision conditions based on the edited data of the policy configuration interface, and then arranges rules for the decision conditions to generate decision rules for the data analysis model. Then, the data analysis model and the data to be analyzed are uploaded to the server, so that the server calls the data analysis model to score the fields in the data to be analyzed, and then sends the warning fields whose scores output by the data analysis model meet the preset threshold to the terminal device.

[0085] The electronic device for data risk analysis provided by the embodiments of this application can be various types of terminal devices or servers. Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms; the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal device and the server can be directly or indirectly connected through wired or wireless communication methods, and the embodiments of this application do not limit this.

[0086] Taking the server as an example, for example, it can be a server cluster deployed in the cloud, which opens artificial intelligence cloud services (AIaaS, AI as a Service) to users. The AIaaS platform will split and execute several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI-themed mall, and all users can access and use one or more artificial intelligence services provided by the AIaaS platform through the application programming interface.

[0087] For example, one of the artificial intelligence cloud services can be a data risk analysis service, that is, the server in the cloud encapsulates the data risk analysis program provided in the embodiments of the present application. In response to an object's data risk analysis operation, the terminal device calls the data risk analysis service in the cloud service, so that the server deployed in the cloud calls the encapsulated data risk analysis program to obtain the data to be analyzed. In response to the use of the object policy configuration addition operation, a policy configuration interface is displayed to obtain the edited data to determine the decision conditions, and then rule orchestration is performed to generate the decision rules of the data analysis model, score the data to be analyzed, output the warning fields whose scores meet the preset thresholds, and then feedback the warning fields to the terminal device.

[0088] The following takes the collaborative implementation of the data analysis method provided in the embodiments of the present application by the server and the terminal device as an example for illustration. Refer to Figure 1 , Figure 1 which is a schematic diagram of the exemplary system 100 architecture provided in the embodiments of the present application. The terminal device 110 is connected to the server 130 through the network 120. The network 120 can be a wide area network, a local area network, or a combination of the two.

[0089] In some embodiments, the terminal device 110 constructs a data analysis model in response to the selected data to be analyzed, determines the decision rules of the data analysis model in response to the use of the object's policy configuration addition operation, sends the data analysis model to the server 130, the server 130 calls the data analysis model to score the data to be analyzed, determines the warning fields, and then returns the warning fields to the terminal device 110.

[0090] The following will describe the data analysis method provided in the embodiments of the present application with reference to the accompanying drawings. The execution subject of the following data analysis method takes the terminal device as an example, and specifically can be implemented by the terminal device running various computer programs above; of course, according to the understanding of the following content, it is not difficult to see that the data analysis method provided in the embodiments of the present application can also be collaboratively implemented by the terminal device and the server.

[0091] Please refer to Figure 2 , Figure 2 which shows a schematic flowchart of a data analysis method provided in the embodiments of the present application. The method includes:

[0092] Step 201. Obtain the data to be analyzed and construct a data analysis model according to the data to be analyzed.

[0093] In one or more embodiments, the terminal device may obtain data to be analyzed, which may be provided by the user or loaded from a server based on the user's selection. After obtaining the data to be analyzed, the terminal device may construct a data analysis model. The data analysis model is a conceptual tool for describing the structure, attributes, and relationships of data. It defines the organization of the data, as well as the associations and constraints between the data.

[0094] Step 202. In response to a policy configuration addition operation, display a policy configuration interface to receive first edit data input by the user based on the policy configuration interface.

[0095] In one or more embodiments, after the terminal device constructs the data analysis model, the user may define or modify the analysis rules of the data analysis model and provide a policy configuration addition instruction to the terminal device through a policy configuration addition operation. The terminal device may pop up a policy configuration interface for the user to edit according to the addition policy configuration instruction. Correspondingly, the user may input data as the first edit data on the policy configuration interface. The manner in which the user initiates the policy configuration addition operation may be a click operation on an add policy configuration button in the display interface of the terminal device, which is not limited herein.

[0096] Step 203. Determine a decision condition according to the first edit data.

[0097] In one or more embodiments, after the display interface pops up the policy configuration interface, the first edit data may be in the form of judgment conditions. The editing method may be to add judgment conditions on the policy configuration interface and set corresponding relationships for the added multiple judgment conditions. Then, the terminal device may also use the judgment conditions with corresponding relationships on the policy configuration interface as the decision condition.

[0098] Exemplarily, the policy configuration interface may be as Figure 3 shown in the decision representation diagram. The decision table is a tabular form for describing and representing decision rules, and represents complex policy logics through the combination of condition columns, action columns, and decision rules. Figure 3 In, the relationships between the various judgment conditions are edited with the logical relationships of "and" and "or". Among them, the judgment condition that cannot be further divided may be called an atomic condition, and the judgment condition that contains at least two atomic conditions may be called a composite condition. The user may input multiple judgment conditions on the policy configuration interface and then configure corresponding relationship logics for the multiple judgment conditions to form the decision condition. Each relationship logic indicates the relationship between the judgment conditions or pairs of relationship logics to which it belongs. The pair of relationship logics is a composite condition formed by another relationship logic and the subordinate judgment condition, and this composite condition may be used as the judgment condition subordinate to other relationship logics. As Figure 3The relationship logics of box "and" and box "or" are configured in it for the user to select. When the user selects one of the relationship logics, the box where the corresponding relationship logic is located is represented by a thick line box. For example, Figure 3 If the box "or" is selected in the first box in it, then this box "or" is a thick line box. In this policy configuration interface, for each pair of relationship logics, there are an add condition interface and an add group interface. Among them, the add condition interface can add conditions for judgment for the corresponding pair of relationship logics, such as adding atomic conditions, and the add group interface can nest a pair of relationship logics under the corresponding pair of relationship logics, such as adding composite conditions. Figure 3 In the first pair of relationship logics in it, the box "or" is exemplarily selected, and a first group and a second group are added to this first pair of relationship logics. The second pair of relationship logics in the first group selects the box "and", which is represented in the form of a thick line box. And conditions 1 "event_type equals PROCESS_KPROBE" and 2 "function_name equals do_int_module" are added through the add condition interface. The box "and" means that conditions 1 and 2 need to be satisfied simultaneously. Among them, the editing form of the conditions can be in the form of selecting fields, selecting judgment methods, and selecting judgment objects. A delete interface can also be configured for each condition to delete unnecessary conditions. The third pair of relationship logics in the second group selects the box "or" and nests a third group. The fourth pair of relationship logics in the third group selects the box "and", and conditions 3 "event_type equals PROCESS_EXEC" and 4 "process_binary regular expression matches V" are added. The third group can also nest a fourth group. The fifth pair of relationship logics in the fourth group selects the box "or", and conditions 5 "process_args contains -e", 6 "process_args contains -c", and 7 "process_args contains —sh-exec" are added.

[0099] Step 204. Determine the decision rule of the data analysis model according to the decision condition.

[0100] In one or more embodiments, after obtaining the decision condition, the decision condition can be converted into a form that the data analysis model can execute. This executable form can be called a decision rule, that is, the decision rule of the data analysis model is updated.

[0101] Step 205. Call the data analysis model to determine the score of the fields in the data to be analyzed, so as to output warning fields. The warning fields are the fields in the data to be analyzed whose scores meet the preset threshold.

[0102] In one or more embodiments, after determining the decision rule, the data analysis model can be called to evaluate each field in the data to be analyzed, determine whether each field meets the decision rule, that is, find the first atomic condition recursively, store the execution result, and execute layer by layer. Then, the score of the corresponding field is determined according to the judgment result. After the score is determined, the data analysis model can compare the scores of each field in the data to be analyzed based on a pre-configured preset threshold, and determine the field corresponding to the score exceeding the preset threshold as a risk field, which can also be called an alarm field, that is, the field that needs to remind the user. Among them, multiple preset thresholds can also be set, that is, the risk levels of the fields in the data to be analyzed can be divided, such as "low risk, medium risk, high risk, etc.", providing intuitive support for the user to process high-risk fields in a timely manner.

[0103] In the embodiment of the present application, a data analysis model is constructed for the data to be analyzed obtained, and then through the operation of adding strategy configuration by the user, a strategy configuration interface is displayed for the user to edit. Then, the decision condition is determined according to the first edited data input by the user on the strategy configuration interface, the decision rule of the data analysis model is determined by processing the decision condition, and then the data analysis model is called to determine the score of the field of the data to be analyzed, so that the data analysis model outputs the field in the data to be analyzed whose score meets the preset threshold, and this field is called an alarm field. By the above means, the decision rule of data analysis can be configured visually, which can meet the requirements of special rule logic, does not require special customization, and saves costs.

[0104] Optionally, on the basis of the above Figure 2 corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, before calling the data analysis model to determine the score of the field in the data to be analyzed to output the alarm field, the method further includes:

[0105] Expanding the fields of the data to be analyzed according to the extension plug-in of the data analysis model.

[0106] In one or more embodiments, a method for expanding the data of the data analysis model is introduced. The data analysis model can be configured with extension plug-ins, which can expand the fields in the data to be analyzed. For example, built-in common plug-ins such as employee information query, time format conversion, and nested field extraction are used to extract new fields from the fields of the data to be analyzed to expand the data of the data analysis model. Among them, the extension plug-in can be factory-configured or a private plug-in uploaded by the user, which is not limited here.

[0107] In an embodiment of the present application, a method for expanding data of a data analysis model is provided. Through the above method, new fields can be extracted from the data to be analyzed through an extension plugin, which can provide more suitable data for the decision-making rules of the subsequent data analysis model and improve the effect or accuracy of subsequent processing.

[0108] Optionally, on the basis of the corresponding respective embodiments above, in another optional embodiment provided by the embodiment of the present application, after expanding the fields of the data to be analyzed according to the extension plugin of the data analysis model, the method further includes: Figure 2 Abstracting the data to be analyzed with the expanded fields according to the aggregation requirements of the data analysis model to obtain a first field;

[0109] Invoking the data analysis model to determine the scores of the fields in the data to be analyzed, and the output warning fields include:

[0110] Invoking the data analysis model to determine the score of the first field to output warning fields.

[0111]

[0112] In one or more embodiments, a method for abstracting the data to be analyzed is introduced. Abstract processing is to aggregate some metric fields from the data and perform refined identification on the policy data, so that subsequent policy decisions are more accurate. In the embodiment of the present application, the data analysis model is configured with aggregation requirements, and the data to be analyzed with the expanded fields can be further integrated. That is, the aggregated fields, the expanded fields, and the original fields of the data to be analyzed can be collectively referred to as the first field. Among them, the data can be grouped before aggregation. Grouping refers to dividing the original data into multiple groups using specific conditions. Aggregation here refers to performing certain operations (such as aggregation, transformation, etc.) on the data in each group, and finally integrating the calculated results. Correspondingly, when the data analysis model determines the scores of the fields in the data to be analyzed, it can also determine the scores of the expanded fields and the aggregated fields, that is, determine the scores of the first field. Among them, the aggregation condition in the abstract processing is a processing function for field processing, and the processing function can be calculating the total, calculating the total number of different items, summing, finding the maximum value, finding the minimum value, finding the average value, etc.

[0113] In the embodiment of the present application, a method for abstracting the data to be analyzed is provided. Through the above method, abstracting the data to be analyzed with the expanded fields based on the aggregation requirements can accurately generate data that meets the business problems without viewing the original data, so that the user can interact with the data at a higher granularity level.

[0114] Figure 2 Optionally, in the above Figure 2Based on the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, before extending the fields of the data to be analyzed according to the extension plug-in of the data analysis model, the method further includes:

[0115] In response to a preprocessing addition operation, display a preprocessing interface to receive second editing data input by a user based on the preprocessing interface;

[0116] Determine the extension plug-in according to the second editing data.

[0117] In one or more embodiments, a method of adding preprocessing data is introduced. After the terminal device constructs a data analysis model, the user can preprocess the fields in the data analysis model. For example, through a preprocessing addition operation, the user provides a preprocessing addition instruction to the terminal device, so that the terminal device can pop up a preprocessing interface for the user to edit according to the preprocessing addition instruction. The user can select the fields to be extended and the corresponding plug-ins on the preprocessing interface, that is, indicate to use the plug-in to extend the corresponding fields to be extended. The fields to be extended and the corresponding plug-ins can be referred to as second editing data, and the plug-in can be referred to as an extension plug-in. Among them, the way for the user to initiate the preprocessing addition operation can be a click operation on the preprocessing addition button in the display interface of the terminal device, which is not limited here.

[0118] Exemplarily, please refer to Figure 4 the schematic diagram of the preprocessing interface shown in Figure 4 where the user can add original fields, configure corresponding plug-ins for the original fields to extend, edit the extension parameters of the plug-ins, and then name the extended fields. Exemplarily, for example, the original fields to be extended "OriginStaffName" and "Operator" added through the preprocessing addition button. The plug-in used for the original field "OriginStaffName" is exemplarily a private plug-in, and the extended parameter is "the second parameter of the plug-in, type str". The plug-in is a code file, and the method usage of the plug-in is agreed here, referring to the second input parameter of the method, and the output is of string type. The newly extended fields are named "leader; deptname; staff_status; title". The plug-in used for the original field "Operator" is exemplarily a private plug-in, and the extended parameter is "op", and the newly extended field is named "op_leader". Among them, for the already edited original fields, an operation interface can also be provided, that is, a removal button is configured for each original field, and the user can delete the unnecessary original fields through the removal button. The position of the removal button is exemplarily as Figure 4 shown in. The newly extended fields can also be previewed through the data preview button.

[0119] In an embodiment of the present application, a method for adding preprocessed data is provided. Through the above method, an editing path for preprocessing is visually provided, and the data to be analyzed is expanded into a form that meets the task requirements, improving the effect or accuracy of subsequent processing, or making the data suitable for specific algorithms or task abstraction processing.

[0120] Optionally, based on the corresponding embodiments above, in another optional embodiment provided by the embodiments of the present application, before abstracting the data to be analyzed after expanding the fields according to the aggregation requirements of the data analysis model to obtain the first field, the method further includes: Figure 2 In response to the abstract processing addition operation, display an abstract processing interface to receive third editing data input by the user based on the abstract processing interface;

[0121] Determine the aggregation requirements according to the third editing data.

[0122] In one or more embodiments, a method for adding abstract processing data is introduced. After the terminal device expands the fields of the data to be analyzed, the user can perform abstract processing on the expanded data to be analyzed. For example, through the abstract processing addition operation, an abstract processing addition instruction is provided to the terminal device, so that the terminal device can pop up an abstract processing interface for the user to edit according to the abstract processing addition instruction. The user can select the fields to be aggregated on the abstract processing interface, such as the aggregation fields selected from the data to be analyzed after expanding the fields, and then edit the aggregation conditions for the aggregation fields to perform processing calculations on the aggregation fields. The third editing data may at least include the aggregation fields and the corresponding aggregation conditions, and may also include the index name of the abstract processing and the naming of the new fields after abstract processing of the aggregation fields. The aggregation requirements may be the requirements for aggregating the aggregation fields using the aggregation conditions. Among them, the way for the user to initiate the abstract processing addition operation may be a click operation on the abstract processing addition button in the display interface of the terminal device, which is not limited here.

[0123] Exemplarily, please refer to the schematic diagram of the abstract processing interface shown in

[0124] Exemplarily, please refer to Figure 5 the schematic diagram of the abstract processing interface shown, Figure 5The example shows the abstract indicator names "cnt" and "min_time" added by adding an abstract processing button. The aggregation condition used by the abstract indicator name "cnt" is exemplarily to find the total number, the grouping fields are "OriginStaffName" and "Operator", the aggregation field is "Operator", and the new field is "cnt". The aggregation condition used by the abstract indicator name "min_time" is exemplarily to find the minimum value, the grouping fields are "OriginStaffName" and "Operator", the aggregation field is "Timestamp", and the new field is "min_time". Among them, for the abstract indicator names that have been edited, an operation interface can also be provided, that is, a removal button is configured for each abstract indicator name, and the user can delete unnecessary abstract indicator names through the removal button. The position of the removal button is exemplified as follows Figure 5 Aggregation can be performed after grouping or without grouping. Aggregation after grouping is equivalent to counting the number of times the aggregated field appears in the grouped field.

[0125] In an embodiment of the present application, a method for adding abstract processing data is provided. Through the above method, visualization provides an editing path for abstract processing, aggregates the data to be analyzed into fields that meet higher granularity, and improves the effect or accuracy of subsequent processing.

[0126] Optionally, in the above Figure 2 On the basis of the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, the decision rule of the data analysis model is determined according to the decision condition, including:

[0127] The decision conditions are arranged according to the arrangement algorithm to determine the decision rules of the data analysis model. The arrangement algorithm is based on the principle of the shortest rule set.

[0128] In one or more embodiments, a method of arranging decision rules is introduced. After the object is used to generate decision conditions through editing, the terminal device can also use a predefined arrangement algorithm to re-arrange the rules set for the object, and use the re-arranged rules as the decision rules of the data analysis model. The arrangement algorithm uses the shortest rule set principle to select easy-to-execute rules to execute first, that is, to put the time-consuming and costly rules at the end as much as possible. Specifically, the shortest rule set principle may include: 1. Atomic conditions take precedence over compound conditions. 2. Compound conditions with fewer atomic conditions are better than compound conditions with more. 3. The "or" rule of the compound condition is placed before the "and" rule. 4. The atomic conditions are prioritized according to the difficulty of matching, for example, string comparison has a higher priority than regular matching. Among them, when arranging the rules, each rule in the rule set can also be numbered, the hit rule can be marked with a number, and appended to the original data model, so that it can be directly located when viewing risk data.

[0129] In an embodiment of the present application, a method for arranging decision rules is provided. Through the above method, the decision conditions are rearranged using the principle of the shortest rule set, and the rearranged rules are used as the decision rules of the data analysis model. The data analysis model can preferentially execute the rules that are easy to execute, and place the rules with long execution time and high cost at the end. If a rule is hit, it is directly asserted, and the subsequent rules will not be executed, saving time.

[0130] Optionally, based on the corresponding embodiments above, in another optional embodiment provided by the embodiments of the present application, before calling the data analysis model to determine the score of the fields in the data to be analyzed and output the warning fields, the method further includes: Figure 2 Before calling the data analysis model to determine the score of the fields in the data to be analyzed and output the warning fields, the method further includes:

[0131] Split and execute the decision rules in the data analysis model whose rule complexity exceeds the preset range.

[0132] In one or more embodiments, a method for splitting decision rules is introduced. After determining the decision rules of the data analysis model, it is also possible to determine whether to perform split execution based on the rule complexity of the decision rules. When it is determined that the rule complexity of the decision rules exceeds the preset range, it can be split into multiple conditions for multiple devices to execute. For example, multiple devices are used to determine the scores of the fields in the data to be analyzed according to the split conditions, and then the scores are aggregated to the terminal device. Among them, the splitting logic is that only parallel rules can be split, and the splitting depth can be defaulted to 1 layer. Exemplarily, if a rule set contains single conditions A, B, and C, if all three are single conditions, they are not split. If there are composite conditions nested within the three conditions, and there are further nested conditions within the composite conditions, splitting may need to be considered. The execution processes of these three conditions are distributed to three machines for operation, and the results are finally aggregated. In actual operation, whether to split can depend on: 1. The data volume. 2. The nesting depth within the composite conditions. 3. The actual condition combination.

[0133] Exemplarily, after the data model is established, it enters the rule engine, and the rule engine executes the rules. The specific policy execution process can refer to Figure 6As shown, in step 601, obtain a rule matching table, which is generated by a user editing in a policy configuration interface. That is, when the user inputs the judgment conditions and the logical relationship between the judgment conditions in the policy configuration interface, the rule matching table is a set of all the input judgment conditions and the logical relationship therebetween. In step 602, arrange the decision rules according to an arrangement algorithm, that is, rearrange the rule matching table set by the user through the shortest rule set principle, and select the rules that are easy to execute to execute first, that is, place the rules with large time consumption and high cost at the end as much as possible. In step 603, determine whether to split the execution according to the rule complexity. That is, exemplarily, the rule complexity of the decision rule can be determined according to the data volume, the nesting depth (number of nested levels) in the composite conditions in the decision rule, and the combination situation between the actual conditions. The higher the rule complexity, the longer the time required for the data analysis model to process. When the rule complexity exceeds a threshold, the rule can be split for distributed execution. Therefore, it can be determined whether the rule complexity exceeds the threshold. If so, execute step 604, otherwise execute step 606. In step 604, split the decision rule into multiple sub-rules. That is, when the rule complexity of the decision rule exceeds the threshold, separate the conditions with too large data volume, too many nestings, and too many combinations. The separated rule conditions can be called sub-rules. In step 605, send the multiple sub-rules to multiple worker nodes, and each worker node bears one or more sub-rules. That is, distribute the sub-rules to multiple worker nodes for processing in a distributed manner to accelerate the execution speed of the rules. In step 606, execute the rules. Among them, when the decision rule is split for execution, the terminal device and each worker node execute the corresponding sub-rules, that is, analyze the fields according to the sub-rules. When the decision rule is not split, the terminal device directly executes the decision rule. In step 607, store the execution result. Among them, for the sub-rules split for execution, each worker node needs to report the result to the terminal device after executing each sub-rule or all sub-rules. In step 608, determine whether there are still rules not executed, that is, the terminal device and the worker nodes respectively determine whether the sub-rules they bear have been executed. If so, execute step 606, otherwise execute step 609. In step 609, output the execution result. Among them, when the decision rule is not split, the terminal device directly outputs the execution result of the decision rule. If the decision rule is split for execution, the terminal device also needs to summarize and make a decision on the execution results reported by each worker node, and then output the summary decision result.

[0134] In an embodiment of the present application, a method for splitting a decision rule is provided. Through the above method, splitting a complex decision rule for other devices to execute can improve the execution speed of the rule and save time.

[0135] Optionally, in the above Figure 2Based on the corresponding embodiments, in another optional embodiment provided by the embodiments of the present application, before constructing a data analysis model according to the data to be analyzed, the method further includes:

[0136] In response to a first selection operation on a target data source among multiple data sources, extract the data to be analyzed from the target data source, where the multiple data sources include at least two of a container database, an elasticsearch database, a distributed data warehouse, a distributed file system, offline data, a distributed message flow processing platform, and a script.

[0137] In one or more embodiments, a method for obtaining the data to be analyzed is introduced. The terminal device can configure interfaces for multiple data sources, and then the user can indicate the target data source for extracting the data to be analyzed to the terminal device according to requirements. Specifically, the user can indicate the target data source to the terminal device from multiple data sources through a first selection operation, and then the terminal device responds to the first selection operation and extracts the data to be analyzed from the target data source. The diversification of data sources supports combining the data of different database products to establish a unified large data model. Among them, the supported data sources can include at least two of a container database (CDB), an elasticsearch database (es), a distributed data warehouse (TDW), a hadoop distributed file system (HDFS), offline data, a distributed message flow processing platform (kafka), and a script. The first selection operation can be a click operation by the user on the target data source among the multiple data sources provided on the display interface of the terminal device, which is not limited herein.

[0138] Exemplarily, a schematic diagram of the basic field configuration interface for editing the data to be analyzed can be referred to Figure 7 as shown. The basic field configuration interface can edit the name of the process for constructing the data analysis model this time, the person in charge of editing this process, and the relevant description of this process, as well as the referenced process, task cycle, and consumed resources, etc. The person in charge can be the account owner or a reassigned person. Among them, Figure 7 the process name takes the simulation process as an example, and the person in charge takes person A as an example. The relevant description can facilitate determining some relevant information of this process, Figure 7 which is not shown. The basic field configuration interface provides multiple data source types, such as cdb, es, tdw, kafka, script, Figure 7Exemplarily, es is selected in the form of solid circles. The basic field configuration interface can also set access credentials. Only when the credentials pass can this process be executed. When there are no qualified credentials, qualified credentials can also be configured through the credential management interface. The basic field configuration interface can also edit the query statement to improve the flexible selectivity of the data to be analyzed. The query type can be a regular query. Then, in response to the addition button, fields can be extracted from es according to the edited query statement. For example Figure 7 The original field names in it, such as Operator, ActionTime, OriginStaffName, ingest_timestamp, Timestamp, and the mapped field names can be set according to requirements. Correspondingly, the data of a database table can be risk-identified from different dimensions. Some fields are required by one policy, while another policy may not require them. Then, for some fields extracted that are not relevant to the current risk assessment, the selected fields can be deleted through the delete button. The basic field configuration interface is also configured with a data preview button to automatically build a data analysis model after data preview.

[0139] In the embodiments of the present application, a method for obtaining data to be analyzed is provided. Through the above method, the diversification of data sources supports combining the data of different database products, and at the same time supports multiple database products, enabling unobstructed data pulling and improving the flexibility of data selection.

[0140] Optionally, based on the above Figure 2 corresponding respective embodiments, in another optional embodiment provided by the embodiments of the present application, calling the data analysis model to determine the score of the fields in the data to be analyzed, and the warning fields to be output include:

[0141] Calling the data analysis model to determine the score of the fields that hit the decision rule according to the numerical interval or numerical comparison. The numerical interval is an editable interval of the numerical values of the fields in the data to be analyzed.

[0142] In one or more embodiments, a method for scoring data to be analyzed is introduced. In addition to editing rules through the policy configuration interface, the user can also configure scoring indicators for the fields that hit the decision rule to determine the score. For example, by editing the numerical interval to divide the scoring score stage for the field, and giving corresponding scores according to the numerical interval into which the field falls. It is also possible to perform numerical comparison, such as equal to, not equal to, or regular match, and give corresponding scores when the numerical value of the field is equal to, not equal to, or regular matches the numerical range.

[0143] Exemplarily, the scoring indicators can refer to Figure 8 the schematic diagram shown, and new scoring indicators can be added through the new indicator button, and the score calculation method corresponding to each numerical interval can be configured for each scoring indicator. Such as Figure 8The cnt indicator is newly added. cnt is a field value. Judgment conditions can be added for the cnt indicator. For example, when the value range is in [0, 100), the score of this field is 50; when the value range is in [100, 5000), the score of this field is 50+(cnt - 100)*0.055; when the value range is in [5000, 10000), the score of this field is 100+(cnt - 1000)*0.0125; when the value range is in [10000, 999999), the score of this field is 150+(cnt - 5000)*0.01. Equal to, not equal to, or regular matching is not shown in the figure. The terminal device can also be configured with a scoring test function, such as Figure 8 the basic score of the policy can be edited in it, and the total score calculation formula. For example, for the cnt indicator, the test result can be output through the test cases of the scoring test. Among them, the scoring test can also provide a test data input interface for quick testing. For the newly added indicators and the newly added conditions of each indicator, corresponding delete buttons can also be configured to delete the unnecessary indicators or conditions.

[0144] The terminal device can also provide a custom interface for scoring display, and the data fields to be alarmed and displayed to the viewing object can be selected. When displaying, some data fields that are not needed for simplicity can be not selected. For example, please refer to Figure 9 the schematic diagram of the custom interface for result display shown, indicating the data fields that can be selected to be retained ("retained fields"). For example, time, node_name, cluster, event_type, process_exec, process_exit, kprobe, uprobe, tracepoint. All data fields can be selected by the "select all" button, or some of the data fields can be selected and retained. After the user of the object determines the selection by submitting the button, the terminal device can preview the information corresponding to the retained data fields. For example, please refer to Figure 10 the schematic diagram of result preview shown. Taking the selection of all data fields as an example, the result can display the information with results, such as time: time 1, node_name: 10.0.0.2, cluster: {}, event_type: PROCESS_KPROBE, kprobe: {}, and also includes the score event_scroe: 0 and the level event_level: low risk.

[0145] In the embodiment of the present application, a method for scoring data to be analyzed is provided. Through the above method, scoring is performed according to the value range or value comparison, and targeted scoring is performed on the name rule data, so as to perform fine-grained operation differentiation after risk warning.

[0146] Optionally, in the above Figure 2Based on the corresponding various embodiments, in another optional embodiment provided by the embodiments of the present application, after determining the decision rule of the data analysis model according to the decision condition, the method further includes:

[0147] Verify the effectiveness of the decision rule through custom data.

[0148] In one or more embodiments, a method for verifying the effectiveness of a decision rule is introduced. The terminal device can configure unit tests and test each editing unit (extracting data to be analyzed, preprocessing, abstract processing, determining decision rules, scoring) of the entire process through custom data. For decision rules, since decision rules can configure very complex rule sets, there may not be real data to test the rationality of the rule settings. Using unit tests can construct data by itself to verify the effectiveness of each rule link. It is also possible to configure the situation through a comprehensive test strategy of real data and simulated data, which is not limited here.

[0149] The schematic diagram of the unit test function can be referred to Figure 11 as shown, and a unit test button can be configured on the basis of Figure 9 When the unit test button is triggered, the test function can be provided by selecting the editing unit of the process. The corresponding test process can be configured by the test process button.

[0150] In the embodiments of the present application, a method for verifying the effectiveness of a decision rule is provided. Through the above method, custom data is used to verify the effectiveness of the decision rule, which can timely verify the effectiveness of each rule link and improve the test efficiency.

[0151] Optionally, based on the corresponding various embodiments above, in another optional embodiment provided by the embodiments of the present application, the method further includes: Figure 2 Obtain the task cycle;

[0152] According to the task cycle, trigger the step of obtaining the data to be analyzed in a loop, and construct a data analysis model according to the data to be analyzed.

[0153] In one or more embodiments, a method for triggering risk assessment is introduced. For the already constructed data analysis model, the data analysis model can be executed in a loop by indicating the task cycle, that is, continuously obtain the data to be analyzed according to the task cycle for evaluation. Exemplarily, for example, it is indicated to obtain data every other day for risk assessment. The task cycle can be obtained by editing by the user.

[0154] Exemplarily, please refer to

[0155] Exemplarily, please refer to Figure 12Schematic diagram of the task editing interface shown. The task editing interface may include: task name: Risk Grading; task description: DLP Alarm Risk Grading; person in charge: B; alarm configuration: [Data Security] DLP Alarm Risk Grading; generate work order: Yes (indicating that a work order needs to be generated); reference process: [Data Security] DLP Alarm Risk Grading (data analysis model); task direction: data security; the task cycle can be selected to generate or manually generate. When selecting to generate, the minimum interval can be selected from minutes, hours, weeks, dates, and months. Taking minutes as an example, it can be selected in a cyclic or specified manner. For the cyclic mode, it starts from "*" minutes (e.g., 0) and executes once every "*" minutes. For the specified mode, it executes at the specified "*" minute, and "*" is editable. For manual generation, the object is used to execute the input execution interval or specify the execution time, which will not be elaborated here. In the figure, the example of cyclic execution with a minute interval for selection generation is shown. The task editing interface can also configure the consumed resources, such as the normal consumption for editing. The task editing interface can also configure the alarm template, such as event level, name, whether it is the leave period, the occurred event, department, total amount of externally sent high-risk data, etc. After editing, the task can be confirmed to execute through the confirmation button. Among them, the generated work order can be linked to the work order system, and the generated work order is sent to the work order system for storage or display.

[0156] In an embodiment of the present application, a method for triggering risk assessment is provided. Through the above method, data risk assessment is executed cyclically, improving the timeliness of discovering data risks.

[0157] Optionally, based on the above Figure 2 corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, the method further includes:

[0158] Display the alarm fields.

[0159] In one or more embodiments, a method for risk alarm is introduced. After the data analysis model outputs the alarm fields based on the scores of the fields in the data to be analyzed and the fields processed by extension and abstraction, the terminal device can also display the alarm fields on the display interface for the user to view and process.

[0160] In an embodiment of the present application, a method for risk alarm is provided. Through the above method, the user can process in a timely manner based on the displayed alarm fields.

[0161] Optionally, based on the above Figure 2 corresponding embodiments, in another optional embodiment provided by the embodiment of the present application, after calling the data analysis model to determine the scores of the fields in the data to be analyzed to output the alarm fields, the method further includes:

[0162] Notify the alarm system to issue an alarm.

[0163] In one or more embodiments, a method for risk warning is introduced. After the data analysis model outputs the warning fields, the terminal device can directly issue a warning, or send a warning notification to the warning system, and the warning system publishes the warning information, and the form of the warning information can be the form indicated by the warning template.

[0164] The data risk assessment architecture in the embodiments of the present application can be as Figure 13 shown. This architecture includes a data layer, a platform layer, and an application layer. Among them, the data layer provides the data sources required by the policies, and currently supports the streaming data type kafka, offline data: ES, CDB, TDW, HDFS, and script preprocessed data: script.

[0165] Platform layer: includes a policy center, a scheduling center, an operation center, and a monitoring center. The core is the policy center, which is the logical function of policy development, including data models, preprocessing, policy management, scorecards, unit tests, policy tuning, etc. The scheduling center works after the process is configured, including task distribution, task queue, and result storage. The tasks are divided into two categories: periodic tasks and resident tasks. The operation center provides the work order events, real-time metrics, and task view analysis required for operation. The monitoring center provides the output of policy warnings and the monitoring of the policy operation process. The platform layer may also include credential management / plugin management / user management / black and white lists, etc.

[0166] The application layer may include corresponding policy processes, task scheduling, warning systems, operation systems, and management systems.

[0167] Among them, the architecture of the task scheduling center can be referred to Figure 14 shown. This architecture mainly consists of 4 parts: a task distribution unit (server), a message middleware (message broker), a task execution unit (worker), and a task execution result storage (task result store). First, multiple servers initiate asynchronous tasks (async task) or the scheduler celery beat initiates a periodic task, and sends the task to the queue of the message broker. When the task reaches the message broker, it will distribute it to the corresponding celery worker for processing (distribute task). When the task is processed, its result is stored in the task result store (store task result). Celery is a simple, flexible, and reliable distributed system for processing a large number of messages, and at the same time provides the tools required to maintain such a system for operations, and can also be used for task scheduling.

[0168] This architecture can utilize the resource advantages of cloud native and be deployed distributively. The task queue and task processors are deployed on multiple machines to achieve parallel processing of tasks and load balancing. This enables the application to handle a large number of tasks and has high scalability. Asynchronous processing of tasks, putting time-consuming tasks into the task queue, allows the application to continue processing other requests without waiting for the tasks to complete. This improves the response performance and throughput of the application.

[0169] In an embodiment of the present application, a method of risk warning is provided. Through the above method, the warning system is notified to issue a warning, and the warning of the warning system can notify the user object faster, improving the warning efficiency.

[0170] The data analysis device in the present application will be described in detail below. Please refer to Figure 15 , Figure 15 which is a schematic diagram of an embodiment of the data analysis device in an embodiment of the present application. The data analysis device 150 includes:

[0171] A construction unit 1501, configured to obtain data to be analyzed and construct a data analysis model according to the data to be analyzed;

[0172] A display unit 1502, configured to respond to a policy configuration addition operation and display a policy configuration interface to receive first edit data input by a user object based on the policy configuration interface;

[0173] A determination unit 1503, configured to determine decision conditions according to the first edit data; and determine decision rules of the data analysis model according to the decision conditions;

[0174] A scoring unit 1504, configured to call the data analysis model to determine the score of the fields in the data to be analyzed, so as to output warning fields, where the warning fields are the fields in the data to be analyzed whose scores meet a preset threshold.

[0175] In an embodiment of the present application, a data analysis device is provided. Through the above device, the decision rules of data analysis can be configured visually, which can meet the requirements of special rule logics without the need for special customization, saving costs.

[0176] Optionally, on the basis of the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in an embodiment of the present application, the data analysis device 150 further includes a processing unit 1505, and the processing unit 1505 is specifically configured to: Figure 15 Expand the fields of the data to be analyzed according to the extension plugin of the data analysis model.

[0177]

[0178] In an embodiment of the present application, a data analysis device is provided. Through the above device, new fields can be extracted from the data to be analyzed through an extension plugin, which can provide more suitable data for the decision rules of subsequent data analysis models and improve the effect or accuracy of subsequent processing.

[0179] Optionally, based on the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the processing unit 1505 is further configured to: Figure 15 Abstract the data to be analyzed after expanding the fields according to the aggregation requirements of the data analysis model to obtain the first field;

[0180] The scoring unit 1504 is specifically configured to:

[0181] Call the data analysis model to determine the score of the first field to output the warning field.

[0182] In an embodiment of the present application, a data analysis device is provided. Through the above device, abstract processing is performed on the data to be analyzed after expanding the fields based on the aggregation requirements, and data that meets the business problems can be accurately generated without viewing the original data, so that the user can interact with the data at a higher granularity level.

[0183] Optionally, based on the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the display unit 1502 is further configured to:

[0184] Respond to the preprocessing addition operation, display the preprocessing interface to receive the second edited data input by the user based on the preprocessing interface; Figure 15 The determination unit 1503 is further configured to: determine the extension plugin according to the second edited data.

[0185] In an embodiment of the present application, a data analysis device is provided. Through the above device, a visual editing path for preprocessing is provided, the data to be analyzed is expanded into a form that meets the task requirements, the effect or accuracy of subsequent processing is improved, or the data is made suitable for specific algorithm or task abstraction processing.

[0186] Optionally, based on the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the display unit 1502 is further configured to:

[0187] Respond to the abstract processing addition operation, display the abstract processing interface to receive the second edited data input by the user based on the preprocessing interface;

[0188] Optionally, based on the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the display unit 1502 is further configured to: Figure 15 Respond to the abstract processing addition operation, display the abstract processing interface to receive the second edited data input by the user based on the preprocessing interface;

[0189] Respond to the abstract processing addition operation, display the abstract processing interface to receive the second edited data input by the user based on the preprocessing interface;

[0190] The determination unit 1503 is further configured to:

[0191] Determine an aggregation requirement according to third editing data.

[0192] In an embodiment of the present application, a data analysis device is provided. Through the above device, an editing path for abstract processing is visually provided, and the data to be analyzed is aggregated into fields that meet a higher granularity, improving the effect or accuracy of subsequent processing.

[0193] Optionally, on the basis of the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the determination unit 1503 is specifically configured to: Figure 15 Arrange decision conditions according to an arrangement algorithm to determine the decision rules of the data analysis model, and the arrangement algorithm is the principle of the shortest rule set.

[0194] Arrange the decision conditions according to the arrangement algorithm to determine the decision rules of the data analysis model, and the arrangement algorithm is the principle of the shortest rule set.

[0195] In an embodiment of the present application, a data analysis device is provided. Through the above device, the decision conditions are rearranged using the principle of the shortest rule set, and the rearranged rules are used as the decision rules of the data analysis model. The data analysis model can preferentially execute the rules that are easy to execute, and the rules with large time consumption and large cost are placed at the end. If a hit occurs, it is directly asserted, and subsequent rules will not be executed, saving time.

[0196] Optionally, on the basis of the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the data analysis device 150 further includes a splitting unit 1506, and the splitting unit 1506 is specifically configured to: Figure 15 Split and execute the decision rules in the data analysis module whose rule complexity exceeds a preset range.

[0197] Split and execute the decision rules in the data analysis module whose rule complexity exceeds a preset range.

[0198] In an embodiment of the present application, a data analysis device is provided. Through the above device, splitting complex decision rules for other devices to execute can improve the execution speed of the rules and save time.

[0199] Optionally, on the basis of the corresponding embodiment above, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the data analysis device 150 further includes an extraction unit 1507, and the extraction unit 1507 is specifically configured to: Figure 15 In response to a first selection operation on a target data source among multiple data sources, extract data to be analyzed from the target data source, and the multiple data sources include at least two of a container database, an elasticsearch database, a distributed data warehouse, a distributed file system, offline data, a distributed message stream processing platform, and a script.

[0200] In response to a first selection operation on a target data source among multiple data sources, extract data to be analyzed from the target data source, and the multiple data sources include at least two of a container database, an elasticsearch database, a distributed data warehouse, a distributed file system, offline data, a distributed message stream processing platform, and a script.

[0201] In an embodiment of the present application, a data analysis device is provided. Through the above device, data sources are diversified, data from different database products can be combined, and multiple database products can be supported simultaneously, data can be pulled without obstacles, and the flexibility of data selection can be improved.

[0202] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the scoring unit 1504 is specifically configured to:

[0203] The data analysis model is called to determine the score of the field that hits the decision rule according to the numerical range or numerical comparison. The numerical range is an editable range of the numerical value of the field of the data to be analyzed.

[0204] In an embodiment of the present application, a data analysis device is provided. By using the device, a score is determined according to a numerical interval or numerical comparison, and a targeted score is performed on the name rule data, so as to make fine-grained operation distinctions after risk alerts.

[0205] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the data analysis device 150 further includes a verification unit 1508, and the verification unit 1508 is specifically configured to:

[0206] Verify the effectiveness of decision rules through custom data.

[0207] In an embodiment of the present application, a data analysis device is provided. By using the device, the validity of decision rules can be verified by customizing data, and the validity of each regular link can be verified in a timely manner, thereby improving test efficiency.

[0208] Optionally, in the above Figure 15 On the basis of the corresponding embodiment, in another embodiment of the data analysis device 150 provided in the embodiment of the present application, the data analysis device 150 further includes a trigger unit 1509, and the trigger unit 1509 is specifically configured to:

[0209] Get the task cycle;

[0210] The construction unit 1501 is triggered according to the task cycle cycle.

[0211] In an embodiment of the present application, a data analysis device is provided, through which data risk assessment is cyclically performed to improve the timeliness of discovering data risks.

[0212] Optionally, in the above Figure 15Based on the corresponding embodiment, in another embodiment of the data analysis device 150 provided in the embodiments of the present application, the display unit 1502 is further configured to:

[0213] Display the alarm field.

[0214] In the embodiments of the present application, a data analysis device is provided. Through the above device, the user can process in a timely manner according to the displayed alarm field.

[0215] Optionally, based on the above Figure 15 Based on the corresponding embodiment, in another embodiment of the data analysis device 150 provided in the embodiments of the present application, the data analysis device 150 further includes an alarm unit 1510, and the alarm unit 1510 is specifically configured to:

[0216] Notify the alarm system to issue an alarm.

[0217] In the embodiments of the present application, a data analysis device is provided. Through the above device, the alarm system is notified to issue an alarm, and the alarm of the alarm system can notify the user faster, improving the alarm efficiency.

[0218] Figure 16 FIG. is a schematic structural diagram of a computer device provided in the embodiments of the present application. The computer device 300 may vary greatly due to configuration or performance differences, and may include one or more central processing units (CPUs) 322 (for example, one or more processors) and a memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage media 330 may be transient storage or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the computer device. Further, the central processing unit 322 may be configured to communicate with the storage media 330 and execute a series of instruction operations in the storage media 330 on the computer device 300.

[0219] The computer device 300 may further include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and so on.

[0220] The steps performed by the terminal device in the above embodiments may be based on the Figure 16 shown computer device structure.

[0221] In an embodiment of the present application, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0222] In an embodiment of the present application, a computer program product is further provided, including a computer program. When the computer program is executed by a processor, the steps of the methods described in the foregoing embodiments are implemented.

[0223] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0224] In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0225] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0226] In addition, each functional unit in the various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0227] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store program codes.

[0228] As described above, the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of various embodiments of this application.

Claims

1. A data analysis method, characterized in that, Including: Obtain data to be analyzed, and construct a data analysis model according to the data to be analyzed; In response to a policy configuration addition operation, display a policy configuration interface to receive first edit data input by a user based on the policy configuration interface; Determine decision conditions according to the first edit data; Determine decision rules of the data analysis model according to the decision conditions; Call the data analysis model to determine scores of fields in the data to be analyzed, so as to output warning fields, where the warning fields are fields in the data to be analyzed whose scores meet a preset threshold.

2. The method according to claim 1, wherein Before calling the data analysis model to determine scores of fields in the data to be analyzed so as to output warning fields, the method further includes: Expand fields of the data to be analyzed according to an extension plugin of the data analysis model.

3. The method according to claim 2, wherein After expanding fields of the data to be analyzed according to the extension plugin of the data analysis model, the method further includes: Perform abstraction processing on the data to be analyzed with expanded fields according to aggregation requirements of the data analysis model to obtain first fields; The calling the data analysis model to determine scores of fields in the data to be analyzed so as to output warning fields includes: Call the data analysis model to determine scores of the first fields so as to output the warning fields.

4. The method according to claim 2, wherein Before expanding fields of the data to be analyzed according to the extension plugin of the data analysis model, the method further includes: In response to a preprocessing addition operation, display a preprocessing interface to receive second edit data input by the user based on the preprocessing interface; Determine the extension plugin according to the second edit data.

5. The method according to claim 3, characterized in that, Before performing abstraction processing on the data to be analyzed with expanded fields according to aggregation requirements of the data analysis model to obtain first fields, the method further includes: In response to an abstraction processing addition operation, display an abstraction processing interface to receive third edit data input by the user based on the abstraction processing interface; Determine the aggregation requirements according to the third edit data.

6. The method according to claim 1, characterized in that The determining the decision rules of the data analysis model according to the decision conditions includes: Orchestrate the decision conditions according to an orchestration algorithm to determine the decision rules of the data analysis model, where the orchestration algorithm is the principle of the shortest rule set.

7. The method according to claim 1, wherein Before calling the data analysis model to determine scores of fields in the data to be analyzed so as to output warning fields, the method further includes: Split and execute the decision rules in the data analysis model whose rule complexity exceeds a preset range.

8. The method according to claim 1, wherein Before constructing a data analysis model according to data to be analyzed, the method further includes: In response to a first selection operation on a target data source among multiple data sources, extract the data to be analyzed from the target data source, where the multiple data sources include at least two of a container database, an elastic search database, a distributed data warehouse, a distributed file system, offline data, a distributed message flow processing platform, and a script.

9. The method according to claim 1, wherein The calling the data analysis model to determine scores of fields in the data to be analyzed so as to output warning fields includes: Call the data analysis model to determine the score for the fields that hit the decision rule according to a numerical range or a numerical comparison, where the numerical range is an editable range of the numerical values of the fields of the data to be analyzed.

10. The method according to claim 1, wherein After determining the decision rule of the data analysis model according to the decision condition, the method further includes: Verify the effectiveness of the decision rule by customizing data.

11. The method according to claim 1, characterized in that, The method further includes: Obtain the task cycle; Trigger the step of obtaining the data to be analyzed in a loop according to the task cycle, and construct the data analysis model according to the data to be analyzed.

12. The method according to claim 1, characterized in that, The method further includes: Display the warning fields.

13. The method according to claim 1, characterized in that, After calling the data analysis model to determine the scores of the fields in the data to be analyzed and output the warning fields, the method further includes: Notify the alarm system to issue an alarm.

14. A data analysis device, characterized in that, Includes: A construction unit for obtaining the data to be analyzed and constructing a data analysis model according to the data to be analyzed; A display unit for responding to a policy configuration addition operation, displaying a policy configuration interface to receive first edit data input by a user based on the policy configuration interface; A determination unit for determining a decision condition according to the first edit data; Determine the decision rule of the data analysis model according to the decision condition; A scoring unit for calling the data analysis model to determine the scores of the fields in the data to be analyzed and output warning fields, where the warning fields are the fields in the data to be analyzed whose scores meet a preset threshold.

15. A computer device, characterized in that, Includes: A memory, a transceiver, a processor, and a bus system; Wherein, the memory is used to store programs; The processor is used to execute the programs in the memory, including executing the method according to any one of claims 1 to 13; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

16. A computer-readable storage medium includes instructions that, when running on a computer, cause the computer to execute the method according to any one of claims 1 to 13.

17. A computer program product, characterized in that, When the computer program product is executed on a computer, the computer executes the method according to any one of claims 1 to 13.

Images