Data processing method and device based on multi-source data anomaly detection, equipment and medium

Abstract

The invention discloses a data processing method and device based on multi-source data exception detection, equipment and a medium, and the method comprises the steps: obtaining at least one candidate processing list corresponding to at least one data processing request in response to at least one data processing request generated when at least one data source meets a corresponding exception triggering condition; the at least one data processing request corresponds to the same data processing operation; performing data filtering on the at least one candidate processing list to obtain a target processing list; and executing the data processing operation based on the target processing list. The data processing efficiency and accuracy after the fault can be effectively improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of data processing technology, and in particular to a data processing method, device, equipment and medium based on multi-source data anomaly detection. Background Art

[0002] In online trading scenarios, system failures cannot be completely avoided due to hardware failures, software errors, network problems, man-made external attacks, etc. In traditional technical solutions, developers or maintenance personnel usually write scripts based on the failure conditions to perform maintenance work after the failure. However, this method is inefficient and error-prone when processing large amounts of user data. Summary of the invention

[0003] In view of the above-mentioned defects or deficiencies in the prior art, it is desirable to provide a data processing method, apparatus, device and medium based on multi-source data anomaly detection, which can effectively improve the efficiency and accuracy of data processing after a failure.

[0004] In a first aspect, an embodiment of the present application provides a data processing method based on multi-source data anomaly detection, comprising:

[0005] In response to at least one data processing request generated by at least one data source when a corresponding abnormal trigger condition is satisfied, obtaining at least one candidate processing list corresponding to the at least one data processing request; the at least one data processing request corresponds to the same data processing operation;

[0006] Performing data filtering on the at least one candidate processing list to obtain a target processing list;

[0007] The data processing operation is performed based on the target processing list.

[0008] In some embodiments, before the at least one data processing request generated in response to at least one data source when a corresponding abnormal trigger condition is met, the method further includes:

[0009] For each of the data sources, performing anomaly detection according to anomaly detection rules corresponding to the data source;

[0010] When the abnormality detection result of the data source meets the abnormality triggering condition corresponding to the data source, obtaining the data processing rule corresponding to the abnormality triggering condition;

[0011] Based on the data processing rule, the data processing request is generated.

[0012] In some embodiments, filtering the at least one candidate processing list to obtain a target processing list includes:

[0013] Obtaining a historical processing list and historical processing results corresponding to the historical processing list;

[0014] The at least one candidate processing list is filtered based on the historical processing results to obtain the target processing list.

[0015] In some embodiments, it also includes:

[0016] After executing the data processing operation based on the target processing list, the data processing rules corresponding to the data processing request are optimized based on the processing result.

[0017] In some embodiments, the data processing rule includes a data processing strategy, and optimizing the data processing rule corresponding to the data processing request based on the processing result includes:

[0018] Obtain at least two processing results respectively corresponding to the target processing list after executing the data processing operation according to at least two data processing strategies;

[0019] Comparing the at least two processing results, and determining a preferred processing strategy among the at least two data processing strategies;

[0020] The data processing rules are optimized using the preferred processing strategy.

[0021] In some embodiments, the data processing rules include rules for generating the candidate processing list, and the optimizing the data processing rules corresponding to the data processing request based on the processing result includes:

[0022] Acquire at least two candidate processing lists generated based on generation rules of at least two candidate processing lists corresponding to the same abnormal trigger condition;

[0023] Based on the processing result, determining a preferred generation rule in the at least two candidate processing lists;

[0024] The data processing rules are optimized by utilizing the preferred generation rules.

[0025] In a second aspect, an embodiment of the present application provides a data processing device based on multi-source data anomaly detection, comprising:

[0026] a response module, configured to obtain, in response to at least one data processing request generated by at least one data source when a corresponding abnormal trigger condition is satisfied, at least one candidate processing list corresponding to the at least one data processing request; the at least one data processing request corresponds to the same data processing operation;

[0027] A filtering module, used for filtering data of the at least one candidate processing list to obtain a target processing list;

[0028] A processing module is used to perform the data processing operation based on the target processing list.

[0029] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the embodiment of the present application when executing the program.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the embodiment of the present application.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor, it implements the method described in the embodiment of the present application.

[0032] The data processing method, device, equipment and medium based on multi-source data anomaly detection provided by the embodiments of the present application can respond to at least one data processing request generated by at least one data source when the corresponding anomaly triggering condition is met and at least one corresponding candidate processing list, and filter data on at least one candidate processing list, obtain the target processing list and perform data processing operations, thereby realizing a triggered response to anomalies and effectively improving the response efficiency of data processing operations. At the same time, by filtering data on the candidate processing list, the accuracy of data processing can be effectively improved, and the accuracy of user retrieval can be improved.

[0033] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0035] Figure 1 A schematic diagram of a process of a data processing method based on multi-source data anomaly detection provided by an embodiment of the present application is shown;

[0036] Figure 2 A schematic diagram of a process of a data processing method based on multi-source data anomaly detection provided by another embodiment of the present application is shown;

[0037] Figure 3 A schematic diagram of a process flow of a data processing method based on multi-source data anomaly detection provided by another embodiment of the present application is shown;

[0038] Figure 4 A schematic flow chart of a data processing method based on multi-source data anomaly detection provided by another embodiment of the present application is shown;

[0039] Figure 5 A schematic flow chart of a data processing method based on multi-source data anomaly detection provided by another embodiment of the present application is shown;

[0040] Figure 6 A schematic diagram of a process flow of a data processing method based on multi-source data anomaly detection provided by a specific embodiment of the present application is shown;

[0041] Figure 7 A block diagram of a data processing device based on multi-source data anomaly detection provided by an embodiment of the present application is shown;

[0042] Figure 8 A schematic diagram of the structure of a computer system of an electronic device or server suitable for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0044] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0045] With the popularization of the Internet, online transactions have become a common trading method. However, system failures cannot be completely avoided due to hardware failures, software errors, network problems, external man-made attacks, and other reasons.

[0046] Although traditional fault handling is effective in actual operations, it also has obvious shortcomings. For example, traditional solutions rely on developers or maintenance personnel to write scripts for fault recovery based on the fault situation. This method is inefficient and prone to errors when processing large amounts of user data. Often, when the operation reaches the user, the user has already been lost; or, because system failures usually have a chain reaction, different failures will continuously affect the same user, causing the recovery operation to repeatedly reach the same user and disturb the user.

[0047] Based on this, the present application proposes a data processing method based on multi-source data anomaly detection, which can automatically trigger fault handling operations based on data anomalies, greatly improving the efficiency of fault handling. At the same time, by filtering data on the candidate processing list, it can effectively reduce the repeated processing of the same data, thereby improving the accuracy of data processing while reducing disturbances to users and improving user experience.

[0048] In order to further illustrate the technical solution provided by the embodiment of the present application, this is described in detail below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiment of the present application provides the method operation instruction steps shown in the following embodiments or drawings, more or less operation instruction steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided by the embodiment of the present application. The method may be executed in the order of the method shown in the embodiment or drawings or in parallel during the actual processing process or when the device is executed.

[0049] Please refer to Figure 1 , Figure 1 FIG. 1 is a flow chart of a data processing method based on multi-source data anomaly detection provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0050] Step 101 , in response to at least one data processing request generated by at least one data source when a corresponding abnormal triggering condition is satisfied, obtaining at least one candidate processing list corresponding to the at least one data processing request.

[0051] Wherein, at least one data processing request corresponds to the same data processing operation. Optionally, the same data processing operation is a data retrieval operation, and preferably, the same data processing operation is a user retrieval operation.

[0052] It should be noted that at least one data source is at least one data collection node set up on the online platform, which is used to collect real-time data at at least one data collection node on the online platform. At least one data source can be set at different stages in the same business process. The data collected by at least one data source includes but is not limited to system logs, business orders at each node, user behavior, etc. It should be understood that at least one data source can be set at an important node on the online platform, can also be set at a high-frequency failure node, and can also be set according to data processing requirements, and this application does not make specific restrictions.

[0053] It should also be noted that if any data source meets the corresponding abnormal triggering condition, it means that the data source is abnormal and prone to failures caused by the abnormality, such as payment failure, insufficient inventory, etc. caused by the extended response time due to system performance degradation. Further, a corresponding data processing request is generated when any data source meets the corresponding abnormal triggering condition.

[0054] That is to say, in an embodiment of the present application, when any data source meets the corresponding abnormal trigger condition, a data processing request will be generated, and when at least one data source meets the corresponding abnormal trigger condition, at least one corresponding data processing request will be generated, and a corresponding candidate processing list can be obtained according to each data processing request.

[0055] It should be noted that the candidate processing list is a list of data to be restored, optionally, the candidate processing list is a list of data to be salvaged, and preferably, the candidate processing list is a list of users to be salvaged. That is to say, the data processing request generated by the data source abnormality trigger in the embodiment of the present application can determine the list of users affected by the fault. In order to reduce the bad experience of users caused by the fault and reduce the business willingness reduced by the fault, the user list is used as the salvage user list to guide the user to return and increase the business willingness through the subsequent salvage data processing operation.

[0056] Among them, the candidate processing list corresponds one-to-one to the data processing request, that is, the candidate processing list corresponds one-to-one to the exception triggering mechanism. In other words, in an embodiment of the present application, the same data source can generate multiple different data processing requests according to different exception triggering conditions, and each data processing request corresponds to a candidate processing list.

[0057] Step 102: Filter data on at least one candidate processing list to obtain a target processing list.

[0058] It should be noted that after an exception occurs, during the data processing process, the same data may appear in multiple candidate processing lists due to a cascading failure, or the operated data may still appear in at least one candidate processing list due to a time deviation. At this time, it is easy to operate the same data repeatedly, which seriously affects the efficiency of the overall data processing. When the candidate processing list is a user list to be retrieved, it is also easy to disturb the user due to repeated operations. Based on this, before executing the data processing operation, the embodiment of the present application filters the data of at least one candidate processing list to obtain the target processing list, so as to reduce redundant list data, thereby improving the efficiency of the overall data processing operation, and when the candidate processing list is a user list to be retrieved, it reduces the disturbance to the user and improves the user experience.

[0059] Step 103, executing data processing operations based on the target processing list.

[0060] It should be noted that when the data processing operation is a user retrieval operation, the data processing operation includes but is not limited to sending reminder information to users in the target processing list to inform the users that the anomaly has been fixed and they can continue the operation.

[0061] Therefore, the data processing based on multi-source data anomaly detection provided by the embodiment of the present application can respond to at least one data processing request generated by at least one data source when the corresponding anomaly triggering condition is met and at least one corresponding candidate processing list, and filter the data of at least one candidate processing list, obtain the target processing list and perform data processing operations, thereby realizing a triggered response to anomalies and effectively improving the response efficiency of data processing operations. At the same time, by filtering the data of the candidate processing list, the accuracy of data processing can be effectively improved, and the accuracy of user retrieval can be improved.

[0062] In a possible embodiment, if Figure 2 As shown, before responding to at least one data processing request generated by at least one data source when a corresponding abnormal triggering condition is met, it also includes:

[0063] Step 201 : For each data source, anomaly detection is performed according to anomaly detection rules corresponding to the data source.

[0064] It should be noted that each data source may correspond to at least one anomaly detection rule. Specifically, data at a data source node is collected in real time, and the data is detected according to at least one anomaly detection rule to obtain an anomaly detection result.

[0065] Among them, the abnormal detection rules include but are not limited to abnormal behavior detection, transaction status detection and node performance detection, etc. Exemplarily, abnormal behavior detection can be the detection of the continuous state of any node, such as frequent feedback of login failures, large number of invalid orders submitted, etc.; transaction status detection can be the detection of the state of the transaction node, such as payment failure, order cancellation caused by insufficient inventory, etc.; node performance detection can be the detection of the response time and error rate of any node, such as extended response time, increased error rate, etc.

[0066] Step 202: When the data source anomaly detection result meets the anomaly triggering condition corresponding to the data source, a data processing rule corresponding to the anomaly triggering condition is obtained.

[0067] Specifically, after obtaining real-time data from a data source, anomaly detection is performed on the real-time data according to at least one anomaly detection rule corresponding to the data source to obtain at least one anomaly detection result, and then it is further determined whether the anomaly detection result meets the anomaly trigger condition corresponding to the data source. If so, the data processing rule corresponding to the anomaly trigger condition is obtained; if not, real-time data is continued to be obtained for anomaly detection.

[0068] It should be noted that the data processing rules include the generation rules of the candidate processing list and the data processing strategy. Among them, the generation rules of the candidate processing list are used to generate the candidate processing list corresponding to the abnormal triggering condition, and form a corresponding relationship with the data processing request based on the abnormal triggering condition. The data processing strategy is used to perform corresponding data processing operations on the candidate processing list.

[0069] Step 203: Generate a data processing request based on the data processing rule.

[0070] Optionally, the data processing request includes data processing rules. Specifically, the data processing request includes generation rules of a candidate processing list and a data processing strategy.

[0071] That is to say, when the anomaly detection result corresponding to the real-time data of the data source meets the anomaly trigger condition, the generation rules and data processing strategies of the candidate processing list corresponding to the anomaly trigger condition are obtained, and then a data processing request is generated based on the generation rules and data processing strategies of the candidate processing list, so as to obtain the corresponding candidate processing list according to the data processing request, and perform data processing operations after obtaining the target processing list.

[0072] Therefore, the embodiment of the present application can automatically generate data processing requests according to pre-set anomaly detection rules and anomaly trigger conditions without human intervention, greatly improving data processing efficiency, providing reliable timeliness guarantee for user retrieval, and effectively increasing user return volume.

[0073] In a possible embodiment, if Figure 3 As shown, step 102, filtering data on at least one candidate processing list to obtain a target processing list, includes:

[0074] Step 1021, obtaining a historical processing list and the historical processing results corresponding to the historical processing list.

[0075] Step 1022: Filter data on at least one candidate processing list based on historical processing results to obtain a target processing list.

[0076] It should be noted that the historical processing list is a data processing list corresponding to the data processing request triggered by the previous exception. Optionally, the exception trigger corresponding to the historical processing list is a historical processing list within a preset time before the current exception trigger, that is, the exception trigger corresponding to the historical processing list may be associated with the current exception trigger, such as upstream and downstream exceptions. Therefore, in order to avoid repeated contact with users due to chain exceptions and cause disturbance to users, the embodiment of the present application proposes to obtain a historical processing list and its corresponding historical processing results, and use the historical processing results to filter data on at least one candidate processing list to obtain a target processing list.

[0077] It should be understood that the historical processing result is the data information that has been reflowed, and preferably, the historical processing result is the user data that has been reflowed. In other words, the data processing operation corresponding to the previous historical processing list has caused the user to reflow. In order to avoid disturbing the user, the user data is filtered in the current exception so that it does not appear in the target processing list, thereby achieving the goal of not reaching the user again in the data processing operation triggered by the current exception.

[0078] In some feasible embodiments, in the process of filtering the data of the candidate processing list, in addition to filtering the returned customers, it also includes filtering the repeated data in at least one candidate processing list, so that only one of the repeated data is retained in the target processing list to avoid repeated operations on the same data. In addition, since the real-time data of at least one data source is collected in the embodiment of the present application, when filtering the data of at least one candidate processing list, data analysis can also be performed based on the real-time data of at least one data source to determine the willingness of each user to continue the business. For example, the willingness to continue the business evaluation index corresponding to each user is calculated based on the real-time data. If the willingness to continue the business evaluation index reaches the preset threshold, it means that the user has a high willingness to continue the business, so the user data is retained. If the willingness to continue the business evaluation index does not reach the color number threshold, it means that the user does not have a high willingness to continue the business. Therefore, in order to avoid disturbing the user, the user data is filtered.

[0079] Before analyzing the real-time data, data processing such as data cleaning and standardization may be performed on the real-time data from different data sources to improve the accuracy of the data analysis.

[0080] Therefore, the data processing method based on anomaly detection provided in the embodiment of the present application can effectively improve the data quality in the target processing list by filtering data on at least one candidate processing list, provide accurate data support for subsequent data processing operations, and effectively avoid disturbing returned users or low-willing customers, thereby improving user experience.

[0081] In a feasible embodiment, in order to further optimize the reliability of data processing rules, improve the accuracy of data processing, and promote effective user return, the embodiment of the present application further proposes to optimize the data processing rules corresponding to the data processing request based on the processing results after executing the data processing operation in the target processing list.

[0082] Specifically, in some embodiments, for the data processing strategy in the data processing rule, based on the data processing result, the data processing rule corresponding to the data processing request is optimized, such as Figure 4 As shown, including:

[0083] Step 401, obtaining at least two processing results corresponding to the target processing list after executing data processing operations according to at least two data processing strategies.

[0084] It should be noted that the target processing list for executing data processing operations according to at least two data processing strategies may come from the same type of abnormal triggers or from different types of abnormal triggers. The processing result is valid data after the data processing operation is performed using the corresponding data processing strategy. Optionally, the processing result is valid retrieved data. Preferably, the processing result is user return flow data, including but not limited to user return flow.

[0085] Step 402: Compare at least two processing results and determine a preferred processing strategy among at least two data processing strategies.

[0086] Among them, the preferred processing strategy is a data processing strategy with more valid data after the data processing operation. Optionally, the preferred processing strategy is a data processing strategy with more valid retrieved data. Preferably, the preferred processing strategy is a data processing strategy with higher user return data.

[0087] Step 403, optimize the data processing rules using the optimal processing strategy.

[0088] Optionally, the data processing rules can be optimized by adjusting the weight of the preferred processing strategy. That is, when the data processing rule determines the data processing strategy corresponding to the current abnormal trigger according to the weight value of each preferred processing strategy, by adjusting the weight value corresponding to the preferred processing strategy, for example, increasing the weight value of the preferred processing strategy, the possibility of the preferred processing strategy being determined as the data processing strategy corresponding to the abnormal trigger can be effectively increased, thereby achieving an increase in the effective data volume of data processing.

[0089] It should be understood that when the target processing list is from the same type of abnormal trigger, the preferred processing strategy can effectively reflect the user return effect of the abnormal trigger, and the use of the preferred processing strategy in the post-abnormal trigger can further improve the return effect of users affected by the abnormality. When the target processing list comes from different types of abnormal triggers, the preferred processing strategy can effectively reflect the return effect of platform users, and the use of the preferred processing strategy in the post-abnormal trigger can further improve the return effect of the platform's overall users.

[0090] Therefore, in the embodiment of the present application, by comparing and analyzing the execution results of different data processing strategies and then optimizing the data processing rules, the data processing rules can use the preferred processing strategy as much as possible, thereby increasing the amount of effective data for subsequent data processing.

[0091] In other embodiments, for the generation rule of the candidate processing list in the data processing rule, the data processing rule corresponding to the data processing request is optimized based on the data processing result, such as Figure 5 As shown, including:

[0092] Step 501 : obtaining at least two candidate processing lists generated based on generation rules of at least two candidate processing lists corresponding to the same abnormal triggering condition.

[0093] Step 502, based on the processing result, determine the preferred generation rules in at least two candidate processing lists.

[0094] Among them, the preferred generation rule is a generation rule with more valid data after the data processing operation. Optionally, the preferred generation rule is a generation rule with more valid retrieved data in the candidate processing list. Preferably, the preferred generation rule is a generation rule with higher user return data in the candidate processing list.

[0095] Step 503, optimizing data processing rules by using the optimal generation rules.

[0096] Optionally, the data processing rules can be optimized by adjusting the weights of the preferred generation rules. That is, when the data processing rules determine the generation rules of the candidate processing list corresponding to the current abnormal trigger according to the weight values ​​of the preferred generation rules, by adjusting the weight values ​​corresponding to the preferred generation rules, for example, increasing the weight values ​​of the preferred generation rules, the possibility of the preferred generation rules being determined as the generation rules of the candidate processing list corresponding to the abnormal trigger can be effectively increased, thereby achieving an increase in the effective data volume of data processing.

[0097] It should be understood that when the exception triggering conditions are the same, different candidate processing lists are generated using different candidate processing list generation rules. If the final processing results of different candidate processing lists are different, it reflects the common characteristics of the reflowing users affected by this type of exception. Therefore, using the preferred generation rules when the exception is triggered later can further improve the reflow effect of users affected by the exception.

[0098] It should also be understood that since the exception triggers are based on real-time data from different data sources, the amount of data or data dimensions on which the generation rules of the candidate processing list rely are not the same. Therefore, it is not appropriate to conduct comparative analysis on the candidate processing lists triggered by different types of exceptions, so as to avoid the situation where some exceptions based on the same generation rules cannot obtain the candidate processing list that meets the generation rules.

[0099] In a specific embodiment, Figure 6 As shown, the data processing method based on anomaly detection provided by the embodiment of the present application is divided into three stages, namely, the data collection stage, the data analysis stage and the data processing stage. In the data collection stage, real-time data from multiple data sources are obtained, including but not limited to system logs, order data, user behavior data, buried point data, etc., and then the data analysis stage performs anomaly detection on the real-time data of different data sources, such as error detection on system logs, state anomaly detection on order data, and behavior anomaly detection on user behavior data and buried point data. Based on anomaly detection, the corresponding candidate processing lists are generated respectively, and then the target processing list, that is, the target retrieval list, is obtained by filtering. The data processing stage executes the user retrieval operation, obtains the retrieval result, and returns the retrieval result to the data analysis stage for iterative optimization of the data processing rules.

[0100] In summary, the data processing method based on anomaly detection provided by the embodiment of the present application can respond to at least one data processing request generated by at least one data source when the corresponding anomaly triggering condition is met and at least one corresponding candidate processing list, and filter the data of at least one candidate processing list, obtain the target processing list and perform data processing operations, thereby realizing a triggered response to anomalies and effectively improving the response efficiency of data processing operations. At the same time, by filtering the data of the candidate processing list, the accuracy of data processing can be effectively improved, and the accuracy of user retrieval can be improved.

[0101] It should be noted that although the operations of the method of the present invention are described in a particular order in the drawings, this does not require or imply that the operations must be performed in this particular order or that all illustrated operations must be performed to achieve desired results.

[0102] Figure 7A block diagram of a data processing device based on multi-source data anomaly detection provided by an embodiment of the present application is shown.

[0103] like Figure 7 As shown, the data processing device 10 based on multi-source data anomaly detection includes:

[0104] A response module 11, configured to obtain, in response to at least one data processing request generated by at least one data source when a corresponding abnormal trigger condition is met, at least one candidate processing list corresponding to the at least one data processing request; the at least one data processing request corresponds to the same data processing operation;

[0105] A filtering module 12, configured to filter data of the at least one candidate processing list to obtain a target processing list;

[0106] The processing module 13 is used to perform the data processing operation based on the target processing list.

[0107] In some embodiments, the response module 11 is further configured to:

[0108] For each of the data sources, performing anomaly detection according to anomaly detection rules corresponding to the data source;

[0109] When the abnormality detection result of the data source meets the abnormality triggering condition corresponding to the data source, obtaining the data processing rule corresponding to the abnormality triggering condition;

[0110] Based on the data processing rule, the data processing request is generated.

[0111] In some embodiments, the filtering module 12 is further configured to:

[0112] Obtaining a historical processing list and historical processing results corresponding to the historical processing list;

[0113] The at least one candidate processing list is filtered based on the historical processing results to obtain the target processing list.

[0114] In some embodiments, the processing module 13 is further configured to:

[0115] After executing the data processing operation based on the target processing list, the data processing rules corresponding to the data processing request are optimized based on the processing result.

[0116] In some embodiments, the data processing rules include a data processing strategy, and the processing module 13 is further used to:

[0117] Obtain at least two processing results respectively corresponding to the target processing list after executing the data processing operation according to at least two data processing strategies;

[0118] Comparing the at least two processing results, and determining a preferred processing strategy among the at least two data processing strategies;

[0119] The data processing rules are optimized using the preferred processing strategy.

[0120] In some embodiments, the data processing rules include the generation rules of the candidate processing list, and the processing module 13 is further used to:

[0121] Acquire at least two candidate processing lists generated based on generation rules of at least two candidate processing lists corresponding to the same abnormal trigger condition;

[0122] Based on the processing result, determining a preferred generation rule in the at least two candidate processing lists;

[0123] The data processing rules are optimized by utilizing the preferred generation rules.

[0124] It should be understood that the modules or modules described in the data processing device 10 based on multi-source data anomaly detection are similar to those described in the reference Figure 1 The various steps in the described method correspond to each other. Therefore, the operations and features described above for the method are also applicable to the data processing device 10 based on multi-source data anomaly detection and the modules contained therein, and will not be repeated here. The data processing device 10 based on multi-source data anomaly detection can be pre-implemented in a browser or other security application of an electronic device, or can be loaded into the browser or its security application of an electronic device by downloading or the like. The corresponding modules in the data processing device 10 based on multi-source data anomaly detection can cooperate with the modules in the electronic device to implement the solution of the embodiment of the present application.

[0125] For the several modules or units mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0126] Reference below Figure 8 , Figure 8 A schematic diagram of the structure of a computer system of an electronic device or server suitable for implementing an embodiment of the present application is shown.

[0127] like Figure 8As shown, the computer system includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 into the random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation instructions of the system are also stored. The CPU 801, the ROM 802 and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0128] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0129] In particular, according to an embodiment of the present application, the above reference flow chart Figure 2 The described process can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present application are executed.

[0130] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium such as a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operating instructions of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the aforementioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operating instruction, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The units or modules involved in the embodiments described in the present application may be implemented by software or by hardware. The units or modules described may also be arranged in a processor, for example, they may be described as: a processor includes a response module, a filtering module and a processing module. Among them, the names of these units or modules do not constitute a limitation on the units or modules themselves under certain circumstances. For example, the response module may also be described as "responding to at least one data processing request generated by at least one data source when a corresponding abnormal trigger condition is met, obtaining at least one candidate processing list corresponding to the at least one data processing request; the at least one data processing request corresponds to the same data processing operation".

[0133] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the data processing method based on multi-source data anomaly detection described in the present application.

[0134] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A data processing method based on multi-source data anomaly detection, characterized in that: include: In response to at least one data processing request generated by at least one data source when a corresponding abnormal trigger condition is satisfied, obtaining at least one candidate processing list corresponding to the at least one data processing request; The at least one data processing request corresponds to the same data processing operation; Performing data filtering on the at least one candidate processing list to obtain a target processing list; The data processing operation is performed based on the target processing list.

2. The data processing method based on multi-source data anomaly detection according to claim 1 is characterized in that: Before the at least one data processing request generated in response to at least one data source when a corresponding abnormal triggering condition is met, the method further includes: For each of the data sources, performing anomaly detection according to anomaly detection rules corresponding to the data source; When the abnormality detection result of the data source meets the abnormality triggering condition corresponding to the data source, obtaining the data processing rule corresponding to the abnormality triggering condition; Based on the data processing rule, the data processing request is generated.

3. The data processing method based on multi-source data anomaly detection according to claim 1 is characterized in that: The filtering of the at least one candidate processing list to obtain a target processing list includes: Obtaining a historical processing list and historical processing results corresponding to the historical processing list; The at least one candidate processing list is filtered based on the historical processing results to obtain the target processing list.

4. The data processing method based on multi-source data anomaly detection according to claim 1 is characterized in that: Also includes: After executing the data processing operation based on the target processing list, the data processing rules corresponding to the data processing request are optimized based on the processing result.

5. The data processing method based on multi-source data anomaly detection according to claim 4 is characterized in that: The data processing rule includes a data processing strategy, and the optimizing the data processing rule corresponding to the data processing request based on the processing result includes: Obtain at least two processing results respectively corresponding to the target processing list after executing the data processing operation according to at least two data processing strategies; Comparing the at least two processing results, and determining a preferred processing strategy among the at least two data processing strategies; The data processing rules are optimized using the preferred processing strategy.

6. The data processing method based on multi-source data anomaly detection according to claim 4 is characterized in that: The data processing rules include generation rules of the candidate processing list, and the optimizing the data processing rules corresponding to the data processing request based on the processing result includes: Acquire at least two candidate processing lists generated based on generation rules of at least two candidate processing lists corresponding to the same abnormal trigger condition; Based on the processing result, determining a preferred generation rule in the at least two candidate processing lists; The data processing rules are optimized by utilizing the preferred generation rules.

7. A data processing device based on multi-source data anomaly detection, characterized in that: include: a response module, configured to, in response to at least one data processing request generated by at least one data source when a corresponding abnormal trigger condition is satisfied, obtain at least one candidate processing list corresponding to the at least one data processing request; The at least one data processing request corresponds to the same data processing operation; A filtering module, used for filtering data of the at least one candidate processing list to obtain a target processing list; A processing module is used to perform the data processing operation based on the target processing list.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the data processing method based on multi-source data anomaly detection as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data processing method based on multi-source data anomaly detection as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method based on multi-source data anomaly detection described in any one of claims 1 to 6 is implemented.

Images