Data processing

Abstract

Embodiments of the present specification provide a data processing method and apparatus, a terminal, a computer-readable storage medium, and a computer program product. The method comprises: a feature checking system generating a probe factor on the basis of offline full features and a table association corresponding thereto; an online feature extraction system performing feature extraction on online incremental data on the basis of the probe factor, so as to obtain online incremental features; and the feature checking system performing feature matching between the offline full features and the online incremental features to obtain a matching result, determining an abnormal feature from the online incremental features on the basis of the matching result, and outputting alarm information regarding the abnormal feature.

Description

Data processing Technical Field

[0001] The embodiments of this specification relate to the field of data processing technology, and in particular, to data processing methods, devices, terminals, computer-readable storage media, and computer program products. Background Art

[0002] In the financial business scenario, business data can be divided into full data and incremental data. Full data generally refers to historical business data, and incremental data generally refers to new business data generated on the day.

[0003] Business data has a significant impact on public opinion risk and asset loss risk in financial services, so it is necessary to properly verify and process both full and incremental data. However, the processing of business data in related technologies is not yet perfect.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this specification, and therefore may include information that does not constitute prior art known to ordinary technicians in this field.

[0005] Summary of the Invention

[0006] The embodiments of this specification provide a data processing method, apparatus, terminal, computer-readable storage medium, and computer program product, which can match and analyze offline full features and online incremental features, thereby improving the accuracy of online incremental data.

[0007] Other features and advantages of the embodiments of the present specification will become apparent from the following detailed description, or may be learned in part from the practice of the present specification.

[0008] According to the first aspect of the embodiments of this specification, a data processing method is provided, which includes: generating a detection factor based on offline full features and their corresponding table association relationships through a feature verification system; extracting features from online incremental data based on the above-mentioned detection factors through an online feature extraction system to obtain online incremental features; performing feature matching on the above-mentioned offline full features and the above-mentioned online incremental features through the above-mentioned feature verification system to obtain matching results; and determining abnormal features from the above-mentioned online incremental features based on the above-mentioned matching results, and outputting alarm information about the above-mentioned abnormal features.

[0009] In one embodiment of the present specification, before generating the exploration factor based on the offline full features and their corresponding table association relationships, the method further includes: performing feature extraction on the offline full data through an offline feature extraction system to obtain a first full feature; marking the first full feature through the feature verification system to filter out a second full feature from the first full feature, wherein the second full feature meets the first preset standard; judging whether the second full feature meets the second preset standard through the feature verification system; if the second full feature meets the second preset standard, determining the second full feature as the offline full feature through the feature verification system and storing it in the feature storage module.

[0010] In one embodiment of the present specification, before the offline feature extraction system is used to perform feature extraction on the offline full data to obtain the first full feature, the method further includes: configuring the basic information of the offline full data through the feature verification system, and inputting the basic information of the offline full data into the offline feature extraction system, wherein the basic information includes one or more of the table name, table fields and the table association relationship; obtaining the offline full data and the basic information of the offline full data through the offline feature extraction system, and performing feature extraction on the offline full data according to the basic information of the offline full data to obtain the first full feature.

[0011] In one embodiment of the present specification, after extracting features from the online incremental data according to the above-mentioned detection factors to obtain online incremental features, the above-mentioned method further includes: storing the above-mentioned online incremental feature data in the above-mentioned feature storage module through the online feature extraction system.

[0012] In one embodiment of the present specification, before generating the exploration factor based on the offline full features and their corresponding table association relationships, the method further includes: obtaining target features from the feature storage module through the feature verification system; and classifying the target features through the feature verification system to determine the offline full features and the online incremental features from the target features.

[0013] In one embodiment of the present specification, the feature matching of the offline full features and the online incremental features by the feature checking system includes: the feature matching of the offline full features and the online incremental features by the feature checking system to determine whether the online incremental features are a subset of the offline full features; the determining of abnormal features from the online incremental features based on the matching results includes: determining the abnormal features from the online incremental features when the matching result is that the online incremental features are not a subset of the offline full features.

[0014] In one embodiment of the present specification, after determining the abnormal feature from the online incremental feature based on the matching result, the method further includes: judging whether the abnormal feature complies with a preset rule; outputting the alarm information about the abnormal feature includes: outputting the alarm information about the abnormal feature through the feature verification system when the abnormal feature does not comply with the preset rule.

[0015] In one embodiment of the present specification, when the above-mentioned abnormal feature does not comply with the above-mentioned preset rules, before the above-mentioned feature verification system outputs the alarm information about the above-mentioned abnormal feature, the above-mentioned method further includes: judging whether the priority of the above-mentioned abnormal feature is lower than the preset priority through the above-mentioned feature verification system; the above-mentioned output of the alarm information about the above-mentioned abnormal feature through the above-mentioned feature verification system includes: when the priority of the above-mentioned abnormal feature is lower than the above-mentioned preset priority, outputting the alarm information about the above-mentioned abnormal feature through the above-mentioned feature verification system; after the above-mentioned judgment of whether the above-mentioned abnormal feature complies with the preset rules, the above-mentioned method further includes: when the priority of the above-mentioned abnormal feature is not lower than the above-mentioned preset priority, performing service fuse through the above-mentioned feature verification system.

[0016] According to a second aspect of an embodiment of this specification, a data processing device is provided, which includes: a generation module for: generating a detection factor based on offline full features and their corresponding table association relationships through a feature verification system; a feature extraction module for: performing feature extraction on online incremental data based on the above-mentioned detection factors through an online feature extraction system to obtain online incremental features; a matching module for: performing feature matching on the above-mentioned offline full features and the above-mentioned online incremental features through the above-mentioned feature verification system to obtain matching results; and, determining abnormal features from the above-mentioned online incremental features based on the above-mentioned matching results, and outputting alarm information about the above-mentioned abnormal features.

[0017] According to the third aspect of the embodiments of this specification, a terminal is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method described in the first aspect when executing the computer program.

[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data processing method described in the first aspect is implemented.

[0019] According to a fifth aspect of the embodiments of this specification, a computer program product is provided. When the computer program product is run on a computer or a processor, the computer or processor implements the data processing method described in the first aspect when executing the computer or processor.

[0020] The data processing method, device, terminal, computer-readable storage medium, and computer program product provided in the embodiments of this specification have the following technical effects.

[0021] The solution provided by the exemplary embodiment of this specification is suitable for checking online incremental features to determine whether there are abnormal features in the online incremental features and to issue an alarm for the abnormal features.

[0022] Specifically, the feature verification system generates exploration factors based on offline full features and their corresponding table relationships. The online feature extraction system extracts features from online incremental data based on the exploration factors, generating online incremental features. The feature verification system then matches the offline full features with the online incremental features to obtain matching results. Based on the matching results, the system identifies abnormal features from the online incremental features and outputs warnings regarding these abnormal features. This improves the accuracy of online incremental data and reduces the risk of asset losses in financial services.

[0023] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are incorporated into and constitute a part of this specification, illustrate embodiments consistent with this specification, and together with the specification, are used to explain the principles of this specification. Obviously, the drawings described below are only some embodiments of this specification, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0025] FIG1 is a flow chart of a data processing method provided by an exemplary embodiment of this specification;

[0026] FIG2 is a flowchart of determining offline full-volume features provided by an exemplary embodiment of this specification;

[0027] FIG3 is an architecture diagram of a data processing method provided in an exemplary embodiment of this specification;

[0028] FIG4 is an architecture diagram of a data processing method provided in another exemplary embodiment of this specification;

[0029] FIG5 is a flow chart of feature checking provided by an exemplary embodiment of this specification;

[0030] FIG6 is an architecture diagram of a data processing method provided in yet another exemplary embodiment of this specification;

[0031] FIG7 is a structural diagram of a data processing device provided in an exemplary embodiment of this specification;

[0032] FIG8 is a structural diagram of a data processing device provided by another exemplary embodiment of this specification;

[0033] FIG9 is a schematic block diagram of a terminal provided by an exemplary embodiment of this specification. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of this specification more clear, the embodiments of this specification will be further described in detail below with reference to the accompanying drawings.

[0035] When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Instead, they are merely examples of devices and methods consistent with certain aspects of this specification, as detailed in the appended claims.

[0036] In the description of this specification, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. For those skilled in the art, the specific meanings of the above terms in this specification can be understood according to specific circumstances. In addition, in the description of this specification, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0037] In financial business scenarios, every decimal point in business data can potentially represent millions of dollars in financial losses. Furthermore, financial business demands extremely high timeliness in problem discovery. Once a problem is discovered in business data, it must be resolved as quickly as possible. Every day of delay can result in hundreds of points of public opinion risk and tens of thousands of dollars in financial losses.

[0038] The embodiments of this specification provide a data processing method, device, terminal, computer-readable storage medium, and computer program product that can improve the accuracy of online incremental data and reduce the public opinion risk and capital loss risk of financial services.

[0039] FIG1 shows a flow chart of a data processing method provided in an embodiment of this specification.

[0040] S110 , generating exploration factors based on the offline full features and their corresponding table association relationships through the feature verification system.

[0041] Figure 2 shows a flow chart for determining offline full-data features according to an embodiment of this specification, and Figure 3 shows an architecture diagram of a data processing method according to an embodiment of this specification. The method steps shown in Figure 2 can be applied to the architecture diagram shown in Figure 3.

[0042] S210: Perform feature extraction on the offline full data through the offline feature extraction system to obtain a first full feature.

[0043] In an exemplary embodiment, as shown in FIG3 , a business expert can configure basic information for the offline full amount of data that needs to be subjected to feature extraction / feature mining in the feature checking system 31 .

[0044] Among them, the offline full data can be stored in the form of data tables in an offline database (for example, Open Data Processing Service (ODPS), etc.). The basic information of the offline full data may include table name, table fields, and table association relationships.

[0045] In an exemplary embodiment, after the configuration of the basic information is completed, the feature checking system 31 inputs the basic information into the offline feature extraction system 30 .

[0046] Next, the offline feature extraction system 30 associates the table fields between different data tables according to the above basic information, so as to complete the basic offline full-data modeling according to the association relationship between the fields.

[0047] Afterwards, the associated data tables are merged to generate an offline full data collection based on the field associations. Finally, based on this offline full data collection, the fields are analyzed to extract / mine effective features from the offline full data. Based on these features, a feature enumeration set (i.e., the set obtained by enumerating the feature values) is generated. This set is hereinafter referred to as the first full feature.

[0048] S220 , annotating the first full quantity feature through a feature checking system to filter out a second full quantity feature from the first full quantity feature, wherein the second full quantity feature meets a first preset standard.

[0049] In an exemplary embodiment, referring to Figure 3 , the offline feature extraction system 30 may synchronize the obtained first full feature to the offline full feature database, and the feature verification system 31 may obtain (derive) the first full feature from the offline full feature database.

[0050] Afterwards, the business expert can use the feature verification system 31 to annotate the first full feature, thereby filtering out invalid features and confirming the integrity of the features, ultimately obtaining the second full feature. In other words, the first full feature that meets the first preset standard is used as the second full feature. The first preset standard may be that the first full feature is a valid feature and has a certain degree of integrity.

[0051] During this process, the feature verification system 31 can provide algorithm recommendations related to feature matching and feature filtering to assist business experts in making decisions. Furthermore, the feature verification system 31 can also use a correction algorithm based on historical annotation logic to provide correction assistance and recommend unlabeled features for the unlabeled first full set of features.

[0052] S230: Determine whether the second full feature meets the second preset standard through the feature verification system.

[0053] S240: When the second full feature meets the second preset standard, the feature verification system determines the second full feature as an offline full feature and stores it in the feature storage module.

[0054] In the exemplary embodiment, referring to Figure 3, the second full quantity feature may exist in two situations.

[0055] The first case is that the second full feature meets the second preset criterion. The second preset criterion is that the second full feature reaches a final state, that is, all invalid, low-frequency, or low-service-value features are no longer present in the feature, and the core feature fields are retained.

[0056] The second situation is that the second full feature does not meet the second preset standard. In this case, the offline feature extraction system 30 is required to perform the next round of feature extraction / mining and filtering until the second full feature meets the second preset standard, that is, reaches the final state.

[0057] Exemplarily, in the first case, the second full feature can be stored in the feature storage module 32 as the final offline full feature through the feature checking system 31 .

[0058] Exemplarily, in the second case, the second full feature can be stored in the offline feature database and feature storage module 32 through the feature verification system 31, so that the offline feature extraction system 30 can subsequently obtain the second full feature from the feature storage module 32, and perform feature extraction / feature mining on the second full feature again. Thereafter, the extracted / mined features are manually labeled by the feature verification system 31 to filter the above features again, and so on, until the features obtained after manual labeling meet the second preset standards.

[0059] FIG4 shows an architecture diagram of a data processing method provided in another embodiment of this specification.

[0060] In an exemplary embodiment, as shown in FIG4 , the feature verification system 31 can then periodically initiate online feature exploration of the online incremental data. During the online feature exploration process, automated means can be used to understand the content, background, structure, and path of the online incremental data, examine the composition, relationships, and format of the online incremental data, and perform feature extraction on the online incremental data to obtain online incremental features.

[0061] For example, online incremental data can include business data in online databases and online message data. Business data in online databases is fresh, incremental business data generated daily, while online message data is real-time interaction data generated during interactions between business systems. Combining these two as online incremental data ensures that all newly added business data from that day is included in online feature exploration.

[0062] Exemplarily, before performing online feature detection, the feature checking system 31 may obtain the above-mentioned offline full features from the feature storage module 32, and generate detection factors based on the offline full features and their corresponding table associations.

[0063] S120 , performing feature extraction on the online incremental data according to the exploration factor through the online feature extraction system to obtain online incremental features.

[0064] In an exemplary embodiment, referring to Figure 4 , the online feature extraction system 33 may then perform online feature detection on the online incremental data based on the generated detection factors, thereby further extracting features from the online incremental data to obtain online incremental features, which include effective features of the daily incremental data.

[0065] Finally, the online feature extraction system 33 can mark the generation time and generation type of the online incremental features. The generation time can be the specific source time of the online incremental data, and the generation type is online incremental. The online incremental features are further stored in the feature storage module 32.

[0066] S130, through the feature verification system, feature matching is performed on the offline full features and the online incremental features to obtain matching results; and, based on the matching results, abnormal features are determined from the online incremental features, and alarm information about the abnormal features is output.

[0067] FIG5 shows a flow chart of feature checking provided by an embodiment of this specification.

[0068] S510: Perform feature matching on the offline full features and the online incremental features through the feature verification system.

[0069] S520: Determine whether the online incremental features are a subset of the offline full features through the feature verification system.

[0070] S530: When the matching result shows that the online incremental feature is not a subset of the offline full feature, an abnormal feature is determined from the online incremental feature by a feature checking system.

[0071] In an exemplary embodiment, for a financial business, if the business itself has not changed, then after the business has been running for a long time, the online incremental features extracted from the online incremental data should be a subset of the offline full features.

[0072] If features that do not appear in the offline full features appear in the online incremental features, there are usually two situations: the first situation is that the offline full features are limited and do not cover all feature information; the second situation is that abnormal features appear.

[0073] For example, suppose the offline full-data features for repayment currencies only include RMB, Hong Kong Dollar, and Macau Pataca, but feature extraction of the online incremental data for a particular day reveals that repayment currencies also include US Dollars. This could be because the repayment currency already supports US Dollar repayments (the first scenario above), or because the US Dollar is an unusual currency and the repayment was originally calculated as RMB (the second scenario above). This could result in financial losses for the financial company.

[0074] In an exemplary embodiment, to prevent abnormal features from affecting financial services, abnormal data can be identified and alerted through the feature verification system 31. The feature verification process is described in detail below.

[0075] Exemplarily, first, the feature checking system 31 may periodically obtain target features from the feature storage module 32 through a scheduled task, and classify the target features, thereby distinguishing between offline full features and online incremental features in the target features.

[0076] Next, feature matching is performed on the acquired offline full features and online incremental features to obtain matching results.

[0077] For example, if the matching result shows that the enumeration values ​​of all offline full features and online incremental features are exactly the same, or the online incremental features are a subset of the offline full features, then it means that the two are completely matched and there are no different features (that is, there are no abnormal features).

[0078] If there are no abnormal features, it indicates that there is no risk and no related operations need to be performed through the feature verification system 31, and the entire process ends.

[0079] For example, if the matching result shows that the online incremental features are not a subset of the offline full features, it means that there are difference features (ie, abnormal features), and the abnormal features need to be determined from the online incremental features.

[0080] S540: Determine whether the abnormal features meet the preset rules through the feature verification system.

[0081] In an exemplary embodiment, after the abnormal features are determined, the abnormal features can be analyzed and judged in the feature verification system 31 to determine whether the abnormal features will cause serious risks and decide whether operations such as alarms or service disconnection are required.

[0082] For example, business experts can configure different preset rules to analyze and judge abnormal features. For example, the preset rules can be configured as filtering rules to filter abnormal features that are within the normal range. The filtering rules can be set to determine whether the fluctuation of the abnormal feature exceeds a preset threshold. If the fluctuation of the abnormal feature is less than the preset threshold, it indicates that the occurrence of the abnormal feature is normal and the impact can be ignored; if the fluctuation of the abnormal feature is greater than the preset threshold, it indicates that the abnormal feature cannot be ignored.

[0083] For example, different preset rules may be configured periodically so that the preset rules are changed according to changes in the business, thereby enabling the feature checking process to better serve the business.

[0084] It should be noted that the specific contents of the above preset rules are only examples and are not limited in this embodiment.

[0085] In an exemplary embodiment, when the abnormal features meet the above-mentioned preset rules, it indicates that there is no risk, and there is no need to perform relevant operations through the feature verification system 31, and the entire process ends.

[0086] S550: When the abnormal feature does not conform to the preset rule, the feature checking system determines whether the priority of the abnormal feature is lower than the preset priority.

[0087] In an exemplary embodiment, in addition to the aforementioned filtering rules, other preset rules can also be configured. For example, grading rules can be configured to classify abnormal characteristics, thereby further verifying the abnormal characteristics. The grading rules can be configured to assign different levels to different types of abnormal characteristics. For example, the aforementioned situation of an incorrect repayment currency type can be given a high priority due to the resulting financial losses.

[0088] For example, if the abnormal feature does not conform to the above preset rules, it is possible to continue to determine whether the abnormal rule conforms to the above classification rules, that is, to determine whether the priority of the abnormal feature is lower than the preset priority.

[0089] For example, suppose the priority is divided into six levels: Very High, High, High, Low, Low, and Very Low, and the preset priority is set to High. If the priority of an exception rule is High, the priority of the exception rule is the same as the preset priority, that is, it is not lower than the preset priority. If the priority of an exception rule is Very High, the priority of the exception rule is higher than the preset priority, that is, it is not lower than the preset priority. If the priority of an exception rule is Low, the priority of the exception rule is lower than the preset priority.

[0090] S560: When the priority of the abnormal feature is lower than the preset priority, the feature checking system outputs an alarm message about the abnormal feature.

[0091] In an exemplary embodiment, when the priority of the abnormal feature is lower than the preset priority, it means that the risk of the abnormal feature is not very high and will not cause direct asset loss or other impacts on the business. The feature verification system 31 can output an alarm information about the abnormal feature to notify the business party that an abnormal feature has occurred.

[0092] For example, after receiving the alarm information, the business expert may further process the abnormal features through the feature checking system 31 .

[0093] In addition, if the abnormal characteristics actually meet the rules in other business scenarios, the abnormal characteristics can be used as metadata to generate a use case corresponding to other business scenarios to supplement subsequent business verification work.

[0094] S570: When the priority of the abnormal feature is not lower than the preset priority, the service is disconnected through the feature verification system.

[0095] In an exemplary embodiment, when the priority of the abnormal feature is not lower than the preset priority, it means that the abnormal feature may have a more serious impact on the business. At this time, the feature verification system 31 can directly fuse the business through the interface provided by the business party, thereby timely avoiding serious losses to the business.

[0096] Among them, business circuit breaker means business interruption. When encountering financial losses within a foreseeable time due to data anomalies or code anomalies, the relevant business can be temporarily interrupted and resumed after the data anomaly or code anomaly problem is repaired.

[0097] By configuring the above-mentioned preset rules for abnormal features through the above-mentioned scheme, the noise generated in the feature verification process can be effectively reduced (i.e., abnormal features whose risk level does not reach the level of triggering alarms or service disconnection), thereby avoiding the problem of invalid alarms caused by massive noise and excessive manual processing costs.

[0098] FIG6 shows an architecture diagram of a data processing method provided in yet another embodiment of this specification.

[0099] In an exemplary embodiment, generally speaking, in the data processing method provided in this specification, after the offline feature extraction system 30 performs feature extraction on the offline full data, the offline full features are obtained, and the offline full features are stored in the feature storage module 32.

[0100] Afterwards, the feature verification system 31 generates a detection factor based on the offline full features and their table association relationships, so that the online feature extraction system 33 extracts features from the online incremental data based on the detection factor, obtains online incremental features, and stores the online incremental features in the feature storage module 32.

[0101] Next, the feature verification system 31 retrieves and classifies the offline full features and online incremental features from the feature storage module 32, performs feature matching on the two, and identifies abnormal features from the online incremental features based on the matching results. Finally, the feature verification system 31 determines the risk of the abnormal features based on pre-set rules, and then further outputs alarm information or performs service circuit breaking operations based on the abnormal features.

[0102] The data processing method provided in the embodiments of this specification can improve the accuracy of online incremental data and reduce the risk of capital loss in financial services.

[0103] The following are device embodiments of this specification, which can be used to implement the method embodiments of this specification. For details not disclosed in the device embodiments of this specification, please refer to the method embodiments of this specification.

[0104] 7 shows a structural diagram of a data processing device provided according to an exemplary embodiment of this specification.

[0105] The data processing device 700 in the embodiment of this specification includes: a generation module 701 , a feature extraction module 702 , and a matching module 703 .

[0106] The generation module 701 is used to generate a detection factor based on the offline full-quantity features and their corresponding table association relationships through a feature verification system.

[0107] The feature extraction module 702 is used to: perform feature extraction on the online incremental data according to the exploration factor through the online feature extraction system to obtain online incremental features.

[0108] The matching module 703 is used to: perform feature matching on the offline full features and the online incremental features through the feature verification system to obtain matching results; and determine abnormal features from the online incremental features based on the matching results, and output alarm information about the abnormal features.

[0109] 8 shows a structural diagram of a data processing device provided according to another exemplary embodiment of this specification.

[0110] Optionally, the above device further includes: a marking module 704 , a storage module 705 , and a judgment module 706 .

[0111] In one possible implementation, the feature extraction module 702 is further used to: perform feature extraction on offline full data through an offline feature extraction system to obtain a first full feature; the labeling module 704 is used to: perform labeling on the first full feature through a feature verification system to filter out a second full feature from the first full feature, wherein the second full feature meets the first preset standard; the judgment module 706 is used to: determine whether the second full feature meets the second preset standard through the feature verification system; the storage module 705 is used to: when the second full feature meets the second preset standard, determine the second full feature as an offline full feature through the feature verification system and store it in the feature storage module.

[0112] Optionally, the above device further includes: a configuration module 707.

[0113] In one possible implementation, the configuration module 707 is used to configure the basic information of the offline full data through the feature verification system, and input the basic information of the offline full data into the offline feature extraction system, wherein the basic information includes one or more of the table name, table fields, and table association relationships; the feature extraction module 702 is also used to obtain the offline full data and the basic information of the offline full data through the offline feature extraction system, and perform feature extraction on the offline full data based on the basic information of the offline full data to obtain the first full feature.

[0114] In a possible implementation, the storage module 705 is further configured to: store online incremental feature data into the feature storage module via an online feature extraction system.

[0115] Optionally, the apparatus further includes an acquisition module 708 and a classification module 709. The acquisition module 708 is configured to acquire target features from the feature storage module via a feature verification system; and the classification module 709 is configured to classify the target features via the feature verification system to determine offline full features and online incremental features from the target features.

[0116] In one possible implementation, the matching module 703 is specifically used to: perform feature matching on the offline full features and the online incremental features through a feature verification system to determine whether the online incremental features are a subset of the offline full features; the matching module 703 is specifically used to: determine abnormal features from the online incremental features when the matching result is that the online incremental features are not a subset of the offline full features.

[0117] In one possible implementation, the judgment module 706 is further used to determine whether the abnormal features comply with preset rules; the matching module 703 is specifically used to output alarm information about the abnormal features through the feature verification system when the abnormal features do not comply with preset rules.

[0118] Optionally, the above device further includes: a service fusing module 710 .

[0119] In one possible implementation, the judgment module 706 is also used to determine whether the priority of the abnormal feature is lower than the preset priority through the feature verification system; the matching module 703 is specifically used to output alarm information about the abnormal feature through the feature verification system when the priority of the abnormal feature is lower than the preset priority; the service fuse module 710 is used to perform service fuse through the feature verification system when the priority of the abnormal feature is not lower than the preset priority.

[0120] It should be noted that the data processing device provided in the above embodiment, when executing the data processing method, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the data processing device provided in the above embodiment and the data processing method embodiment are based on the same concept. Therefore, for details not disclosed in the device embodiment of this specification, please refer to the data processing method embodiment of this specification, and no further details will be given here.

[0121] The serial numbers of the embodiments in this specification are for description only and do not represent the advantages or disadvantages of the embodiments.

[0122] The embodiments of this specification also provide a terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the program.

[0123] FIG9 schematically shows a structure diagram of a terminal according to an exemplary embodiment of this specification. Referring to FIG9 , a terminal 900 includes a processor 901 and a memory 902 .

[0124] In the embodiments of this specification, the processor 901 is the control center of the computer system, which can be the processor of a physical machine or the processor of a virtual machine. The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state.

[0125] In an embodiment of the present specification, the processor 901 is specifically used to: generate a detection factor based on the offline full features and their corresponding table association relationships through a feature verification system; perform feature extraction on the online incremental data based on the detection factors through an online feature extraction system to obtain online incremental features; perform feature matching on the offline full features and the online incremental features through the feature verification system to obtain matching results; and determine abnormal features from the online incremental features based on the matching results, and output alarm information about the abnormal features.

[0126] Furthermore, in one embodiment of the present specification, the processor 901 is further configured to: perform feature matching on the offline full features and the online incremental features through the feature verification system to determine whether the online incremental features are a subset of the offline full features; and determine abnormal features from the online incremental features based on the matching results, including: determining the abnormal features from the online incremental features when the matching result is that the online incremental features are not a subset of the offline full features.

[0127] Optionally, the processor 901 is further used to: obtain target features from the feature storage module through the feature verification system; classify the target features through the feature verification system to determine the offline full features and the online incremental features from the target features.

[0128] Optionally, the processor 901 is also used to determine whether the abnormal features comply with preset rules. The processor 901 is specifically used to output warning information about the abnormal features through the feature verification system when the abnormal features do not comply with the preset rules.

[0129] Optionally, the processor 901 is also used to determine whether the priority of the abnormal feature is lower than the preset priority through the feature verification system; the processor 901 is specifically used to output alarm information about the abnormal feature through the feature verification system when the priority of the abnormal feature is lower than the preset priority; optionally, the processor 901 is also used to perform service fuse through the feature verification system when the priority of the abnormal feature is not lower than the preset priority.

[0130] Optionally, the processor 901 is also used to: perform feature extraction on offline full data through an offline feature extraction system to obtain a first full feature; mark the first full feature through the feature verification system to filter out a second full feature from the first full feature, wherein the second full feature meets the first preset standard; determine whether the second full feature meets the second preset standard through the feature verification system; if the second full feature meets the second preset standard, determine the second full feature as the offline full feature through the feature verification system and store it in the feature storage module.

[0131] Optionally, the processor 901 is also used to: configure the basic information of the offline full data through the feature verification system, and input the basic information of the offline full data into the offline feature extraction system, wherein the basic information includes one or more of the table name, table fields and the table association relationship; obtain the offline full data and the basic information of the offline full data through the offline feature extraction system, and perform feature extraction on the offline full data according to the basic information of the offline full data to obtain the first full feature.

[0132] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more magnetic disk storage terminals and flash memory storage terminals. In some embodiments of this specification, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one instruction, which is executed by the processor 901 to implement the method of the embodiments of this specification.

[0133] In some embodiments, the terminal 900 further includes a peripheral terminal interface 903 and at least one peripheral terminal. The processor 901, memory 902, and peripheral terminal interface 903 may be connected via a bus or signal lines. Each peripheral terminal may be connected to the peripheral terminal interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral terminal includes at least one of a display screen 904, a camera 905, and an audio circuit 906.

[0134] The peripheral terminal interface 903 can be used to connect at least one peripheral terminal related to input / output (I / O) to the processor 901 and the memory 902. In some embodiments of this specification, the processor 901, the memory 902, and the peripheral terminal interface 903 are integrated on the same chip or circuit board; in some other embodiments of this specification, any one or two of the processor 901, the memory 902, and the peripheral terminal interface 903 can be implemented on separate chips or circuit boards. This embodiment of this specification is not specifically limited to this.

[0135] The display screen 904 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 904 is a touch screen display, the display screen 904 is also capable of collecting touch signals on or above the surface of the display screen 904. The touch signals may be input as control signals to the processor 901 for processing. In this case, the display screen 904 may also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments of this specification, there may be one display screen 904, provided on the front panel of the terminal 900; in other embodiments of this specification, there may be at least two display screens 904, provided on different surfaces of the terminal 900 or in a foldable design; in still other embodiments of this specification, the display screen 904 may be a flexible display, provided on a curved or foldable surface of the terminal 900. Furthermore, the display screen 904 may be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 904 may be made of materials such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0136] The camera 905 is used to capture images or videos. Optionally, the camera 905 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and virtual reality (VR) shooting function or other fusion shooting functions. In some embodiments of the present specification, the camera 905 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0137] The audio circuit 906 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals for input to the processor 901 for processing. For the purpose of stereo sound collection or noise reduction, multiple microphones may be provided, respectively, at different locations on the terminal 900. The microphone may also be an array microphone or an omnidirectional microphone.

[0138] Power supply 907 is used to power various components in terminal 900. Power supply 907 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 907 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0139] The terminal structure block diagram shown in the embodiment of this specification does not constitute a limitation on the terminal 900. The terminal 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0140] In this specification, the terms "first," "second," etc. are used for descriptive purposes only and should not be construed as indicating or implying relative importance or order. The term "plurality" refers to two or more, unless expressly limited otherwise. Terms such as "installed," "connected," "connected," and "fixed" should be interpreted broadly. For example, "connected" can mean fixed, removable, or integrally connected; "connected" can mean directly or indirectly through an intermediary. Those skilled in the art will understand the specific meanings of these terms in this specification based on the specific circumstances.

[0141] In the description of this specification, it should be understood that the terms "upper" and "lower" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this specification and simplifying the description, rather than indicating or implying that the device or unit referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, they should not be understood as limitations on this specification.

[0142] The embodiments of this specification also provide a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps of the above embodiments. If the components of the above-described data processing device are implemented as software functional units and sold or used as independent products, they may be stored in the above-described computer-readable storage medium.

[0143] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The above-mentioned computer program product includes one or more computer instructions. When the above-mentioned computer program instructions are loaded and executed on a computer, the above-mentioned process or function according to the embodiment of this specification is generated in whole or in part. The above-mentioned computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The above-mentioned computer instructions can be stored in a computer-readable storage medium or transmitted by the above-mentioned computer-readable storage medium. The above-mentioned computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The above-mentioned computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available media may be magnetic media (eg, floppy disks, hard disks, tapes), optical media (eg, digital versatile discs (DVDs)), or semiconductor media (eg, solid state disks (SSDs)).

[0144] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0145] The above description is merely a specific embodiment of this specification, but the scope of protection of this specification is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this specification should be covered by the scope of protection of this specification. Therefore, equivalent variations made according to the claims of this specification are still within the scope of protection of this specification.

Claims

1. A data processing method, characterized in that: include: Generate exploration factors based on offline full-volume features and their corresponding table associations through the feature verification system; By using an online feature extraction system, feature extraction is performed on the online incremental data according to the detection factor to obtain online incremental features; The feature checking system performs feature matching on the offline full features and the online incremental features to obtain matching results; and, based on the matching results, abnormal features are determined from the online incremental features, and alarm information about the abnormal features is output.

2. The data processing method according to claim 1, characterized in that: Before generating the exploration factor according to the offline full-volume features and the corresponding table association relationship, the method further includes: Perform feature extraction on the offline full data through the offline feature extraction system to obtain the first full feature; The first full quantity feature is annotated by the feature checking system to filter out a second full quantity feature from the first full quantity feature, wherein the second full quantity feature meets a first preset standard; Determining, by the feature checking system, whether the second full quantity feature meets a second preset standard; When the second full feature meets the second preset standard, the second full feature is determined as the offline full feature by the feature verification system and stored in the feature storage module.

3. The data processing method according to claim 2, characterized in that: Before extracting features from the offline full data by the offline feature extraction system to obtain the first full feature, the method further includes: configuring basic information of the offline full data through the feature checking system, and inputting the basic information of the offline full data into the offline feature extraction system, wherein the basic information includes one or more of a table name, a table field, and a table association relationship; The offline full data and basic information of the offline full data are obtained through the offline feature extraction system, and feature extraction is performed on the offline full data according to the basic information of the offline full data to obtain the first full feature.

4. The data processing method according to claim 2, characterized in that: After extracting features from the online incremental data according to the detection factor to obtain online incremental features, the method further includes: The online incremental feature data is stored in the feature storage module through an online feature extraction system.

5. The data processing method according to claim 4, characterized in that: Before generating the exploration factor according to the offline full-volume features and the corresponding table association relationship, the method further includes: Acquiring target features from the feature storage module through the feature checking system; The target features are classified by the feature checking system to determine the target features. The offline full features and the online incremental features.

6. The data processing method according to any one of claims 1 to 5, wherein the feature checking system is used to perform feature matching on the offline full feature and the online incremental feature, comprising: By means of the feature checking system, feature matching is performed on the offline full feature and the online incremental feature to determine whether the online incremental feature is a subset of the offline full feature; Determining the abnormal feature from the online incremental feature according to the matching result includes: When the matching result is that the online incremental feature is not a subset of the offline full feature, the abnormal feature is determined from the online incremental feature.

7. The data processing method according to any one of claims 1 to 5, characterized in that: After determining the abnormal feature from the online incremental feature according to the matching result, the method further includes: Determine whether the abnormal feature meets the preset rules; The outputting of warning information about the abnormal feature includes: When the abnormal feature does not conform to the preset rule, the feature checking system outputs warning information about the abnormal feature.

8. The data processing method according to claim 7, characterized in that: In the case where the abnormal feature does not conform to the preset rule, before outputting the warning information about the abnormal feature through the feature checking system, the method further includes: Determining, by the feature checking system, whether the priority of the abnormal feature is lower than a preset priority; The outputting of warning information about the abnormal feature by the feature checking system includes: When the priority of the abnormal feature is lower than the preset priority, outputting warning information about the abnormal feature through the feature checking system; After determining whether the abnormal feature meets the preset rule, the method further includes: When the priority of the abnormal feature is not lower than the preset priority, the service is disconnected through the feature checking system.

9. A data processing device, characterized in that: include: A generation module is used to generate exploration factors based on the offline full-volume features and their corresponding table association relationships through a feature verification system; A feature extraction module is used to: extract features from online incremental data according to the detection factor through an online feature extraction system to obtain online incremental features; The matching module is used to: perform feature matching on the offline full feature and the online incremental feature through the feature checking system to obtain a matching result; and determine from the online incremental feature according to the matching result The abnormal feature is detected and warning information about the abnormal feature is output.

10. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the data processing method according to any one of claims 1 to 8 is implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising instructions, which, when executed on a computer or a processor, causes the computer or the processor to execute the data processing method according to any one of claims 1 to 8.

Images