Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of big data. A specific embodiment of the method comprises the steps of generating alarm information corresponding to a field in response to monitoring that the field of a basic data layer is subjected to information change; wherein the information change comprises logic adjustment and data exception; and sending the alarm information to a downstream data layer based on a pre-configured upstream and downstream dependency relationship. According to the embodiment, the upstream field information change can be automatically obtained, the alarm notification is sent to the downstream data layer according to the upstream and downstream dependency relationship, the data processing efficiency is improved, and the data processing cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method and device. Background Art

[0002] Currently, in offline data scenarios, whether upstream data is abnormal is usually determined by manual judgment and identification, and then the scope and degree of impact of downstream data are manually determined.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the related art:

[0004] Manual judgment of abnormalities in upstream data leads to inefficient data processing and requires a lot of labor costs. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data processing method and device, which can automatically obtain upstream field information changes and send alarm notifications to the downstream data layer according to the upstream and downstream dependencies, thereby improving data processing efficiency and reducing data processing costs.

[0006] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0007] In response to monitoring that information changes occur in fields of the basic data layer, alarm information corresponding to the fields is generated; wherein the information changes include logic adjustments and data anomalies;

[0008] Send alert information to the downstream data layer based on pre-configured upstream and downstream dependencies.

[0009] Optionally, after sending the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependency relationship, the method further includes:

[0010] In response to the downstream data layer receiving the alarm information, a target abnormal field corresponding to the downstream data layer is determined based on the alarm information.

[0011] Optionally, the downstream data layer includes an intermediate layer and an application layer;

[0012] In response to the downstream data layer receiving the alarm information, determining a target abnormal field corresponding to the downstream data layer based on the alarm information includes:

[0013] In response to the middle layer receiving the alarm information, the alarm information is compared with the acquired historical data snapshot, and the impact scope of the abnormality is determined based on the comparison result;

[0014] The exception impact range is sent to the application layer, and in response to the application layer receiving the exception impact range, a target exception field is determined according to the exception impact range.

[0015] Optionally, after determining the target abnormal field corresponding to the downstream data layer based on the alarm information, the method further includes:

[0016] The target abnormal field corresponding to the downstream data layer is repaired according to the preset abnormal repair mechanism, and the repair result is fed back to the upstream basic data layer.

[0017] Optionally, in response to monitoring that a field of the basic data layer has an information change, generating alarm information corresponding to the field includes:

[0018] In response to monitoring that a numeric field exceeds a preset change threshold or monitoring that a character field exceeds a preset proportion, alarm information corresponding to the field is generated.

[0019] Optionally, after the repair result is fed back to the upstream basic data layer, it includes:

[0020] In response to the base data layer receiving the repair result, a synchronous repair process is performed on the fields of the base data layer based on the repair result.

[0021] Optionally, the method further comprises:

[0022] The information changes and repair results are recorded in the dispatching platform so that the dispatching platform can process the information changes according to the repair results.

[0023] According to a second aspect of an embodiment of the present invention, there is provided a data processing device, including:

[0024] A monitoring module, configured to generate alarm information corresponding to a field in response to monitoring information changes in a field of the basic data layer; wherein the information changes include logic adjustments and data anomalies;

[0025] The alarm module is used to send alarm information to the downstream data layer based on the pre-configured upstream and downstream dependencies.

[0026] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0027] one or more processors;

[0028] a memory for storing one or more programs,

[0029] When one or more programs are executed by one or more processors, the one or more processors are enabled to implement the method of any of the above embodiments.

[0030] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method of any of the above embodiments is implemented.

[0031] According to a fifth aspect of an embodiment of the present invention, there is provided a computer program product, including a computer program, which implements the method of any of the above embodiments when the computer program is executed by a processor.

[0032] An embodiment of the above invention has the following advantages or beneficial effects: by responding to information changes in the fields of the monitored basic data layer, generating alarm information corresponding to the fields; wherein the information changes include logical adjustments and data anomalies; sending the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependencies; thereby being able to automatically obtain upstream field information changes, and issuing alarm notifications to the downstream data layer based on the upstream and downstream dependencies, thereby improving data processing efficiency and reducing data processing costs.

[0033] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0035] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present invention;

[0036] Figure 2 is a schematic diagram of the main flow of offline data processing according to a preferred embodiment of the present invention;

[0037] Figure 3 is a schematic diagram of the main process of automatic early warning configuration according to a preferred embodiment of the present invention;

[0038] Figure 4 is a schematic diagram of the main process of offline field data abnormality monitoring according to a preferred embodiment of the present invention;

[0039] Figure 5 is a schematic diagram of the main process of a data anomaly feedback repair mechanism according to a preferred embodiment of the present invention;

[0040] Figure 6 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0041] Figure 7 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0042] Figure 8 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0044] It should be noted that the acquisition, storage and application of personal information etc. involved in the embodiments of the present invention are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0045] Currently, in offline data scenarios, whether upstream data is abnormal is usually determined by manual judgment and identification, and then the scope and degree of impact of downstream data are manually judged, resulting in inefficient data processing and requiring a lot of labor costs.

[0046] In view of this, according to one aspect of an embodiment of the present invention, a data processing method is provided.

[0047] Figure 1 FIG. 1 is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention. Figure 1 As shown, the data processing method according to the embodiment of the present invention includes the following steps S101 to S102.

[0048] Step S101, in response to monitoring that information changes occur in fields of the basic data layer, generating alarm information corresponding to the fields; wherein the information changes include logic adjustments and data anomalies.

[0049] The basic data layer is a layer used to store and manage core data in the system. The fields in the basic data layer include the core elements of the business logic, such as user information, order status, financial records or configuration parameters. Information changes in fields refer to changes in the status or value of the field. Information changes mainly include logical adjustments and data anomalies. Logical adjustments refer to structural changes in field definitions or attributes in the basic data layer, such as modifying the data type of a field (such as changing from integer to string), adjusting the constraints of the field (such as changing the primary key, foreign key relationship, etc.), or changing the business logic meaning of the field (such as changing from "user age" to "year of birth", etc.). Data anomalies refer to data values ​​in the field that fail to meet the expected status or rules, such as null values ​​in the field, values ​​outside the reasonable range, or format errors.

[0050] To monitor the information changes of fields in the basic data layer, event-driven log analysis technology can be used to capture field change information through real-time analysis of data operation logs. For example, the database operation log can record the time, field name, and change content of the add, delete, and modify operations. These log data can be parsed and generated by automated tools to generate change events. In addition, a rule-based real-time detection mechanism can be used to monitor changes in field values ​​in real time by configuring predefined rules and triggers. For example, set the legal range or data format of the field, and generate an alarm message immediately when the value exceeds the range or does not match the format. In addition, it is also possible to combine scheduled scheduling tasks to regularly perform full checks on field data to identify potential problems.

[0051] The generated alarm information is used to notify the operation and maintenance personnel, system administrators or scheduling platform. The alarm information can include the field name, change type, change time, impact scope and detailed description, etc. For example, when a logical adjustment of a field is detected, the alarm information can include the adjusted logical rules, the potential risks caused by the change and the impact assessment on the downstream data layer, etc.; when a data anomaly is detected, the alarm information can describe the type of anomaly, the specific abnormal value and the possible cause, etc. The way to generate alarm information can be through real-time triggers, combined with field change rules, directly calling the alarm service when anomalies are monitored, and pushing detailed alarm information to predefined receiving channels, such as emails, SMS or instant messaging tools, etc. In addition, the change events in the log can also be matched with the rules to generate a structured alarm report, which can be displayed in the monitoring platform through visualization tools. For example, clear alarm information can be generated, such as "Field name: inventory; data anomaly: value is -10, which does not meet the non-negative number constraint" or "Field name: user age; logical adjustment: data type is changed from integer to string", etc. In the above way, comprehensive monitoring and timely response to changes in field information in the basic data layer can be achieved and alarm information can be generated.

[0052] Step S102: sending the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependency relationship.

[0053] Upstream and downstream dependencies refer to the logical connection and data dependency between the upstream data layer and the downstream data layer, which defines the direction of data flow. For example, information changes in the fields of the upstream data layer will directly or indirectly affect the downstream tables or applications that depend on the fields. Upstream and downstream dependencies can be reflected through field associations, calculation formulas, task links, etc. For example, the fields of the upstream order data table may be the basis for the calculation of downstream financial statements, and an intermediate processing table may rely on data from multiple upstream sources for aggregation and cleaning. To configure upstream and downstream dependencies, you can define input and output fields for each data processing task based on the task configuration function of the scheduling platform, and clarify the flow direction and dependency of data; for example, define the mapping relationship and transmission rules at the field level through the task configuration list. In addition, to configure upstream and downstream dependencies, you can also use the data lineage analysis tool to automatically generate field-level upstream and downstream dependencies by analyzing the logs and configuration files of the data processing process, and support users to manually adjust and improve them.

[0054] The downstream data layer is a system or surface structure that depends on the upstream data processing results, such as the middle layer, the business layer directly facing the user, and the analysis report, etc. The upstream data layer can be a basic data layer or other data layers located upstream. Specifically, when the alarm information is sent to the downstream data layer based on the upstream and downstream dependency relationship, the alarm information can be pushed directly to the relevant person in charge downstream through the event-driven mechanism. When the field is monitored to have information changes, the alarm information can also be generated based on the scheduling platform according to the pre-defined dependency rules, and the downstream data processing tasks or applications can be notified in real time through the message queue, HTTP interface (application programming interface) or instant messaging tools. This method can quickly spread information and is suitable for scenarios with high real-time requirements. In addition, the alarm information can also be summarized and recorded in a centralized monitoring platform through a regular synchronization mode, and then the alarm information is sent to the downstream data layer by the scheduling task at a fixed time. This method is suitable for scenarios that require batch processing and analysis, such as merging multiple changes into one update notification to reduce the frequency of operations. Through the above method, the alarm information can be accurately and efficiently transmitted to the downstream data layer, so that the downstream data layer can adjust the processing logic or strategy in time when the data changes, thereby reducing the chain effects caused by data anomalies or logic changes, and improving the overall reliability and stability of data processing.

[0055] The embodiment of the present invention generates alarm information corresponding to the field in response to information changes in the fields of the basic data layer being monitored; wherein the information changes include logic adjustments and data anomalies; and sends the alarm information to the downstream data layer based on pre-configured upstream and downstream dependencies; thereby, it is possible to automatically obtain upstream field information changes and send alarm notifications to the downstream data layer according to the upstream and downstream dependencies, thereby improving data processing efficiency and reducing data processing costs.

[0056] Optionally, after sending the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependency relationship, the method further includes: in response to the downstream data layer receiving the alarm information, determining a target abnormal field corresponding to the downstream data layer based on the alarm information.

[0057] When the alarm information is sent to the downstream data layer, the relevant person in charge of the downstream data layer or the scheduling platform parses and processes the received alarm information. The alarm information contains the name of the changed field, the change type, the change details, and the potential impact range, etc., and the target abnormal field related to the alarm can be determined based on this information. Specifically, the relevant person in charge of the downstream data layer or the scheduling platform can locate the fields that may be affected in the downstream data layer by comparing the names or identifiers of the fields in the alarm information; they can also deduce the target fields or data modules that need special attention based on the change type and field dependencies.

[0058] The execution of alarm processing can be carried out in many ways. One way is for the relevant person in charge of the downstream data layer to perform the processing, that is, to send the alarm information to the person in charge of the current task or table. After receiving the alarm information, the person in charge of the table makes a judgment, performs relevant operations to determine the target abnormal field, and synchronizes the alarm information and the person in charge's operation information to the scheduling platform for record. Another way is to execute the processing task through the scheduling platform, upload the alarm information to the centralized scheduling platform, and the scheduling platform automatically distributes the task to the relevant downstream data layer to determine the target abnormal field according to the pre-configured exception handling strategy. For example, in a distributed system, the scheduling platform can trigger coordinated processing between multiple data nodes to ensure that the impact range of the alarm information is thoroughly covered. It is suitable for complex multi-layer data systems and can effectively improve the efficiency and accuracy of alarm processing.

[0059] Optionally, the downstream data layer includes an intermediate layer and an application layer; in response to the downstream data layer receiving the alarm information, determining a target abnormal field corresponding to the downstream data layer based on the alarm information, including: in response to the intermediate layer receiving the alarm information, comparing the alarm information with the acquired historical data snapshot, and determining the abnormal impact scope based on the comparison result; sending the abnormal impact scope to the application layer, and in response to the application layer receiving the abnormal impact scope, determining the target abnormal field according to the abnormal impact scope.

[0060] In a layered data system, the downstream data layer can include the middle layer and the application layer, which handle the business logic and user interaction of the data respectively. When the downstream data layer receives the alarm information, the system processes the alarm in a hierarchical manner so that the impact of the information change can be quickly located and resolved. First, the middle layer immediately starts the comparison process after receiving the alarm information. The middle layer generally stores historical data snapshots of key fields. By comparing the field change content in the alarm information with the historical data snapshots, the type and scope of the change can be accurately identified. For example, if the alarm information shows that the value of a certain field is abnormal, and the historical snapshot shows that the field has always complied with the agreed rules, it may point to problems in the collection or processing process of new data. Based on the comparison results, the middle layer can clearly define the scope of abnormal impact, including the affected tables, fields, and upstream and downstream modules that may be affected. The abnormal impact scope can be further encapsulated as structured data and passed to the application layer. Then, after receiving the abnormal impact scope, the application layer further analyzes and locates the target abnormal field. The application layer has more comprehensive business rules and scenario knowledge, so it can interpret the abnormal impact at the business level in combination with the actual usage scenario. For example, the application layer can identify the specific functional modules corresponding to the abnormal fields (such as the inventory display fields in the order confirmation page) based on the mapping relationship between the abnormal fields and the user interface, and identify the target abnormal fields for priority processing based on their criticality in the business process. In addition, the application layer can also use user behavior data and real-time analysis tools to prioritize the scope of abnormal impact, so that the most seriously affected fields can be processed quickly.

[0061] As another embodiment, a multi-layer response mode coordinated by the scheduling platform can be adopted. The middle layer is still used to receive alarm information and perform historical data comparison, but the abnormal impact range will not be directly passed to the application layer, but submitted to the scheduling platform. Based on the global perspective and predefined rules, the scheduling platform comprehensively considers the business priority, dependency and risk assessment results of the abnormal field, and generates an optimized abnormal impact processing plan, which may include recommended repair measures, warning level adjustment suggestions and correction lists of associated fields. The scheduling platform distributes the optimization plan to the application layer. The application layer locates the target abnormal field according to the plan and performs specific repair or adjustment tasks. For example, for key functional modules with frequent user operations, the scheduling platform may give priority to repair tasks, while delaying the processing of secondary functional modules. This can quickly respond to and handle data anomalies, improving system stability and user experience.

[0062] Optionally, after determining the target abnormal field corresponding to the downstream data layer based on the alarm information, it also includes: repairing the target abnormal field corresponding to the downstream data layer according to a preset abnormality repair mechanism, and feeding back the repair result to the upstream basic data layer.

[0063] After the target abnormal field is determined, the downstream data layer first starts the preset abnormal repair mechanism, which performs automated or manually assisted repair operations based on the abnormal type, the business attributes of the field, and the upstream and downstream dependencies. For example, for anomalies caused by logic adjustments, the repair mechanism can readjust the mapping relationship of the field or rebuild the dependent index; for problems caused by data anomalies, the abnormal field can be repaired by re-pulling valid data from the data source or calling the data cleaning process. In some cases, fill values ​​can also be generated according to predefined validation rules to supplement missing or invalid data. For example, for missing timestamp fields, reasonable alternative values ​​can be generated according to time series interpolation. After the repair is completed, the downstream data layer will generate detailed feedback information for the repair results, including the name of the repaired field, the original abnormal state, the description of the repair operation, and the repaired data verification results. The feedback information is passed to the upstream basic data layer through a standardized interface or message queue. The feedback results received are reviewed and integrated at the basic data layer, and the repaired field data is synchronized to its original storage. If the repair operation involves the adjustment of upstream and downstream dependencies, the basic data layer can also re-evaluate its own data model or update the configuration rules based on the feedback results to ensure the consistency of the overall data structure.

[0064] As another embodiment, the repair mechanism can be coordinated through a scheduling platform. When the target abnormal field involves multiple subsystems or distributed nodes, the scheduling platform can receive the target field information and select an appropriate repair strategy based on the abnormality type. For example, if the abnormal field comes from the inconsistency of multiple data copies, the scheduling platform can schedule each node to perform a consistency verification process, and select a trusted copy for synchronous repair through a majority rule or a weight priority mechanism. After the repair is completed, the scheduling platform generates a unified feedback report and submits the repair results and process logs to the basic data layer. The basic data layer updates its data snapshot based on this data and marks the status of the repaired field. The participation of the scheduling platform can not only handle cross-node repairs in complex scenarios, but also ensure repair efficiency through global optimization.

[0065] The embodiment of the present invention processes data anomalies in a closed-loop manner through an anomaly repair mechanism, which can not only quickly solve the problem of the target anomaly field, but also feed back the repair results to the upstream, forming a reliable self-repair system and improving the robustness and stability of the data system.

[0066] Optionally, in response to monitoring information changes in fields of the basic data layer, alarm information corresponding to the field is generated, including: in response to monitoring that a numeric field exceeds a preset change threshold or monitoring that a character field exceeds a preset proportion, alarm information corresponding to the field is generated.

[0067] When the information of the fields in the basic data layer changes, for numeric fields, set the change threshold, such as setting the allowed range or upper and lower limits. When the value of the monitored numeric field exceeds the threshold, the alarm logic is immediately triggered to record the changed field name, abnormal type and current value. For character fields, by counting the content distribution and calculating the proportion of characters, when the proportion of a certain character exceeds the preset proportion, an alarm message is generated. The alarm message can include the field type, abnormal description and related context information.

[0068] In addition, it can also be used to monitor time series data. When the field is time series data, the trend change of the data can be monitored through the sliding window technology. For example, if the change rate of a field at multiple consecutive time points in the past is significantly deviated from the historical average, an alarm message will be generated. This method can not only capture single anomalies, but also discover potential continuous problems.

[0069] Optionally, after feeding back the repair result to the upstream basic data layer, the method includes: in response to the basic data layer receiving the repair result, performing synchronous repair processing on the fields of the basic data layer based on the repair result.

[0070] After receiving the repair results, the basic data layer locates the corresponding target field by parsing the field name, repair value, and repair strategy in the repair results. The target field is updated according to the repair value, and the field is checked for consistency according to the repair strategy, so that the repaired data meets the integrity and constraints of the basic data layer. For example, for numeric fields, the system recalculates the field's statistics, such as the average and distribution; for character fields, it verifies whether the data format and encoding rules are correct. After the synchronous repair process is completed, a processing log is generated to record the repair details, and the snapshot of the basic data layer is updated to reflect the latest status.

[0071] As another embodiment, the repair operation of the basic data layer is coordinated through the scheduling platform. After receiving the repair results, the scheduling platform distributes them to multiple basic data modules involved. Each module performs field updates according to its own specific rules and synchronizes the dependencies between each other through a consistency verification mechanism. For example, when the repair result of a field affects other derived fields, the scheduling platform will automatically trigger the recalculation logic of these fields. After the repair is completed, all modules will summarize the update results to the scheduling platform, and the platform will generate a unified repair report, so that the synchronous repair processing of the entire basic data layer is efficient and without omissions.

[0072] Optionally, the method further includes: recording the information change and the repair result to the scheduling platform, so as to utilize the scheduling platform to process the information change according to the repair result.

[0073] After the information is changed and the repair results are generated, the relevant data can be synchronously recorded to the scheduling platform. Specifically, the type of information change (such as logic adjustment or data anomaly), the scope of impact, and the field name, repair strategy, and repair status of the repair result can all be stored in a structured manner on the scheduling platform. After receiving this data, the scheduling platform starts further processing logic for the information change. For example, the scheduling platform can trigger the retry process of other downstream tasks that depend on the target field based on the repair results, or redistribute tasks that have been shelved due to field anomalies. This process enables the repair operation to be updated to the system globally in real time, and achieves seamless connection of data processes through the scheduling platform.

[0074] In addition to recording information changes and repair results, the dispatch platform can also automatically analyze the potential impact of information changes through preset rules. For example, the dispatch platform can predict the possibility of similar anomalies recurring based on historical data of information changes and issue early warnings, further improving the comprehensiveness and efficiency of information change processing.

[0075] Figure 2 FIG. 1 is a schematic diagram of the main process of offline data processing according to a preferred embodiment of the present invention. Figure 2 As shown in the figure, in the offline data lineage optimization relationship, the data dependency starts from the basic data layer and is transmitted to the application layer through the middle layer, that is, there is an upstream and downstream dependency relationship between the basic offline table A, the middle layer offline table B, and the application layer offline table C. When the basic data layer detects a field anomaly, that is, field A of the basic offline table A is abnormal, the abnormal field A is logically adjusted or data is flushed back to repair the data source problem. After the repair is completed, the basic data layer will notify the relevant person in charge of the middle layer of the abnormal field information and the repaired results. After receiving the information, the relevant person in charge of the middle layer compares and analyzes the historical snapshot data corresponding to field A of the middle layer offline table B to locate the scope of the abnormal impact and further evaluate the downstream data that may be affected. After the impact scope is determined, the middle layer will notify the relevant person in charge of the application layer. The application layer evaluates the impact of the abnormality based on the transmitted information and repairs or adjusts field A of the application layer offline table C when necessary. Finally, the application layer feeds back the repair plan to the basic layer to form a closed loop to ensure the stability and consistency of the entire data chain.

[0076] The embodiment of the present invention uses a fully online process to determine whether the fields of the basic data layer have undergone logical changes or data anomalies, track and maintain the dependencies between data, and send early warning notifications to the table owners of the downstream middle layer and application layer in the form of alarms. The middle layer and application layer will compare and check based on field changes and historical data snapshots, locate specific field data anomalies and feedback to the table owner, thereby achieving accurate tracking and dynamic maintenance alarms for the dependencies between upstream and downstream data during offline data field processing, improving data anomaly processing efficiency, reducing the risk of application layer data errors, and the downstream can be notified and repaired in a timely manner as soon as upstream data problems occur. The embodiment of the present invention mainly includes aspects such as automated early warning configuration, offline field data anomaly monitoring, and data anomaly feedback repair mechanism.

[0077] Figure 3 This is a schematic diagram of the main process of automatic early warning configuration according to a preferred embodiment of the present invention. Using the upstream and downstream dependencies of offline data, an alarm message is automatically sent to the downstream dependency configuration table when the upstream field data changes. Specifically, after the logical iteration of the base layer data is completed, the downstream dependency interface is configured on the scheduling platform; then, through the upper and lower dependency interfaces configured on the scheduling platform, the persons in charge of the downstream related intermediate layer tables and application layer tables are automatically notified, and the alarm message of the upstream field change is sent, and the specific logic of the upstream field change, the scope of influence, data fluctuations and other information are synchronized to the person in charge of the downstream data layer. Figure 3 As shown, the logic of field A in the basic offline table A is adjusted. Based on the lineage relationship (upstream and downstream dependencies) of field A, it is traced back to all downstream intermediate layer B tables and the task leaders are notified. The intermediate layers then trace back to all application layer C tables through the lineage relationship of their respective fields A and notify the task leaders.

[0078] Figure 4 FIG. 1 is a schematic diagram of the main process of offline field data abnormality monitoring according to a preferred embodiment of the present invention. Figure 4As shown in the figure, starting from the detection of anomaly in field A of table A, the logical reverse adjustment of field A is performed to restore the normal value of field A and ensure the consistency of data. After the adjustment is completed, the impact range of field A after adjustment on the data is analyzed, and the changes in the data are recorded. Next, the threshold information of field A is set in table B to monitor field changes. When the field value is detected to exceed the preset threshold range, the alarm mechanism is triggered to promptly notify the person in charge of table B so that the abnormal situation can be handled as soon as possible. At the same time, the system will also synchronize the abnormal situation to the person in charge of downstream table C and other related tables to ensure the integrity and accuracy of downstream dependent data. Through this monitoring and feedback mechanism, the system can achieve rapid response and accurate processing of field anomalies, thereby ensuring the overall data quality. In addition, the threshold range and calculation logic related to each field information can also be configured through the scheduling platform. First, the scheduling platform is used to configure the maximum and minimum threshold monitoring for the numeric fields of the offline table; the character fields are configured with the proportion. When the field after the monitoring configuration exceeds the threshold or the proportion exceeds the configuration, an alarm message will be sent to the person in charge of the current task or table. After receiving the information, the person in charge of the table will judge the current information. The specific upstream table is preliminarily located by judging the abnormal alarm information. At the same time, when downstream tasks configure field alarms, if an alarm exceeds the threshold range, the upstream table will be notified for investigation. At the same time, the alarm information and the action information of the person in charge will be synchronized to the scheduling platform for record.

[0079] Figure 5 FIG. 1 is a schematic diagram of the main process of a data anomaly feedback repair mechanism according to a preferred embodiment of the present invention. Figure 5 As shown in the figure, when a field A of table A is detected to be abnormal in the basic layer, the logical reverse adjustment of field A is first performed to restore the correctness of the field. After the adjustment of field A is completed, the adjusted data change range is analyzed, and the repair result is synchronized to the middle layer. The middle layer will set a threshold alarm for field A, and further analyze whether the abnormal field A exceeds the threshold range. If it exceeds the threshold range, the system will notify the middle layer and the downstream B table, C table and other responsible persons, and the relevant responsible persons will adjust the abnormal field A according to the feedback information. The application layer will feedback the repair suggestions for the abnormal field A to the basic layer, thus forming a complete feedback and repair closed loop. In the whole mechanism, each level can quickly locate, process and repair abnormal data through division of responsibilities and real-time information feedback. In addition, after receiving the alarm, the scheduling platform will first call the historical information for operation, and send the alarm information to synchronize the upstream and downstream; after receiving the current repair information, the responsible person can choose to ignore it if there is no objection; if there is an objection, the specific field can be repaired; the repaired logic and operation will be synchronized with the scheduling platform, and the logical repair method will also be sent to the upstream responsible person; the upstream responsible person can adjust and handle it according to the information.

[0080] The embodiment of the present invention can automatically obtain upstream field information changes and notify downstream alarms; by recording manual operations to the scheduling platform, the same problems and exceptions can be quickly handled, realizing a fully automated process, improving the timeliness and accuracy of judgments, and improving the closeness of upstream and downstream linkages of offline data, thereby achieving rapid repair of data anomalies.

[0081] According to a second aspect of an embodiment of the present invention, a data processing device is provided.

[0082] Figure 6 Schematic diagram of main modules of a data processing device according to an embodiment of the present invention. Figure 6 As shown, a data processing device 600 is provided, including:

[0083] The monitoring module 601 is used to generate alarm information corresponding to a field in response to monitoring information changes of a field in the basic data layer; wherein the information changes include logic adjustments and data anomalies;

[0084] The alarm module 602 is used to send alarm information to the downstream data layer based on the pre-configured upstream and downstream dependencies.

[0085] Optionally, the data processing device 600 further includes a determination module, and the abnormality module is used to:

[0086] In response to the downstream data layer receiving the alarm information, a target abnormal field corresponding to the downstream data layer is determined based on the alarm information.

[0087] Optionally, the downstream data layer includes an intermediate layer and an application layer; the exception module is further used for:

[0088] In response to the middle layer receiving the alarm information, the alarm information is compared with the acquired historical data snapshot, and the impact scope of the abnormality is determined based on the comparison result;

[0089] The exception impact range is sent to the application layer, and in response to the application layer receiving the exception impact range, a target exception field is determined according to the exception impact range.

[0090] Optionally, the data processing device 600 further includes a repair module, and the repair module is used to:

[0091] The target abnormal field corresponding to the downstream data layer is repaired according to the preset abnormal repair mechanism, and the repair result is fed back to the upstream basic data layer.

[0092] Optionally, the monitoring module 601 is further used for:

[0093] In response to monitoring that a numeric field exceeds a preset change threshold or monitoring that a character field exceeds a preset proportion, alarm information corresponding to the field is generated.

[0094] Optionally, the data processing device 600 further includes a synchronization module, and the synchronization module is used to:

[0095] In response to the base data layer receiving the repair result, a synchronous repair process is performed on the fields of the base data layer based on the repair result.

[0096] Optionally, the data processing device 600 further includes a recording module, and the recording module is used to:

[0097] The information changes and repair results are recorded in the dispatching platform so that the dispatching platform can process the information changes according to the repair results.

[0098] It should be noted that the specific implementation content of the data processing device of the present invention has been described in detail in the above data processing method, so the repeated content will not be described again here.

[0099] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0100] one or more processors;

[0101] a memory for storing one or more programs,

[0102] When one or more programs are executed by one or more processors, the one or more processors are enabled to implement the method of any of the above embodiments.

[0103] According to a fourth aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method of any of the above embodiments is implemented.

[0104] According to a fifth aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method provided by the first aspect of the embodiment of the present invention is implemented.

[0105] Figure 7 An exemplary system architecture 700 is shown to which the data processing method or data processing device according to the embodiment of the present invention can be applied.

[0106] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, 703, network 704 and server 705. Network 704 is used to provide a medium for communication links between terminal devices 701, 702, 703 and server 705. Network 704 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0107] The user can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 701, 702, 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0108] The terminal devices 701 , 702 , and 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0109] The server 705 may be a server that provides various services, such as a background management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 701, 702, and 703. The background management server may analyze and process the received data such as the data processing request, and feed back the processing result (such as the data processing result - only an example) to the terminal device.

[0110] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 705 , and accordingly, the data processing device is generally disposed in the server 705 .

[0111] It should be understood that Figure 7 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0112] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 of a terminal device suitable for implementing an embodiment of the present invention. Figure 8 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0113] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can execute various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage part 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0114] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 runs a communication process via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that a computer program read therefrom is installed into the storage section 808 as needed.

[0115] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for running the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-mentioned functions defined in the system of the present invention are executed.

[0116] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0117] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually run basically in parallel, and they can sometimes run in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that runs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0118] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor, for example, may be described as: a processor including a monitoring module and an alarm module. The names of these modules do not, in some cases, constitute limitations on the modules themselves, for example, the alarm module may also be described as "a module for sending alarm information to a downstream data layer based on pre-configured upstream and downstream dependencies".

[0119] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: in response to monitoring the information change of the field of the basic data layer, generating alarm information corresponding to the field; wherein the information change includes logic adjustment and data anomaly; sending the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependencies.

[0120] The computer program product provided by the embodiment of the present invention includes a computer program. When the computer program is executed by a processor, the data processing method in the embodiment of the present invention is implemented.

[0121] According to the technical solution of the embodiment of the present invention, it has the following advantages or beneficial effects: by responding to the monitored information changes of the fields of the basic data layer, generating alarm information corresponding to the fields; wherein the information changes include logical adjustments and data anomalies; sending the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependencies; thereby being able to automatically obtain the upstream field information changes, and issuing alarm notifications to the downstream data layer according to the upstream and downstream dependencies, thereby improving data processing efficiency and reducing data processing costs.

[0122] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0123] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

Claims

1. A data processing method, characterized in that: include: In response to monitoring that a field of the basic data layer has an information change, generating alarm information corresponding to the field; wherein the information change includes logic adjustment and data anomaly; The alarm information is sent to the downstream data layer based on the pre-configured upstream and downstream dependencies.

2. The method according to claim 1, characterized in that After sending the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependency relationship, the method further includes: In response to the downstream data layer receiving the alarm information, a target abnormal field corresponding to the downstream data layer is determined based on the alarm information.

3. The method according to claim 2, characterized in that The downstream data layer includes an intermediate layer and an application layer; In response to the downstream data layer receiving the alarm information, determining a target abnormal field corresponding to the downstream data layer based on the alarm information includes: In response to the middle layer receiving the alarm information, the alarm information is compared with the acquired historical data snapshot, and the impact scope of the abnormality is determined based on the comparison result; The abnormal impact range is sent to the application layer, and in response to the application layer receiving the abnormal impact range, a target abnormal field is determined according to the abnormal impact range.

4. The method according to claim 2, characterized in that: After determining the target abnormal field corresponding to the downstream data layer based on the alarm information, the method further includes: The target abnormal field corresponding to the downstream data layer is repaired according to a preset abnormal repair mechanism, and the repair result is fed back to the upstream basic data layer.

5. The method according to claim 1, characterized in that In response to monitoring that a field of the basic data layer has an information change, generating alarm information corresponding to the field includes: In response to monitoring that a numeric field exceeds a preset change threshold or monitoring that a character field exceeds a preset proportion, alarm information corresponding to the field is generated.

6. The method according to claim 1, characterized in that After the repair results are fed back to the upstream basic data layer, it includes: In response to the basic data layer receiving the repair result, a synchronous repair process is performed on the fields of the basic data layer based on the repair result.

7. The method according to claim 1, characterized in that The method further comprises: The information change and the repair result are recorded in the scheduling platform, so that the scheduling platform can process the information change according to the repair result.

8. A data processing device, characterized in that: include: A monitoring module, configured to generate alarm information corresponding to a field in a basic data layer in response to monitoring that the field has an information change; wherein the information change includes logic adjustment and data anomaly; The alarm module is used to send the alarm information to the downstream data layer based on the pre-configured upstream and downstream dependencies.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images