Data loading monitoring method and device, equipment, storage medium and program product

Abstract

The invention discloses a data loading monitoring method and device, equipment, a storage medium and a program product, and relates to the technical field of data loading monitoring, and the method comprises the steps: receiving data loading information called back by a data warehouse; wherein the data loading information comprises first loading information called back when the data warehouse starts to load target data and / or second loading information called back after the data warehouse finishes loading the target data; according to the data loading information, determining whether an abnormal condition exists when the data warehouse loads the target data; and if the data warehouse has an abnormal condition when loading the target data, outputting abnormal condition alarm information. According to the method and the device, when the data in the business system is returned to the data warehouse, the loading condition of the data can be obtained, so that when the data loading is abnormal, the response is timely performed.

Description

Technical Field

[0001] The present application relates to the technical field of data loading monitoring, and in particular to a data loading monitoring method, device, equipment, storage medium and program product. Background Art

[0002] In order to facilitate the management of data in different business systems, it is necessary to return data from business systems to the data warehouse. There are two main ways to return data from business systems to the data warehouse: scheduled task loading and end-of-day batch loading.

[0003] However, no matter whether the data in the business system is returned to the data warehouse through scheduled task loading or end-of-day batch loading, the real-time status of the data in the business system flowing back to the data warehouse cannot be known, so when an abnormality occurs in data loading, it is impossible to respond in time. Summary of the invention

[0004] The main purpose of this application is to provide a data loading monitoring method, device, equipment, storage medium and program product, aiming to solve the technical problem of being unable to respond in time when an abnormality occurs in data loading.

[0005] To achieve the above objectives, the present application proposes a data loading monitoring method, comprising:

[0006] Receive data loading information called back by the data warehouse; wherein the data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing loading the target data;

[0007] Based on the data loading information, determine whether there are any abnormalities in the data warehouse when loading the target data;

[0008] If there is an abnormal situation when the data warehouse loads the target data, an abnormal situation alarm message will be output.

[0009] In one embodiment, the first loading information includes the loading start time, the second loading information includes the loading end time and the loading result, and the step of determining whether there is an abnormality when the data warehouse loads the target data according to the data loading information includes:

[0010] If the loading result is a loading failure, it is determined that an abnormality exists in the data warehouse when loading the target data; and / or,

[0011] Determine the current loading duration based on the loading start time and loading end time;

[0012] If the difference between the current loading duration and the historical loading duration is greater than a first preset threshold, it is determined that an abnormality exists in the data warehouse when loading the target data.

[0013] In one embodiment, before receiving the data loading information called back by the data warehouse, the method further includes:

[0014] Receive target data information of target data synchronized by the business system; the target data information includes the loading date of the target data, the system name of the business system storing the target data, the loading mode, the table name of the data table containing the target data, and the data ready time;

[0015] If the difference between the data-ready time and the preset data-ready time is greater than a second preset threshold, an abnormal situation alarm message is output.

[0016] In one embodiment, after receiving the data loading information called back by the data warehouse, the method further includes:

[0017] Determine whether there is target data information corresponding to the data loading information in the loading data table;

[0018] If the target data information corresponding to the data loading information exists in the loading data table, the loading data table is updated using the data loading information and the target data information corresponding to the data loading information;

[0019] If the target data information corresponding to the data loading information does not exist in the loading data table, the data loading information is inserted into the loading data table.

[0020] In one embodiment, the step of receiving target data information of target data synchronized by the business system includes:

[0021] Determine whether write failure information is received; write failure information is information that the target data information sent by the business system is not successfully written into the preset template;

[0022] If a write failure message is received, an abnormal situation alarm message is output;

[0023] If no write failure information is received, the target data information synchronized by the business system is received.

[0024] In one embodiment, after receiving the data loading information called back by the data warehouse, the method further includes:

[0025] If the number of times the data warehouse calls back the data loading information is greater than the preset number, a callback log table is constructed to store the data loading information called back by the data warehouse.

[0026] In addition, to achieve the above purpose, the present application also proposes a data loading monitoring device, the data loading monitoring device comprising:

[0027] A receiving module, used for receiving data loading information called back by the data warehouse; wherein the data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing loading the target data;

[0028] A determination module is used to determine whether there is an abnormality in the data warehouse when loading target data according to the data loading information;

[0029] The output module is used to output abnormal situation alarm information if there is an abnormal situation when the data warehouse loads the target data.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a data loading monitoring device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the data loading monitoring method described above.

[0031] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data loading monitoring method described above are implemented.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the data loading monitoring method described above are implemented.

[0033] One or more technical solutions proposed in this application have at least the following technical effects:

[0034] After the data warehouse starts loading the target data and completes the loading of the target data, the data warehouse calls back the data loading information. After receiving the callback data loading information, the data loading monitoring device can determine whether there is an abnormality in the data warehouse when loading the target data based on the data loading information. That is, this application uses the callback of data loading information to enable developers to grasp the loading details of the data in a timely manner. When an abnormality occurs in the loaded data, the developer can be notified accurately and quickly, thereby improving the ability to control risks and the responsiveness of problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 A flowchart of the first embodiment of the data loading monitoring method of the present application is provided;

[0038] Figure 2 Schematic diagram of the process of loading target data into the data warehouse and calling back the data loading information;

[0039] Figure 3 A schematic diagram of the system architecture consisting of the business system, data warehouse, and data loading monitoring equipment;

[0040] Figure 4 A schematic diagram of synchronizing target data information from the business system to the data loading monitoring center;

[0041] Figure 5 A flow chart for outputting abnormal situation alarm information;

[0042] Figure 6 This is a schematic diagram of the module structure of the data loading monitoring device according to an embodiment of the present application;

[0043] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the data loading monitoring method in the embodiment of the present application.

[0044] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0046] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0047] The main solution of the embodiment of the present application is: receiving data loading information called back by the data warehouse; wherein the data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing loading the target data; based on the data loading information, determining whether there is an abnormal situation when the data warehouse is loading the target data; if there is an abnormal situation when the data warehouse is loading the target data, outputting abnormal situation alarm information.

[0048] In this embodiment, for the convenience of description, the following description is made with the data loading monitoring device as the execution subject.

[0049] In order to facilitate the management of data in different business systems, the data in the business systems need to be returned to the data warehouse. There are two main ways to return data from the business systems to the data warehouse: scheduled task loading and end-of-day batch loading. For the end-of-day batch loading method, the business system will actively update the ready state flag of the data in the loading control table at the end of the day and set it to Y. When the data warehouse detects that the ready state flag of the loading control table is Y, it starts to batch load all relevant data for the day.

[0050] However, whether loading data from the business system back to the data warehouse through scheduled tasks or end-of-day batch loading, the following problems exist:

[0051] 1. Developers cannot perceive the data loading status of the data warehouse in real time.

[0052] Currently, developers have no channels to obtain data such as the loading progress, loading time, and number of loaded records of the data warehouse. When emergency maintenance or special circumstances occur at the end of the day and the data needs to be re-run, it is easy to miss the impact analysis on the data warehouse loading, resulting in dirty data being loaded into the data warehouse.

[0053] 2. Lack of warning for abnormal loading data.

[0054] When an exception occurs in the data loading process, there is no corresponding monitoring and early warning mechanism, and reliance on manual notification from the data warehouse results in a slow response to the problem.

[0055] 3. Developers cannot feel the historical fluctuations of loading data.

[0056] Developers cannot feel the historical fluctuations of loading data. There is no relevant early warning mechanism for some non-loading error types such as a sharp drop in the amount of loaded data, so the problem is easily overlooked.

[0057] Therefore, when the data warehouse is loading data from the business system, it cannot know the real-time situation of the data in the business system flowing back to the data warehouse, and thus cannot respond in time when an abnormality occurs in data loading.

[0058] The present application provides a solution. After the data warehouse starts loading the target data and completes the loading of the target data, the data warehouse calls back the data loading information. After receiving the callback data loading information, the data loading monitoring device can determine whether there is an abnormality in the data warehouse when loading the target data based on the data loading information. That is, the present application uses the callback of the data loading information to enable the developer to grasp the loading details of the data in a timely manner. When an abnormality occurs in the loaded data, the developer can be notified accurately and quickly, thereby improving the risk control ability and the responsiveness of the problem.

[0059] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a data loading monitoring device, etc. The following takes the data loading monitoring device as an example to illustrate this embodiment and the following embodiments.

[0060] Based on this, the present application embodiment provides a data loading monitoring method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the data loading monitoring method of the present application.

[0061] In this embodiment, the data loading monitoring method includes steps S10 to S30:

[0062] Step S10, receiving data loading information called back by the data warehouse.

[0063] The data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing loading the target data.

[0064] In this embodiment, the first loading information includes the loading start time, and the second loading information includes the loading end time and the loading result. In addition, the first loading information may also include the dataset identifier of the loaded target data and the expected loading amount, and the second loading information may also include the actual loaded data amount.

[0065] A business system can be a database server that generates or collects raw data. A data warehouse can be a database used to store data from one or more business systems.

[0066] Specifically, refer to Figure 2 As shown, Figure 2Flow chart of loading target data for a data warehouse and calling back data loading information. When loading target data in a business system (i.e., source system database), a data warehouse (i.e., data warehouse) sends a request to obtain a data ready flag to the business system. After receiving the acquisition request, the business system returns the data ready flag to the data warehouse. After receiving the data ready flag, the data warehouse determines whether the data ready flag is a preset flag, that is, whether the data ready flag is Y. When the data ready flag is Y, the data warehouse starts to load the target data in the business system, and calls back the first loading information to the downstream data management service application when starting to load the target data, that is, the data warehouse calls back the first loading information to the data loading monitoring device when starting to load the target data. After receiving the first loading information, the data loading monitoring device returns a callback result to the data warehouse to confirm that the callback first loading information has been received. When the data warehouse starts to load the target data, it sends a loading request to the business system, and the business system returns the target data to the business system according to the loading request. After loading all target data, the data warehouse will send second loading information to the data loading monitoring device. After receiving the second loading information, the data loading monitoring device will return a callback result to the data warehouse to confirm that the callback second loading information has been received.

[0067] It should be noted that when the data warehouse calls back the data loading information, due to network timeout and other issues, the data warehouse may not be able to receive the callback result returned by the data loading monitoring device in time. At this time, the data warehouse will repeatedly call back the data loading information to the data loading monitoring device. In order to avoid the loss of data loading information that is repeatedly called back by the data warehouse, so as to facilitate subsequent analysis and operation and maintenance, further, as an optional implementation method, when the data loading monitoring device receives multiple data loading information that are repeatedly called back by the data warehouse, it can construct a callback log table, and use the callback log table to store the multiple data loading information called back by the data warehouse, that is, after step S10, it also includes: if the number of times the data warehouse calls back the data loading information is greater than the preset number, then a callback log table is constructed, and the callback log table is used to store the data loading information called back by the data warehouse. Among them, the preset number of times can be set by the user according to actual needs, and this embodiment does not limit this.

[0068] In addition, when the data warehouse fails to successfully callback the data loading information to the data loading monitoring device, the data warehouse can also write the data loading information into the Kafka message queue, and the data loading monitoring device can asynchronously consume the data loading information in the Kafka message queue at a scheduled time.

[0069] It should be noted that, in actual applications, in order to ensure that the data warehouse can load the target data smoothly, the business system needs to first write the target data information to be loaded into the preset template. When the business system writes the target data information into the preset template, it may not be able to successfully write the target data information due to network delay, resulting in data backlog, and thus unable to ensure the smoothness of the data processing process. Therefore, in order to avoid data backlog and ensure the smoothness of the data processing process, further, as an optional real-time method, before step S10, it also includes:

[0070] Step S8, receiving target data information of target data synchronized by the business system.

[0071] The target data information includes the loading date of the target data, the system name of the business system storing the target data, the loading mode, the table name of the data table containing the target data, and the data ready time.

[0072] In this embodiment, the business system can initiate an RPC (Remote Procedure Call) call through the microservice center to synchronize the target data information of the target data to the data loading monitoring device.

[0073] It should be noted that in order to ensure that the data warehouse can correctly perform the loading operation, as well as to ensure data consistency between the data warehouse and the business system, and to reduce errors caused by inconsistent information, before the data warehouse loads the target data in the business system, the business system needs to write the target data information of the target data into the preset template. When the business system writes the target data information into the preset template through the SDK (Software Development Kit), if the field type in the preset template does not match the data type that the SDK attempts to write, it may cause a write failure, or network delays or network failures may cause the write request to fail to reach the database server where the business system is located, which may also cause a write failure. When the target data information fails to be written, the data warehouse may not be able to correctly perform the loading operation. In this case, in order to discover the problem in time, when the business system fails to write the target data information using the SDK, it can be notified immediately, rather than waiting until the data warehouse attempts to load the target data to discover the problem. Further, as an optional implementation method, step S8 specifically includes:

[0074] Step S81, determining whether write failure information is received.

[0075] The write failure information is information that the target data information sent by the business system is not successfully written into the preset template.

[0076] Step S82: If write failure information is received, output abnormal situation alarm information.

[0077] Step S83: If no write failure information is received, the target data information synchronized by the business system is received.

[0078] Specifically, the business system can write the target data information of the target data through the SDK. When the business system writes the target data information through the SDK, in order to avoid data compatibility issues and ensure that the SDK can write the target data information normally, it is necessary to unify the field types of the target data information of each business system, that is, to unify the target data information types of the target data, and provide a unified table creation statement for the three business systems of Oracle, Mysql, and TDsql.

[0079] When the business system writes the target data information into the preset template through the SDK, the business system can determine whether the target data information is successfully written into the preset template through the exception capture mechanism. When the target data information is successfully written into the preset template, the business system initiates an RPC call through the microservice center to synchronize the target data information to the data loading monitoring device; when the target data information is not successfully written into the preset template, the business system can send a write failure message to the data loading monitoring device. After receiving the write failure message, the data loading monitoring device can output an abnormal situation alarm message to prompt the developer to handle the abnormal situation in time.

[0080] In this embodiment, when the target data information fails to be written, the business system can send a write failure message to the data loading monitoring device. After receiving the write failure message, the data loading monitoring device outputs an abnormal situation alarm message, thereby discovering the problem in time, rather than waiting until the data warehouse attempts to load the target data.

[0081] Step S9: if the difference between the data ready time and the preset data ready time is greater than a second preset threshold, output abnormal situation alarm information.

[0082] In this embodiment, the data-ready time may be the time taken to write the target data information into the preset template. The preset data-ready time may be the average time taken by the business system to write the data information of the data into the preset template within a preset time period, such as the average time taken by the business system to write the data information into the preset template in the past 30 days.

[0083] Specifically, after receiving the target data information, the data loading monitoring device can determine whether there is a network delay problem when the business system writes the target data information into the preset template based on the data ready time in the target data information and the preset data ready time, that is, calculate the difference between the data ready time and the preset data ready time. When the difference between the data ready time and the preset data ready time is greater than the second preset threshold, it is determined that there is a network delay, and an abnormal situation alarm message is output at this time; when the difference between the data ready time and the preset data ready time is not less than the second preset threshold, it is determined that there is no network extension, and no abnormal situation alarm message is output at this time. Among them, the second preset threshold can be set by the user according to actual needs, such as setting the second preset threshold to 60 minutes, and this embodiment does not limit this.

[0084] In this embodiment, after receiving the target data information, the data loading monitoring device determines whether there is a network delay when the business system writes the target data information based on the data ready time in the target data information, and outputs abnormal situation alarm information when there is a network delay, thereby quickly discovering the abnormal delay in the data ready time, intervening in the investigation and solving the problem in time, thereby avoiding data backlogs and ensuring a smooth data processing flow.

[0085] In order to ensure the traceability, monitorability and auditability of the data warehouse during the data loading process and improve the accuracy, completeness and reliability of data processing, further, as an optional implementation, after step S10, it also includes:

[0086] Step S111, determining whether there is target data information corresponding to the data loading information in the loading data table.

[0087] Step S112: if the target data information corresponding to the data loading information exists in the loading data table, the loading data table is updated using the data loading information and the target data information corresponding to the data loading information.

[0088] Step S113: if the target data information corresponding to the data loading information does not exist in the loading data table, insert the data loading information into the loading data table.

[0089] In this embodiment, the loading data table may be stored locally in the data loading monitoring device, and is used to record target data information synchronized by the business system and data loading information called back by the data warehouse.

[0090] Specifically, when the network is stable and there is no delay, the target data information synchronized by the business system will reach the data loading monitoring device before the data loading information called back by the data warehouse. After receiving the target data information synchronized by the business system, the data loading monitoring device will insert the target data information into the data loading table, and when receiving the data loading information corresponding to the target data information, it will use the data loading information and the target data information corresponding to the data loading information to update the data loading table to avoid data loss.

[0091] However, due to the instability of the network, there are some extreme cases, such as the data loading information called back by the data warehouse will arrive at the data loading monitoring device before the target data information synchronized by the business system. If the data loading information called back by the data warehouse arrives before the target data information synchronized by the business system, the operation of directly using the target data information to update the data loading table may fail because the corresponding record cannot be found, resulting in data loss. Therefore, after receiving the data loading information called back by the data warehouse, it is necessary to determine whether there is target data information corresponding to the data loading information in the data loading table. When there is target data information corresponding to the data loading information in the data loading table, the data loading information and the target data information corresponding to the data loading information can be used to update the loading data table, which can ensure that only the existing records are updated to avoid data loss. When there is no target data information corresponding to the data loading information in the loading data table, the data loading information is inserted into the loading data table to avoid data loss.

[0092] In this embodiment, when the data loading information called back by the data warehouse arrives at the data loading monitoring device before the target data information of the business system, if the update operation is directly performed, the update may fail because the record to be updated may not exist yet. By checking the data loading table to determine whether there is target data information corresponding to the data loading information, the risk of data loss can be avoided.

[0093] Step S20: determining, based on the data loading information, whether there is any abnormality in the data warehouse when loading the target data.

[0094] Specifically, when determining whether there is an abnormality in the data warehouse when loading the target data based on the loading information, it is possible to determine whether there is an abnormality in the data warehouse when loading the target data based on the loading result. For example, when the loading result is a loading failure, it is determined that there is an abnormality in the data warehouse when loading the target data. Alternatively, correspondingly, it is also possible to determine whether there is an abnormality in the data warehouse when loading the target data based on the loading start time, the loading end time, and the historical loading duration. For example, the current loading duration is calculated based on the loading start time and the loading end time, and it is determined whether the difference between the current loading duration and the historical loading duration is greater than a first preset threshold. When the difference between the current loading duration and the historical loading duration is greater than the first preset threshold, it is determined that there is an abnormality in the data warehouse when loading the target data. When the difference between the current loading duration and the historical loading duration is not greater than the first preset threshold, it is determined that there is no abnormality in the data warehouse when loading the target data.

[0095] It should be noted that when determining whether there is an abnormality in the data warehouse when loading the target data, it can be determined only based on the loading result, or it can also be determined only based on the loading start time, loading end time and historical loading duration, or it can also be determined based on the loading result, loading start time, loading end time and historical loading duration, that is, if the loading result is a loading failure, it is determined that there is an abnormality in the data warehouse when loading the target data; and / or, according to the loading start time and the loading end time, the current loading duration is determined; if the difference between the current loading duration and the historical loading duration is greater than the first preset threshold, it is determined that there is an abnormality in the data warehouse when loading the target data. Among them, the historical loading duration can be the average time taken for the data warehouse to load data within a preset time period, such as the average time taken for the data warehouse to load data within 30 days before loading the target data. The first preset threshold can be set by the user according to actual needs, such as 30 minutes, and this embodiment does not limit this.

[0096] In this embodiment, when the data warehouse starts to load the target data in the business system, it will call back the first loading information to the data loading monitoring device, so that the data loading monitoring device can determine whether the data warehouse has loaded the target data according to the agreed time when loading the target data through the scheduled task loading mode according to the loading start time in the first loading information. At the same time, after completing the loading of the target data, the data warehouse calls back the second loading information to the data loading monitoring device, and can promptly determine whether there is an abnormality in the data loading according to the loading result in the second loading information, so as to find problems in time.

[0097] Step S30: If an abnormality occurs when the data warehouse loads the target data, an abnormality alarm message is output.

[0098] Specifically, when it is determined that an abnormal situation exists in the data warehouse when loading target data, an abnormal situation alarm information can be output so that developers can handle the abnormal situation in time and ensure the smoothness of the data processing process.

[0099] In this embodiment, after the data warehouse starts loading the target data and completes the loading of the target data, the data warehouse calls back the data loading information. After receiving the callback data loading information, the data loading monitoring device can determine whether there is an abnormality in the data warehouse when loading the target data based on the data loading information. That is, this application uses the callback of data loading information to enable developers to grasp the loading details of the data in a timely manner. When an abnormality occurs in the loaded data, the developer can be notified accurately and quickly, thereby improving the ability to control risks and the responsiveness of problems.

[0100] To facilitate a better understanding of this embodiment, a specific example is provided below.

[0101] Reference Figure 3 As shown, Figure 3 The diagram of the system architecture consisting of the business system, data warehouse, and data loading monitoring device. The business system (i.e., database) writes the target data information of business application 1 and business application 2 into the preset template through the SDK, that is, writes the data control information into the database. After the business system successfully writes the target data information into the preset template, it can initiate an RPC call through the microservice center to synchronize the target data information to the data loading monitoring center, that is, initiate an RPC call through the microservice center to synchronize the data control information to the data management application layer.

[0102] It should be noted that when the business system initiates an RPC call through the microservice center to synchronize the target data information to the data loading monitoring center, the synchronization may fail due to network and other issues. When the business system fails to synchronize the target data information, it can repeat the operation of initiating an RPC call through the microservice center to synchronize the target data information to the data loading monitoring center until the number of repeated operations reaches the preset number. When the business system repeats the above operation the preset number of times, the target data information can be written to Kafka, and the data loading monitoring device can asynchronously consume the target data information in Kafka. For example, refer to Figure 4 As shown, Figure 4 Schematic diagram of synchronizing target data information from the business system to the data loading monitoring center.

[0103] After the business system successfully writes the target data information into the preset template, the business system can send a data ready flag to the data warehouse. After receiving the data ready flag, the data warehouse can load the target data from the business system, that is, the data warehouse processes the data in the database and calls back the data loading information to the data loading monitoring device.

[0104] After receiving the target data information and the data loading information, the data loading monitoring device can determine whether to output abnormal situation alarm information according to the target data information and the data loading information, such as referring to Figure 5 As shown, Figure 5 The figure is a flowchart for outputting abnormal situation alarm information. The triggering of abnormal situation alarm information output by the data loading monitoring device mainly comes from three scenarios: download synchronization interface call, download callback interface call, download synchronization and abnormal topic consumption. When any scenario is triggered, the logical calculation task of the warning rule submitted in the warning thread pool meets the warning conditions and outputs abnormal situation alarm information to prompt the developer.

[0105] The logical computing tasks corresponding to the three scenarios are:

[0106] 1. Call the download synchronization interface.

[0107] When the business system writes the target data information to the preset template through the SDK, but the operation fails, this usually means that the target data information cannot be recorded normally, thus affecting the subsequent data warehouse loading operation. In order to ensure that the data warehouse can successfully load the target data from the business system, when the business system fails to successfully write the target data information to the preset template through the SDK, a write failure message is generated and sent to the data loading monitoring device. When the data loading monitoring device receives the write failure message, it outputs an abnormal situation alarm message.

[0108] 2. Call the next count callback interface.

[0109] The data warehouse will call back the data loading information to the data loading monitoring device when it starts loading the target data and / or ends. After receiving the data loading information called back by the data warehouse, the data loading monitoring device can determine whether to output abnormal situation alarm information based on the data loading information. If the loading result is a loading failure, it is determined that the data warehouse has an abnormal situation when loading the target data, and an abnormal situation alarm information is output. Alternatively, the current loading duration is calculated based on the loading start time and the loading end time, and it is determined whether the difference between the current loading duration and the historical loading duration is greater than a first preset threshold. When the difference between the current loading duration and the historical loading duration is greater than the first preset threshold, it is determined that the data warehouse has an abnormal situation when loading the target data, and an abnormal situation alarm information is output.

[0110] 3. Download data synchronization and abnormal topic consumption.

[0111] After the business system successfully writes the target data information into the preset template through the SDK, the business system can initiate an RPC call through the microservice center to synchronize the target data information to the data loading monitoring device. After receiving the target data information, the data loading monitoring device can determine whether to output abnormal situation alarm information based on the data ready time in the target data information and the preset data ready time, that is, to issue a warning to notify the developer to handle it in time.

[0112] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the data loading monitoring method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0113] This application also provides a data loading monitoring device, please refer to Figure 6 , the data loading monitoring device comprises:

[0114] A receiving module, used for receiving data loading information called back by the data warehouse; wherein the data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing loading the target data;

[0115] A determination module is used to determine whether there is an abnormality in the data warehouse when loading target data according to the data loading information;

[0116] The output module is used to output abnormal situation alarm information if there is an abnormal situation when the data warehouse loads the target data.

[0117] The data loading monitoring device provided by the present application adopts the data loading monitoring method in the above embodiment, which can solve the technical problem of being unable to respond in time when an abnormality occurs in data loading. Compared with the prior art, the beneficial effects of the data loading monitoring device provided by the present application are the same as the beneficial effects of the data loading monitoring method provided by the above embodiment, and the other technical features in the data loading monitoring device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0118] The present application provides a data loading monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the data loading monitoring method in the above-mentioned embodiment one.

[0119] Reference below Figure 7, which shows a schematic diagram of the structure of a data loading monitoring device suitable for implementing the embodiment of the present application. The data loading monitoring device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The data loading monitoring device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Figure 7 As shown, the data loading monitoring device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the data loading monitoring device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the data loading monitoring device to communicate with other devices wirelessly or by wire to exchange data. Although the data loading monitoring device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0121] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be loaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0122] The data loading monitoring device provided by the present application adopts the data loading monitoring method in the above embodiment, which can solve the technical problem of being unable to respond in time when an abnormality occurs in data loading. Compared with the prior art, the beneficial effects of the data loading monitoring device provided by the present application are the same as the beneficial effects of the data loading monitoring method provided by the above embodiment, and the other technical features in the data loading monitoring device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0123] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0124] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0125] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the data loading monitoring method in the above-mentioned embodiment.

[0126] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0127] The computer-readable storage medium may be included in the data loading monitoring device; or may exist independently without being assembled into the data loading monitoring device.

[0128] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the data loading monitoring device, the data loading monitoring device: receives data loading information called back by the data warehouse; wherein the data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing the loading of the target data; determines whether there is an abnormal situation when the data warehouse loads the target data based on the data loading information; if there is an abnormal situation when the data warehouse loads the target data, outputs abnormal situation alarm information.

[0129] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0131] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0132] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned data loading monitoring method, and can solve the technical problem of being unable to respond in time when an abnormality occurs in data loading. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the data loading monitoring method provided in the above-mentioned embodiment, and will not be repeated here.

[0133] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned data loading monitoring method when executed by a processor.

[0134] The computer program product provided by the present application can solve the technical problem of being unable to respond in time when data loading is abnormal. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the data loading monitoring method provided by the above embodiment, which will not be repeated here.

[0135] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A data loading monitoring method, characterized in that: The data loading monitoring method comprises: Receive data loading information called back by the data warehouse; wherein the data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing loading the target data; Determining, according to the data loading information, whether there is an abnormality in the data warehouse when loading the target data; If an abnormal situation occurs when the data warehouse loads the target data, an abnormal situation alarm message is output.

2. The data loading monitoring method according to claim 1, characterized in that: The first loading information includes a loading start time, the second loading information includes a loading end time and a loading result, and the step of determining whether there is an abnormality in the data warehouse when loading the target data according to the data loading information includes: If the loading result is a loading failure, it is determined that an abnormality exists in the data warehouse when loading the target data; and / or, Determine the current loading duration according to the loading start time and the loading end time; If the difference between the current loading duration and the historical loading duration is greater than a first preset threshold, it is determined that an abnormality exists in the data warehouse when loading the target data.

3. The data loading monitoring method according to claim 1, characterized in that: Before receiving the data loading information called back by the data warehouse, the method further includes: Receive target data information of the target data synchronized by the business system; the target data information includes the loading date of the target data, the system name of the business system storing the target data, the loading mode, the table name of the data table containing the target data, and the data ready time; If the difference between the data-ready time length and the preset data-ready time length is greater than a second preset threshold, an abnormal situation alarm message is output.

4. The data loading monitoring method according to claim 3, characterized in that: After receiving the data loading information called back by the data warehouse, the method further includes: Determine whether the target data information corresponding to the data loading information exists in the loading data table; If the target data information corresponding to the data loading information exists in the loading data table, updating the loading data table by using the data loading information and the target data information corresponding to the data loading information; If the target data information corresponding to the data loading information does not exist in the loading data table, the data loading information is inserted into the loading data table.

5. The data loading monitoring method according to claim 3, characterized in that: The step of receiving target data information of the target data synchronized by the business system comprises: Determine whether write failure information is received; the write failure information is information that the target data information sent by the business system is not successfully written into the preset template; If a write failure message is received, an abnormal situation alarm message is output; If no write failure information is received, the target data information synchronized by the business system is received.

6. The data loading monitoring method according to claim 1, characterized in that: After receiving the data loading information called back by the data warehouse, the method further includes: If the number of times that the data warehouse calls back the data loading information is greater than a preset number, a callback log table is constructed, and the callback log table is used to store the data loading information called back by the data warehouse.

7. A data loading monitoring device, characterized in that: The data loading monitoring device comprises: A receiving module, used for receiving data loading information called back by the data warehouse; wherein the data loading information includes first loading information called back by the data warehouse when starting to load the target data and / or second loading information called back by the data warehouse after completing loading the target data; A determination module, used to determine whether there is an abnormality in the data warehouse when loading the target data according to the data loading information; The output module is used to output abnormal situation alarm information if an abnormal situation occurs when the data warehouse loads the target data.

8. A data loading monitoring device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the data loading monitoring method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the data loading monitoring method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the data loading monitoring method according to any one of claims 1 to 6 are implemented.

Images