Real-time bypass monitoring method, device and equipment of ODS and medium

Abstract

The invention provides a real-time bypass monitoring method and device for an ODS, equipment and a medium. The problem that the real-time performance of the ODS cannot be monitored through a current real-time ODS scheme is effectively solved. The method comprises the following steps: the host generates bypass flow time based on target operation information pushed by the source end module and received by the flow processing module, and forwards the target operation information to the ODS module; the host generates bypass ODS time based on ODS operation information generated when the ODS module executes the target operation, and the host generates a comparison instruction based on the newest bypass flow time and bypass ODS time in the REDIS module collected by the collection module; and the comparison module responds to the comparison instruction and compares the newest bypass flow time with the bypass ODS time to obtain a comparison result so as to confirm the real-time state of the ODS module based on the comparison result.

Description

Technical Field

[0001] The present application relates to the field of data monitoring technology, and in particular to a real-time bypass monitoring method, device, equipment and medium for ODS. Background Art

[0002] Data warehouse (DW) is a strategic collection of all types of data support for all levels of decision-making processes in an enterprise. It is created for analytical reporting and decision support purposes, and provides guidance for business process improvement, monitoring time, cost, quality, and control for enterprises that need business intelligence. ODS (Operational Data Store) is a data storage mode in the data warehouse. Its purpose is to support the daily operations of an enterprise. It can store various types of data such as real-time data, historical data, source data, etc., and has functions such as fast access, data integration, data cleaning, and deduplication.

[0003] As the demand for real-time data from product and business personnel continues to increase, new challenges arise. The real-time nature of ODS data is the foundation of the entire real-time data warehouse, and the real-time nature of ODS directly affects the relevant application experience of DW. There are already multiple real-time ODS solutions, but there is a lack of real-time monitoring solutions. ODS provides third-party monitoring, which makes it impossible to discover problems with data in ODS in advance and to avoid crises caused by data problems. Based on ODS monitoring, we can also better troubleshoot and optimize the real-time nature of ODS. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a real-time bypass monitoring method, device, equipment and medium for ODS, which effectively solves the problem that the current real-time ODS solution cannot realize real-time monitoring of ODS.

[0005] In a first aspect, an embodiment of the present application provides a real-time bypass monitoring method of an ODS, which is applied to a data warehouse, wherein the data warehouse includes a source module, an ODS module, a stream processing module, a REDIS module, and a host, wherein the host includes a collection module and a comparison module, and the method includes:

[0006] The host generates a bypass flow time based on the target operation information pushed by the source module and received by the stream processing module, sends the bypass flow time to the REDIS module, and forwards the target operation information to the ODS module; the target operation information includes a target operation;

[0007] The host generates bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, and sends the bypass ODS time to the REDIS module;

[0008] The host generates a comparison instruction based on the latest bypass flow time and bypass ODS time in the REDIS module collected by the collection module, and sends the comparison instruction to the comparison module;

[0009] The comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result.

[0010] In combination with the first aspect, an embodiment of the present application provides a first possible implementation of the first aspect, wherein the comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result, including:

[0011] The comparison module calls a preset comparison method based on the comparison instruction to compare the bypass flow time with the bypass ODS time and generate a comparison result;

[0012] The comparison module determines the real-time status of the ODS module based on the comparison result and generates a real-time status result.

[0013] In combination with the first aspect, the embodiment of the present application provides a second possible implementation of the first aspect, wherein the host further includes a computing module;

[0014] After the comparison module determines the real-time status of the ODS module based on the comparison result and generates the real-time status result, it includes:

[0015] The comparison module generates a delay calculation instruction based on the comparison result, and calls a calculation module based on the delay calculation instruction;

[0016] The calculation module calls a corresponding delay calculation method based on the delay calculation instruction to calculate the delay time corresponding to the real-time status result.

[0017] In combination with the first aspect, the embodiment of the present application provides a third possible implementation of the first aspect, wherein the host further includes an early warning module;

[0018] After the calculation module calculates the delay time corresponding to the real-time status result by calling the corresponding delay calculation method based on the delay calculation instruction, the calculation module includes:

[0019] The host determines whether the delay time and the real-time status result meet a preset warning condition;

[0020] If at least one of the preset warning conditions is met, the host generates a warning signal and calls the warning module based on the warning signal.

[0021] In combination with the first aspect, an embodiment of the present application provides a fourth possible implementation of the first aspect, wherein the host generates a bypass flow time based on the target operation information pushed by the source module and received by the stream processing module, sends the bypass flow time to the REDIS module, and forwards the target operation information to the ODS module, including:

[0022] The stream processing module determines the forwarding order of the target operation information based on the execution order of the target operations in the target operation information;

[0023] The ODS module executes the target operation in the target operation information based on the forwarding order.

[0024] In combination with the first aspect, an embodiment of the present application provides a fifth possible implementation of the first aspect, wherein the host generates a bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, including:

[0025] The ODS module generates a read instruction after receiving the target operation information, and calls a preset read tool based on the read instruction;

[0026] The ODS module controls the reading tool to read the target operation information based on a reading instruction.

[0027] In combination with the first aspect, the embodiment of the present application provides a sixth possible implementation of the first aspect, wherein the comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, and after confirming the real-time state of the ODS module based on the comparison result, further includes:

[0028] The host records the current execution time of the ODS module executing the target operation, and establishes a bypass table based on the current execution time, the bypass flow time, and the bypass ODS time;

[0029] After the host confirms the real-time status of the ODS module, it synchronizes the latest bypass flow time and the bypass ODS time to the bypass table.

[0030] In a second aspect, an embodiment of the present application provides a real-time bypass monitoring device for ODS, which is applied to a data warehouse, wherein the data warehouse includes a source module, an ODS module, a stream processing module, a REDIS module, and a host, wherein the host includes a collection module and a comparison module, and the device includes:

[0031] A push module, used for the host to generate a bypass flow time based on the target operation information pushed by the source module and received by the stream processing module, so as to send the bypass flow time to the REDIS module, and forward the target operation information to the ODS module; the target operation information includes a target operation;

[0032] An execution module, configured for the host to generate a bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, and send the bypass ODS time to the REDIS module;

[0033] A generating module, configured for the host to generate a comparison instruction based on the latest bypass flow time and bypass ODS time in the REDIS module collected by the collecting module, and send the comparison instruction to the comparison module;

[0034] A comparison module is used for the comparison module to respond to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result.

[0035] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of any one of the methods for real-time bypass monitoring of an ODS are performed.

[0036] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program executes the steps of any one of the methods for real-time bypass monitoring of an ODS.

[0037] An ODS real-time bypass monitoring method provided in an embodiment of the present application is applied to a data warehouse, wherein the data warehouse includes a source end module, an ODS module, a stream processing module, a REDIS module, and a host, wherein the host includes a collection module and a comparison module, wherein the method first generates a bypass flow time based on the target operation information pushed by the source end module received by the stream processing module by the host, so as to send the bypass flow time to the REDIS module, and forward the target operation information to the ODS module; the target operation information includes a target operation; secondly, the host generates a bypass ODS time based on the corresponding ODS operation information generated by reading the target operation information and executing the target operation by the ODS module, and the bypass ODS time is sent to the REDIS module; S time is sent to the REDIS module; then the host generates a comparison instruction based on the latest bypass flow time and bypass ODS time in the REDIS module collected by the acquisition module, and sends the comparison instruction to the comparison module; finally, the comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result, thereby realizing the effect of real-time status monitoring of the ODS module based on the bypass monitoring method, and does not affect the data processing of the source module, the stream processing module and the REDIS module, and effectively solves the problem that the current real-time ODS solution cannot realize the real-time monitoring of the ODS. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic flow chart of a first ODS real-time bypass monitoring method provided in an embodiment of the present application is shown;

[0040] Figure 2 A schematic diagram of a process for executing a target operation provided by an embodiment of the present application is shown;

[0041] Figure 3 A schematic diagram of a process for reading target operation information provided by an embodiment of the present application is shown;

[0042] Figure 4 A schematic structural diagram of a first ODS real-time bypass monitoring device provided in an embodiment of the present application is shown;

[0043] Figure 5A schematic structural diagram of a first electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0045] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0046] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0047] Since the real-time performance of ODS data is the basis of the entire real-time data warehouse, and the real-time performance of ODS directly affects the application experience of the data warehouse DW, there are already multiple real-time ODS solutions, but there is a lack of real-time monitoring solutions. The current real-time ODS solutions cannot provide third-party monitoring of ODS, resulting in the inability to discover problems with the data in ODS in advance and the inability to avoid crises caused by data problems.

[0048] Based on this, the embodiments of the present application provide a real-time bypass monitoring method, device, equipment and medium for ODS, which are described below through embodiments.

[0049] Example 1

[0050] To facilitate understanding of this embodiment, a real-time bypass monitoring method for ODS disclosed in an embodiment of the present application is first described in detail. Figure 1The flowchart of a real-time bypass monitoring method of an ODS shown in the figure, a real-time bypass monitoring method of an ODS provided by the present application is applied to a data warehouse, the data warehouse includes a source module, an ODS module, a stream processing module, a REDIS module, and a host, the host includes a collection module and a comparison module, the method includes:

[0051] S101, the host generates a bypass flow time based on the target operation information pushed by the source module and received by the stream processing module, sends the bypass flow time to the REDIS module, and forwards the target operation information to the ODS module; the target operation information includes a target operation;

[0052] S102, the host generates bypass ODS time based on the corresponding ODS operation information generated by reading the target operation information and executing the target operation by the ODS module, and sends the bypass ODS time to the REDIS module;

[0053] S103, the host generates a comparison instruction based on the latest bypass flow time and bypass ODS time in the REDIS module collected by the collection module, and sends the comparison instruction to the comparison module;

[0054] S104. The comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result.

[0055] In the present application, the source module is the source database in the data warehouse, the ODS module is the ODS database, the stream processing module is the stream processing platform kafka, the REDIS module is the REDIS library, and the host is an intelligent device capable of data collection, analysis and processing, including computers, servers and other intelligent devices.

[0056] In step S101, the host adopts a bypass monitoring method, referring to the Shareplex tool in Oracle, the source module uses its own database log and other tools to record the target operation information of the source module on the data in real time and push it to the stream processing module in real time. After the stream processing module receives the target operation information, the host generates a bypass flow time. The target operation information includes the target operation and the database information, architecture information, table information and SQL execution time commit_time where the SQL is located, and other ancillary information. The host generates a bypass flow time based on the target operation information pushed by the source module received by the stream processing module. The bypass flow time is obtained based on the latest SQL execution time max_commit_time. The bypass flow time is expressed as Kafka_time, which is abbreviated as Kt later, so as to send the bypass flow time to the REDIS module. The REDIS module is used to store the bypass flow time and the data generated by the log, and forward the target operation information to the ODS module.

[0057] In the specific implementation process of step S101, there is an embodiment in which the host generates a bypass flow time based on the target operation information pushed by the source module received by the stream processing module, sends the bypass flow time to the REDIS module, and forwards the target operation information to the ODS module, such as Figure 2 As shown, including:

[0058] S1011, the stream processing module determines the forwarding order of the target operation information based on the execution order of the target operations in the target operation information;

[0059] S1012. The ODS module executes the target operation in the target operation information based on the forwarding order.

[0060] In steps S1011-S1012, the source module will record based on the execution order when executing the target operation, that is, after executing an operation and recording the operation, after all operations are completed, the execution order of all operations will be recorded, that is, the target operation information also includes the execution order of the target material, that is, the source module has confirmed the priority of the target operation and determined the execution order according to the priority. The priority can be divided into multiple levels, such as high, medium, low, etc., or a more detailed division method can be adopted, such as using numbers or letters to represent different priorities. The stream processing module determines the forwarding order of the target operation information based on the execution order of the target operation in the target operation information. Based on the forwarding order, the ODS module can determine the order of executing the target operations in the target operation information based on the forwarding order, that is, the ODS module executes the target operation based on the same priority as the source module, thereby achieving the rationality and correspondence of the execution of the target operation.

[0061] In step S102, the ODS module reads the target operation information and executes the corresponding ODS operation information generated by the target operation. When the ODS module completes the target operation, the host generates a bypass ODS time based on the bypass monitoring method. The bypass ODS time is expressed as Ods_commit_time, which is abbreviated as Ct later. That is, the bypass ODS time is obtained based on the commit_time of the latest SQL execution, and the bypass ODS time is sent to the REDIS module. At the same time, the ODS operation information is also sent to the REDIS module, so that the REDIS module stores the bypass ODS time while storing the bypass flow time, the ODS operation information, and the auxiliary information generated when the ODS module executes the target operation.

[0062] In the specific implementation process of step S102, there is an embodiment in which the host generates bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, such as Figure 3 As shown, including:

[0063] S1021, the ODS module generates a read instruction after receiving the target operation information, and calls a preset read tool based on the read instruction;

[0064] S1022: The ODS module controls the reading tool to read the target operation information based on the reading instruction.

[0065] In steps S1021-S1022, the ODS module generates a read instruction after receiving the target operation information sent by the stream processing module, and responds to the read instruction, calling a preset read tool based on the read instruction. The read tool in this application refers to some preset programs, such as JAVA. After calling the read tool, the ODS module controls the JAVA and other reading tools based on the read instruction to read the target operation information in the stream processing module in real time, thereby ensuring the real-time and effectiveness of the reading. After reading the corresponding target operation from the target operation information, the ODS module executes the same execution order as the source module to execute the target operation and generate the corresponding ODS operation information.

[0066] In step S103, the REDIS module stores the bypass flow time and bypass ODS time received in real time, and the host controls the acquisition module to collect the latest bypass flow time and bypass ODS time from the REDIS module according to the preset sampling frequency, wherein the latest bypass flow time and bypass ODS time are respectively represented as Kt_new and Ct_new, and the sampling frequency is generally set to 10-30 minutes in actual use, which can be set specifically according to actual conditions. After collecting the latest bypass flow time and bypass ODS time, the host generates a comparison instruction and sends the comparison instruction to the comparison module; the acquisition module can also be collected according to the existing acquisition technology, which will not be elaborated here.

[0067] In step S104, after receiving the comparison instruction generated by the acquisition module, the comparison module responds to the comparison instruction, compares the latest collected bypass flow time with the bypass ODS time, and obtains the comparison result of the bypass flow time and the bypass ODS time, that is, the comparison result between Kt_new and Ct_new. The comparison result is expressed in various forms, which may be the form in which the bypass flow time Kt_new is greater than the bypass ODS time Ct_new, or in other forms, which are not elaborated here. Based on the comparison result, the host confirms the real-time status of the ODS module. Based on the determined real-time status, if there are continuous delay peaks in the real-time status, the host controls the comparison module and the business link to conduct joint analysis and evaluate the optimization plan to eliminate the delay peak and avoid a greater impact on the execution of the target operation by the ODS.

[0068] In the specific implementation process of step S104, there is an embodiment in which the comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result, including:

[0069] S10411, the comparison module calls a preset comparison method based on the comparison instruction to compare the bypass flow time with the bypass ODS time, and generates a comparison result;

[0070] S10412. The comparison module determines the real-time status of the ODS module based on the comparison result, and generates a real-time status result.

[0071] In steps S10411-S10412, the comparison module calls a preset comparison method based on the received comparison instruction. The preset comparison method is a pre-set rule or algorithm for performing a specific comparison task, including but not limited to direct comparison, difference comparison, threshold comparison, etc. The comparison module will use these preset comparison methods to compare the latest bypass flow time Kt_new with the bypass ODS time Ct_new. The present application adopts a direct comparison method to compare the latest bypass flow time Kt_new with the bypass ODS time Ct_new. The comparison result is that the latest bypass flow time Kt_new is greater than the bypass ODS time Ct_new or the bypass flow time Kt_new is less than the bypass ODS time Ct_new. ODS time Ct_new, or the bypass flow time Kt_new and the bypass ODS time Ct_new, the comparison module determines the real-time status of the ODS module based on the comparison result, and generates a real-time status result. In practice, the confirmed real-time status result of the ODS module includes four states: no delay state, no business state, synchronization interruption state and synchronization state (others), among which the no delay state is expressed as [Kt_new==Ct_new]; the no business state is expressed as [(Kt_new==Ct_new&&systime-Kt_new>=30 minutes]; the synchronization interruption state is expressed as [Ct==Ct_old&&Ct_new!=Kt_new&&Ot_new=Ot].

[0072] In the specific implementation process of step S10412, there is an embodiment in which: the host further includes a computing module;

[0073] After the comparison module determines the real-time status of the ODS module based on the comparison result and generates the real-time status result, it includes:

[0074] S1041211, the comparison module generates a delay calculation instruction based on the comparison result, and calls a calculation module based on the delay calculation instruction;

[0075] S1041212. The calculation module calls the corresponding delay calculation method based on the delay calculation instruction to calculate the delay time corresponding to the real-time status result.

[0076] In steps S1041211-S1041212, after generating the comparison result, the comparison module generates a delay calculation instruction based on the comparison result, that is, calculates the delay time generated by the ODS module when executing the target operation, and calls the calculation module based on the delay calculation instruction. The calculation module responds to the delay calculation instruction, determines the corresponding delay calculation method based on the real-time status result, that is, different statuses correspond to different delay calculation methods, and calculates based on the latest bypass flow time Kt_new and the bypass ODS time Ct_new to obtain the delay time corresponding to the real-time status result, such as no delay state: delay = 0; no business state: delay = 0; synchronization interruption state: delay = Kt_new-Ct_new; synchronization: delay = Kt_new-Ct_new.

[0077] In the specific implementation process of step S10412, there is another embodiment: the host further includes an early warning module;

[0078] After the calculation module calculates the delay time corresponding to the real-time status result by calling the corresponding delay calculation method based on the delay calculation instruction, the calculation module includes:

[0079] S1041221, the host determines whether the delay time and the real-time status result meet a preset warning condition;

[0080] S1041222. If at least one of the preset warning conditions is met, the host generates a warning signal and calls the warning module based on the warning signal.

[0081] In steps S1041211-S1041211, after calculating the delay time, the host sends the latest bypass flow time Kt_new, the bypass ODS time Ct_new and the corresponding delay time to the early warning module, and the early warning module determines whether the delay time and the real-time status result meet the preset early warning conditions. The preset early warning conditions are that the real-time status result is that the ODS module is in a synchronous interruption state or the delay time calculated by the calculation module is greater than 0. If at least one of the preset early warning conditions is met, that is, the real-time status result is a synchronous interruption state or the calculated delay time is greater than 0, or the real-time status result is a synchronous interruption state and the calculated delay time is greater than 0 at the same time, then the host generates an early warning signal, and calls the early warning module based on the early warning signal, and the early warning module issues an early warning through a preset alarm method, such as notifying the staff who monitors through the host or issuing an early warning on the display screen of the host.

[0082] In the specific implementation process of step S104, there is another embodiment in which: the comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, and after confirming the real-time state of the ODS module based on the comparison result, the method further includes:

[0083] S10421, the host records the current execution time of the ODS module executing the target operation, and establishes a bypass table based on the current execution time, the bypass flow time, and the bypass ODS time;

[0084] S10422. After the host confirms the real-time status of the ODS module, it synchronizes the latest bypass flow time and the bypass ODS time to the bypass table.

[0085] In steps S10421-S10422, the host records the current execution time of the ODS module executing the target operation, the current execution time is ods_update_time (subsequently recorded as Ot), and after the host confirms the real-time status of the ODS module, it synchronizes the latest bypass flow time Kt_new and the bypass ODS time Ct_new to the bypass table to manage the execution of the target operation by the ODS module, and this table can be displayed on the display screen of the host, which is convenient for the staff to read or master. The bypass table is shown in Table 1.

[0086] Table 1 Bypass table

[0087]

[0088] Example 2

[0089] The present application also provides a real-time bypass monitoring device for ODS, such as Figure 4 The block diagram of a real-time bypass monitoring device for ODS is shown. The functions implemented by the real-time bypass monitoring device for ODS correspond to the steps of executing a real-time bypass monitoring method for ODS on the terminal device. The device can be understood as a component of a server including a processor, which is applied to a data warehouse. The data warehouse includes a source module, an ODS module, a stream processing module, a REDIS module, and a host. The host includes a collection module and a comparison module. The device includes:

[0090] A push module 401, used for the host to generate a bypass flow time based on the target operation information pushed by the source module received by the stream processing module, to send the bypass flow time to the REDIS module, and forward the target operation information to the ODS module; the target operation information includes a target operation;

[0091] An execution module 402 is used for the host to generate a bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, and send the bypass ODS time to the REDIS module;

[0092] A generating module 403, configured for the host to generate a comparison instruction based on the latest bypass flow time and bypass ODS time in the REDIS module collected by the collecting module, and send the comparison instruction to the comparison module;

[0093] The comparison module 404 is used for the comparison module to respond to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result.

[0094] In a feasible implementation, the comparison module includes:

[0095] A first calling module, configured for the comparison module to call a preset comparison method based on the comparison instruction to compare the bypass flow time with the bypass ODS time and generate a comparison result;

[0096] The first judgment module is used for the comparison module to judge the real-time status of the ODS module based on the comparison result and generate a real-time status result.

[0097] In a feasible implementation manner, the comparison module further includes:

[0098] A second calling module, configured for the comparison module to generate a delay calculation instruction based on the comparison result, and to call a calculation module based on the delay calculation instruction;

[0099] A calculation module is used for the calculation module to call a corresponding delay calculation method based on the delay calculation instruction to calculate the delay time corresponding to the real-time status result.

[0100] In a feasible implementation, the comparison module further includes:

[0101] A second judgment module, used by the host to judge whether the delay time and the real-time status result meet a preset warning condition;

[0102] The third calling module is used for the host to generate an early warning signal and call the early warning module based on the early warning signal if at least one of the preset early warning conditions is met.

[0103] In a feasible implementation manner, the push module includes:

[0104] a forwarding module, used for the stream processing module to determine the forwarding order of the target operation information based on the execution order of the target operations in the target operation information;

[0105] The first execution module is used for the ODS module to execute the target operation in the target operation information based on the forwarding order.

[0106] In a feasible implementation, the generation module includes:

[0107] A fourth calling module, configured for the ODS module to generate a read instruction after receiving the target operation information, and to call a preset read tool based on the read instruction;

[0108] The reading module is used for the ODS module to control the reading tool to read the target operation information based on the reading instruction.

[0109] In a feasible implementation, the comparison module further includes:

[0110] An establishment module is used for the host to record the current execution time of the ODS module executing the target operation, and to establish a bypass table based on the current execution time, the bypass flow time, and the bypass ODS time;

[0111] A synchronization module is used to synchronize the latest bypass flow time and the bypass ODS time to the bypass table after the host confirms the real-time status of the ODS module.

[0112] Example 3

[0113] The present application also provides an electronic device, such as Figure 5 As shown, it includes: a processor 501, a memory 502 and a bus 503, the memory 502 stores machine-readable instructions executable by the processor 501, when the electronic device is running, the processor 501 communicates with the memory 502 through the bus 503, and when the machine-readable instructions are executed by the processor 501, any one of the steps of the real-time bypass monitoring method of ODS is executed.

[0114] Example 4

[0115] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the methods for real-time bypass monitoring of an ODS are executed.

[0116] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0117] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0119] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application can essentially be embodied in the form of a software product, or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a platform server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, ROM, RAM, magnetic disks, or optical disks.

[0120] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A real-time bypass monitoring method for ODS, characterized in that: Applied to a data warehouse, the data warehouse includes a source module, an ODS module, a stream processing module, a REDIS module, and a host, the host includes a collection module and a comparison module, and the method includes: The host generates a bypass flow time based on the target operation information pushed by the source module and received by the stream processing module, sends the bypass flow time to the REDIS module, and forwards the target operation information to the ODS module; the target operation information includes a target operation; The host generates bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, and sends the bypass ODS time to the REDIS module; The host generates a comparison instruction based on the latest bypass flow time and bypass ODS time in the REDIS module collected by the collection module, and sends the comparison instruction to the comparison module; The comparison module responds to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result.

2. The method according to claim 1, characterized in that The comparison module compares the latest bypass flow time with the bypass ODS time in response to the comparison instruction to obtain a comparison result, so as to confirm the real-time state of the ODS module based on the comparison result, including: The comparison module calls a preset comparison method based on the comparison instruction to compare the bypass flow time with the bypass ODS time and generate a comparison result; The comparison module determines the real-time status of the ODS module based on the comparison result and generates a real-time status result.

3. The method according to claim 2, characterized in that The host also includes a computing module; After the comparison module determines the real-time status of the ODS module based on the comparison result and generates the real-time status result, it includes: The comparison module generates a delay calculation instruction based on the comparison result, and calls a calculation module based on the delay calculation instruction; The calculation module calls a corresponding delay calculation method based on the delay calculation instruction to calculate the delay time corresponding to the real-time status result.

4. The method according to claim 3, characterized in that The host also includes an early warning module; After the calculation module calculates the delay time corresponding to the real-time status result by calling the corresponding delay calculation method based on the delay calculation instruction, the calculation module includes: The host determines whether the delay time and the real-time status result meet a preset warning condition; If at least one of the preset warning conditions is met, the host generates a warning signal and calls the warning module based on the warning signal.

5. The method according to claim 1, characterized in that The host generates a bypass flow time based on the target operation information pushed by the source module and received by the flow processing module, sends the bypass flow time to the REDIS module, and forwards the target operation information to the ODS module, including: The stream processing module determines the forwarding order of the target operation information based on the execution order of the target operations in the target operation information; The ODS module executes the target operation in the target operation information based on the forwarding order.

6. The method according to claim 1, characterized in that The host generates bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, including: The ODS module generates a read instruction after receiving the target operation information, and calls a preset read tool based on the read instruction; The ODS module controls the reading tool to read the target operation information based on a reading instruction.

7. The method according to claim 1, characterized in that After the comparison module compares the latest bypass flow time with the bypass ODS time in response to the comparison instruction to obtain a comparison result, and confirms the real-time state of the ODS module based on the comparison result, the method further includes: The host records the current execution time of the ODS module executing the target operation, and establishes a bypass table based on the current execution time, the bypass flow time, and the bypass ODS time; After the host confirms the real-time status of the ODS module, it synchronizes the latest bypass flow time and the bypass ODS time to the bypass table.

8. A real-time bypass monitoring device for ODS, characterized in that: Applied to a data warehouse, the data warehouse includes a source module, an ODS module, a stream processing module, a REDIS module, and a host, the host includes a collection module and a comparison module, and the device includes: A push module, used for the host to generate a bypass flow time based on the target operation information pushed by the source module and received by the stream processing module, so as to send the bypass flow time to the REDIS module, and forward the target operation information to the ODS module; the target operation information includes a target operation; An execution module, configured for the host to generate a bypass ODS time based on the corresponding ODS operation information generated by the ODS module reading the target operation information and executing the target operation, and send the bypass ODS time to the REDIS module; A generating module, configured for the host to generate a comparison instruction based on the latest bypass flow time and bypass ODS time in the REDIS module collected by the collecting module, and send the comparison instruction to the comparison module; A comparison module is used for the comparison module to respond to the comparison instruction to compare the latest bypass flow time with the bypass ODS time to obtain a comparison result, so as to confirm the real-time status of the ODS module based on the comparison result.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of a real-time bypass monitoring method for an ODS as described in any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the real-time bypass monitoring method of an ODS as claimed in any one of claims 1 to 7 are executed.

Images