Business data monitoring method and device, electronic equipment, medium and program product

Abstract

The application discloses a business data monitoring method and device, electronic equipment, medium and product. The method comprises the following steps: acquiring a unified event stream containing whole-stage change information of a business process; performing partition processing on the unified event stream based on a business identifier to obtain an event substream corresponding to each business identifier; and performing business state monitoring on the event substream corresponding to each business identifier. The method can solve the problem of poor real-time performance of business monitoring.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a business data monitoring method, device, electronic device, readable storage medium, computer program product, and an anomaly monitoring and alarm method for financial data. Background Technology

[0002] In sectors like finance, where timeliness of business processes is extremely critical, real-time monitoring of the entire business data lifecycle is essential. Currently, existing technologies generally rely on periodic polling and database queries. For example, business data is synchronized to the analytics platform via T+1 reconciliation or background tasks, and then time stamps are compared using program scripts to identify expired documents. However, in practice, it has been found that existing methods suffer from high alarm latency, making it difficult to meet the demands of second-level real-time monitoring. Therefore, existing methods struggle to achieve efficient and timely end-to-end status monitoring of massive amounts of business data. Summary of the Invention

[0003] In view of the above problems, this application provides a business data monitoring method, device, electronic device, readable storage medium, computer program product, and an anomaly monitoring and alarm method for financial data, which can solve the problem of poor real-time performance in business monitoring.

[0004] Firstly, this application provides a business data monitoring method, including: Obtain a unified event stream containing change information for all stages of the business process; The unified event stream is partitioned based on the service identifier to obtain the event sub-stream corresponding to each service identifier; The service status is monitored for each event substream corresponding to the service identifier.

[0005] In the above technical solution, this method can break through the limitations of traditional fragmented data collection and solve the problem of data silos, thereby providing complete and coherent data for end-to-end monitoring. At the same time, through the precise aggregation of single business data, it can also avoid cross-interference of multi-business data, improve the targeting of monitoring and processing efficiency, thereby helping to achieve real-time perception and precise location of anomalies, thus avoiding the problem of high alarm latency and meeting the business monitoring needs of high timeliness fields.

[0006] In some implementations, obtaining a unified event stream containing change information across all stages of a business process includes: Retrieve raw data change events for each business database in the preset data source; Each of the original data change events is filtered to obtain valid business information for each business database; wherein, the valid business information includes at least key business information, which includes at least user identifier, business identifier, business status, and event occurrence time; Each piece of valid business information is standardized to obtain a change event object corresponding to each business database. All the aforementioned change event objects are merged to obtain a unified event flow containing change information for all stages of the business process.

[0007] In the above technical solution, this method can extract core and effective business information by accurately filtering raw data change events, ensuring the relevance and availability of monitoring data; at the same time, it can unify the format of change events in various business databases by standardizing the processing, eliminating heterogeneous differences between different data sources; finally, it can form a unified event flow covering the entire business process by merging, providing a high-quality data foundation for subsequent accurate partitioning and real-time monitoring.

[0008] In some implementations, obtaining the original data change events for each business database in a preset data source includes: Log data from each business database in a preset data source is obtained through a pre-deployed source connector; The log data of each of the business databases is parsed to obtain the log parsing data of each of the business databases; The operation data for each of the business databases is obtained based on the parsed log data. The original data change event for each of the business databases is generated based on the operation data.

[0009] In the above technical solution, the method can directly capture log data from the business database through the source connector, realize real-time data change collection, and get rid of the latency limitation of traditional timed queries. At the same time, through the layer-by-layer parsing and transformation of log data, the operation behavior of the database is accurately extracted and the original data change events are generated, thereby ensuring the authenticity and accuracy of subsequent data processing.

[0010] In some implementations, the step of partitioning the unified event stream based on service identifiers to obtain event sub-streams corresponding to each service identifier includes: Obtain the business identifier from each change event object in the unified event stream; The unified event stream is partitioned according to the business identifier to obtain multiple event sub-streams; wherein, each event sub-stream contains only all change event objects corresponding to a single business identifier.

[0011] In the above technical solution, the method can achieve precise partitioning of a unified event stream based on business identifiers, ensuring that a single event sub-stream focuses on the full-process change information of the corresponding business, avoiding the mixing and interference of different business data, and greatly improving the targeting and processing efficiency of subsequent business status monitoring.

[0012] In some implementations, monitoring the service status of the event substream corresponding to each service identifier includes: Define a state storage variable for each of the service identifiers; wherein, the state storage variable is used to store the event state and timeout timestamp of the change event object corresponding to the service identifier; Based on the change event object in the event sub-stream and the state storage variable, the business status corresponding to each business identifier is monitored.

[0013] In the above technical solution, the method can accurately retain the event status and timeout timestamp of the entire business process by configuring a dedicated state storage variable for each business identifier, thus building a state tracking benchmark for a single business. At the same time, relying on the linkage monitoring of event sub-streams and state storage variables, the business progress status is dynamically verified, thereby timely capturing abnormal situations that deviate from the normal process, thereby improving the accuracy and timeliness of business status monitoring.

[0014] In some implementations, monitoring the service status corresponding to each service identifier based on the change event object in the event substream and the state storage variable includes: The event sub-stream corresponding to the currently monitored target service identifier is detected. When the arrival of the first change event object is detected and the state storage variable corresponding to the target service identifier is empty, the current timestamp, the event state of the first change event object, and the preset timeout period are obtained. Calculate the timeout timestamp based on the preset timeout period and the current timestamp; The event status and the timeout timestamp are stored in the status storage variable corresponding to the target service identifier, and the current timer is determined based on the timeout timestamp; When a subsequent change event object corresponding to the target service identifier is detected to arrive within the preset timeout period, it is determined that the business process corresponding to the target service identifier has not experienced any abnormalities. Delete the current timer, update the state storage variable, and determine a new current timer; If a new subsequent change event object is detected again within a new preset timeout period determined based on the updated state storage variable before the business process corresponding to the target business identifier has ended, then the steps of deleting the current timer, updating the state storage variable, and determining a new current timer are executed.

[0015] In the above technical solution, this method can build a dynamic end-to-end timeout monitoring mechanism by configuring dedicated state storage variables and timers for the target business identifier. At the same time, it updates the state variables and timeout timestamps in real time based on the event arrival node and resets the timer synchronously to achieve accurate tracking of the business process progress. Furthermore, by using the arrival of the event as the criterion for judging the normal process, it can also avoid the lag of traditional timed polling, thereby identifying anomalies at the first moment when a timeout occurs, and greatly improving the real-time performance and accuracy of business status monitoring.

[0016] In some embodiments, the method further includes: If no subsequent change event object corresponding to the target service identifier is detected within the preset timeout period, and the current timer is triggered, it is determined that the business process corresponding to the target service identifier has an anomaly, and the step of obtaining the abnormal business data corresponding to the abnormal target service identifier is executed.

[0017] In the above technical solution, the method can immediately determine the business process is abnormal and trigger the abnormal data acquisition process if no subsequent change event object is detected within a preset timeout period, thereby avoiding the alarm delay in the traditional polling mode and realizing the second-level identification and response of abnormalities.

[0018] In some embodiments, the method further includes: When an anomaly is detected in the business status, obtain the abnormal business data corresponding to the target business identifier that caused the anomaly; An abnormal business document object is generated based on the abnormal business data; wherein, the abnormal business document object includes at least the identifier of the target business that caused the abnormality and the event status of the timeout. The attribution matching is determined based on the abnormal business document object and the preset attribution method to obtain the matching result; wherein, the attribution method is a configurable attribution method or a large model attribution method; Based on the matching results, obtain the cause of the anomaly and the solution to the anomaly; The output includes the abnormal business document object, the cause of the abnormality, and the solution to the abnormality, as well as business abnormality alarm information.

[0019] In the above technical solution, the method can generate standardized abnormal business document objects based on abnormal business data and clearly integrate core abnormal information. At the same time, it combines two attribution methods, namely configuration-based or large model, to complete accurate attribution matching, quickly locate the root cause of the abnormality and provide corresponding solutions. Finally, it outputs complete alarm information containing abnormal documents, causes and solutions, and can also realize full-process management from abnormal identification to abnormal alarm, thereby greatly improving the efficiency of abnormal handling and helping to reduce business risks.

[0020] Secondly, this application provides a business data monitoring device, comprising: The first acquisition unit is used to acquire a unified event stream containing change information for all stages of the business process. A partitioning unit is used to perform partitioning processing on the unified event stream based on service identifiers to obtain an event sub-stream corresponding to each service identifier; The monitoring unit is used to monitor the service status of the event sub-stream corresponding to each service identifier.

[0021] In the above technical solution, the device can break through the limitations of traditional fragmented data collection and solve the problem of data silos, thereby providing complete and coherent data for end-to-end monitoring. At the same time, through the precise aggregation of single business data, it can also avoid cross-interference of multi-business data, improve the targeting of monitoring and processing efficiency, thereby helping to achieve real-time perception and precise location of anomalies, thereby avoiding the problem of high alarm latency and meeting the business monitoring needs of high timeliness fields.

[0022] Thirdly, this application provides a method for monitoring and alerting anomalies in financial data, including: A unified event flow that obtains change information for all stages of financial business processes from multi-source heterogeneous financial databases; The unified event stream is partitioned based on the financial business identifier to obtain multiple event sub-streams that correspond one-to-one with the financial business identifier. For each event sub-stream, the financial business status is monitored, and when a timeout anomaly is detected in the financial business status, the abnormal business data corresponding to the target business identifier of the timeout anomaly is obtained; the financial business status includes at least acceptance, clearing, and settlement. Output timeout alarm information based on the abnormal business data.

[0023] In the above technical solution, the method can realize real-time tracking of each branch business based on the entire financial business process; at the same time, by focusing on the status of key links such as acceptance, clearing, and settlement, it can quickly identify timeout anomalies and simultaneously complete the location of abnormal data and alarm output, thereby achieving efficient and accurate monitoring of financial business anomalies.

[0024] Fourthly, this application provides an electronic device, the electronic device including a memory and a processor, the memory for storing a computer program, the processor running the computer program to cause the electronic device to perform the business data monitoring method described in any one of the first aspects.

[0025] Fifthly, this application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the business data monitoring method described in any one of the first aspects.

[0026] Sixthly, this application provides a computer program product, which includes a computer program that, when executed by a processor, performs the business data monitoring method described in any one of the first aspects.

[0027] The beneficial effects of this application are as follows: it enables comprehensive identification of business anomalies based on data results, thereby solving the problem of missing anomaly scenarios in traditional manual data logging methods; simultaneously, by monitoring the database in real time, it can also provide immediate alerts when anomalies occur, improving the real-time performance of alerts; furthermore, relying on the event-driven model, it can significantly reduce the polling pressure on the database; in addition, based on the Flink distributed architecture, it can achieve horizontal scaling by adding computing nodes, giving it higher performance and strong scalability; finally, by using CDC to read logs for access, it can achieve low intrusion into the source system without making any changes to the business code, thereby minimizing the impact on the normal operation of the business system. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 This is a flowchart illustrating the business data monitoring method in some embodiments of this application; Figure 2 This is a schematic diagram of the overall architecture of a business data monitoring system according to some embodiments of this application; Figure 3 This is a schematic diagram of the structure of the business data monitoring device in some embodiments of this application; Figure 4 This is a flowchart illustrating the abnormal monitoring and alarm method for financial data in some embodiments of this application; Figure 5 This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation

[0030] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0032] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces) unless otherwise explicitly defined.

[0033] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0034] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0035] Existing technologies generally rely on periodic polling and database queries. For example, business data is synchronized to the analysis platform through T+1 reconciliation or background tasks, and then time-out documents are identified by comparing timestamps using program scripts. However, existing methods suffer from high alarm latency, making it difficult to meet the requirements of second-level real-time monitoring. On the other hand, due to the increasing complexity of business logic, traditional scripts struggle to provide a comprehensive and accurate description, resulting in limited monitoring coverage and ultimately failing to achieve accurate and efficient real-time anomaly monitoring and location for massive amounts of business data.

[0036] To address the aforementioned technical issues, this application provides a business data monitoring method. This method, based on the perspective of business results, uses "whether business document data reaches the expected state within a specified time" as the core criterion for anomaly judgment. Then, by sorting out the complete state nodes and SLA standards of the entire business chain through data flow, and relying on the high-performance state-based computing and time processing capabilities of the stream processing engine, a dedicated "monitoring state machine" is built for each business transaction to accurately capture abnormal business data. At the same time, by integrating configurable strategies and AI large model capabilities, it can also achieve rapid location of the root cause of anomalies and timely delivery of alarm information.

[0037] like Figure 1 As shown, some embodiments of this application provide a business data monitoring method, which includes: S110. Obtain a unified event flow containing change information for all stages of the business process.

[0038] S120. Perform partitioning processing on the unified event stream based on the business identifier to obtain the event sub-stream corresponding to each business identifier.

[0039] S130. Monitor the service status of the event sub-stream corresponding to each service identifier.

[0040] In this embodiment, the above-mentioned business status is set differently according to different application scenarios. Taking the financial scenario as an example, the business status can be document acceptance status, clearing completion status, settlement initiation status, etc.

[0041] In this embodiment, the service status monitoring in this application can be anomaly monitoring. When an anomaly is detected in the service status, the abnormal service data corresponding to the target service identifier of the anomaly is obtained. Subsequently, the method can also perform anomaly analysis, alarm, anomaly interception, anomaly repair, and other processes based on the abnormal service data.

[0042] In this embodiment, the business status monitoring in this application can also monitor whether a specific business status occurs. If it does, the associated business process corresponding to that business status is triggered. For example, when the business status is monitored to change to "start clearing", the associated risk control system's real-time transaction verification process is automatically triggered.

[0043] In this embodiment, business status monitoring can also record statistical data on business status, so as to perform relevant analysis based on the statistical data.

[0044] In this embodiment, a unified event stream refers to a real-time event data stream that integrates data change information from each stage of a business process from initiation to completion, forming a unified format and a coherent time sequence. The acquisition of the unified event stream is real-time and continuous.

[0045] In this embodiment, the service identifier refers to the feature code or character combination used to uniquely identify each service, which can achieve accurate differentiation between different services.

[0046] In this embodiment, an event sub-stream refers to a sub-data stream that corresponds one-to-one with a single business identifier and contains the entire process change information of that business after the unified event stream is partitioned according to the business identifier.

[0047] In this embodiment, the target service identifier refers to the unique identifier corresponding to the service whose service status is monitored to be abnormal.

[0048] In this embodiment, abnormal business data refers to relevant data in the business process corresponding to the target business identifier that deviates from normal business logic or fails to meet preset state requirements.

[0049] For example, the abnormal business data may include, but is not limited to: timeout data where the document acceptance status has exceeded the preset time limit and has not yet been transferred to the settlement completion status; error data where the fund transfer amount in the settlement completion status does not match the document amount; and missing data where no compliant settlement voucher has been collected in the settlement initiation status.

[0050] As an optional implementation, before obtaining the unified event flow containing change information for all stages of the business process, the method further includes: Deploy source connectors for each business database in the preset data source; The preset data source includes multiple independent business databases, each storing business data from different stages of the business process.

[0051] In the above embodiments, the method can break through the limitations of traditional fragmented data collection and solve the problem of data silos, thereby providing complete and coherent data for end-to-end monitoring. At the same time, through the precise aggregation of single business data, it can also avoid cross-interference of multi-business data, improve the targeting of monitoring and processing efficiency, thereby helping to achieve real-time perception and precise location of anomalies, thereby avoiding the problem of high alarm latency and meeting the business monitoring needs of high timeliness fields.

[0052] In some embodiments, step S110 includes: S111. Obtain the original data change events for each business database in the preset data source; S112. Filter and process each original data change event to obtain valid business information for each business database; among which, valid business information includes at least key business information, which includes at least user identifier, business identifier, business status and event occurrence time; S113. Standardize each valid business information to obtain the change event object corresponding to each business database; S114. Merge all change event objects to obtain a unified event flow containing change information for all stages of the business process.

[0053] In this embodiment, the original data change event refers to the unprocessed original event record generated when data addition, modification, deletion or other operations occur in each business database of the preset data source.

[0054] In this embodiment, effective business information refers to the core information with business analysis value extracted after filtering the original data change events, which includes at least user identifier, business identifier, business status, and event occurrence time.

[0055] In this embodiment, standardization refers to the process of organizing valid business information from different business databases according to a unified format, field naming rules, and data specifications.

[0056] In this embodiment, the change event object refers to the event data unit formed by the standardized processing of valid business information, which has a unified structure and can be directly used for merging.

[0057] For example, this method filters the captured raw data change events to obtain valid business information while discarding invalid information. Then, the method uniformly parses and encapsulates the valid information into a standard Java object `BillEvent` (i.e., completes standardization processing and generates the corresponding change event object). This `BillEvent` object contains key information such as: seller ID, business identifier (bizNo), current business status (status, such as document acceptance, settlement completion, settlement initiation), and event occurrence time (eventTime).

[0058] In this embodiment, the method can merge change event object streams from multiple data sources into a unified event stream containing change information from all business stages through Flink's union operator.

[0059] In the above embodiments, the method can extract core and effective business information by accurately filtering raw data change events, ensuring the relevance and availability of monitoring data; at the same time, it can unify the format of change events in various business databases by standardizing the processing, eliminating heterogeneous differences between different data sources; finally, it can form a unified event stream covering the entire business process by merging, providing a high-quality data foundation for subsequent accurate partitioning and real-time monitoring.

[0060] In some embodiments, step S111 includes: Log data from each business database in a preset data source is obtained through a pre-deployed source connector; The log data of each business database is parsed to obtain the log parsing data of each business database; Obtain the operation data for each business database based on log parsing data; Generate raw data change events for each business database based on the operational data.

[0061] In this embodiment, the source connector refers to a dedicated component pre-deployed in the system to connect to various business databases of a preset data source and realize log data collection and transmission.

[0062] In this embodiment, log data refers to the original log file that records data addition, modification, deletion and other operations and related contexts during the operation of the business database.

[0063] In this embodiment, log parsing data refers to the structured and identifiable data information obtained after parsing the format and extracting the content of the original log data in the business database.

[0064] In this embodiment, operational data refers to the core data that is filtered and extracted from log parsing data and can reflect specific database operation behaviors (such as adding, modifying, and deleting) and related data.

[0065] For example, this method no longer uses database queries, but instead deploys a Flink CDC source connector for each source data table (document table, settlement table, and clearing table). This source connector simulates the interaction with a MySQL slave database, reading and parsing the database's binary log in real time to accurately capture INSERT, UPDATE, and other operations on the relevant data tables. It then converts these operations into JSON-formatted data change events, thereby generating the original data change events.

[0066] In the above embodiments, the method can directly capture log data of the business database through the source connector, realize real-time collection of data changes, and get rid of the latency limitations of traditional timed queries. At the same time, by parsing and transforming the log data layer by layer, the operation behavior of the database is accurately extracted and the original data change events are generated, thereby ensuring the authenticity and accuracy of subsequent data processing.

[0067] In some embodiments, step S120 includes: S121. Obtain the business identifier from each change event object in the unified event flow; S122. Perform data partitioning on the unified event stream according to the business identifier to obtain multiple event sub-streams; where each event sub-stream contains only all change event objects corresponding to a single business identifier.

[0068] In this embodiment, data partitioning refers to the process of splitting a unified event stream containing multiple business information into different data partitions based on the business identifier in the change event object, thereby achieving independent aggregation of single business data.

[0069] In this embodiment, the event sub-stream specifically refers to an independent data stream that contains only all change event objects corresponding to a single business identifier after the unified event stream has been partitioned, ensuring the integrity and independence of the data throughout the entire process of a single business.

[0070] For example, to process all change event objects corresponding to the same business identifier (bizNo) continuously, this method performs a Flink keyBy(BillEvent::getBizNo) partitioning operation on the unified event stream. This operation ensures that all change event objects associated with the same business identifier, regardless of their source data table, will be distributed to the same processing instance (Task), thus laying the foundation for subsequent stateful monitoring of business status.

[0071] In the above embodiments, the method can achieve precise partitioning of a unified event stream based on the business identifier, ensuring that a single event sub-stream focuses on the full-process change information of the corresponding business, avoiding the mixing and interference of different business data, and greatly improving the targeting and processing efficiency of subsequent business status monitoring.

[0072] In some embodiments, step S130 includes: S131. Define a state storage variable for each service identifier; wherein, the state storage variable is used to store the event state and timeout timestamp of the change event object corresponding to the service identifier; S132. Based on the change event objects and state storage variables in the event sub-stream, monitor the business status corresponding to each business identifier.

[0073] In this embodiment, the state storage variable refers to a dedicated variable defined separately for each service identifier, used to persistently store the event state of the change event object corresponding to the service identifier, as well as core monitoring parameters such as the preset timeout timestamp.

[0074] In this embodiment, the event status refers to the stage status of the business reflected by the change event object, such as acceptance, clearing, settlement, etc., which is the core basis for judging whether the business is abnormal.

[0075] In this embodiment, the timeout timestamp refers to the timestamp corresponding to the preset time threshold for each service state, which is used to determine whether the service state exceeds the specified processing time.

[0076] In this embodiment, business status monitoring refers to the process of comparing the change event objects in the event sub-stream with the data in the status storage variables in real time, and continuously tracking whether the business status corresponding to each business identifier conforms to the preset rules.

[0077] For example, this method can use the processing logic inherited from FlinkKeyedProcessFunction (the core low-level processing function for keyed streams in Apache Flink) for the event sub-streams after partitioning, and rely on the two core mechanisms provided by Flink to carry out business status monitoring.

[0078] The first mechanism is the ValueState-based mechanism. This mechanism allows you to define a ValueState. <billstate>This is a state storage variable of type BillState. BillState is a custom object specifically used to store the event state (e.g., CREATED) corresponding to the current business identifier (bizNo), as well as the timestamp of the timeout timer registered for that business state.

[0079] The second mechanism is based on the TimerService mechanism. This mechanism allows for the registration and deletion of time-based timers, providing a core time-based basis for subsequently determining whether a business status has timed out.

[0080] In the above embodiments, the method can accurately retain the event status and timeout timestamp of the entire business process by configuring a dedicated state storage variable for each business identifier, thereby building a state tracking benchmark for a single business. At the same time, relying on the linkage monitoring of event sub-streams and state storage variables, the business progress status is dynamically verified, thereby timely capturing abnormal situations that deviate from the normal process, and thus improving the accuracy and timeliness of business status monitoring.

[0081] In some embodiments, step S132 includes: The event sub-stream corresponding to the target business identifier currently being monitored is detected. When the arrival of the first change event object is detected and the state storage variable corresponding to the target business identifier is empty, the current timestamp, the event status of the first change event object, and the preset timeout period are obtained. Calculate the timeout timestamp based on the preset timeout period and the current timestamp; Store the event status and timeout timestamp in the status storage variable corresponding to the target service identifier, and determine the current timer based on the timeout timestamp; When a subsequent change event object corresponding to the target business identifier is detected to arrive within the preset timeout period, it is determined that the business process corresponding to the target business identifier has not experienced any abnormalities. Delete the current timer, update the state storage variables, and determine the new current timer; If a new subsequent change event object is detected again within the new preset timeout period determined based on the updated state storage variables before the business process corresponding to the target business identifier has ended, the steps of deleting the current timer, updating the state storage variables, and determining a new current timer will be executed.

[0082] In this embodiment, the first change event object refers to the first change event object detected by the monitoring system in the event sub-stream corresponding to the target service identifier, which is the starting event that triggers the service status monitoring process.

[0083] In this embodiment, the current timestamp refers to the timestamp corresponding to the specific time recorded by the system when the monitoring system detects the arrival of the first change event object, and it is the basis for calculating the timeout timestamp.

[0084] In this embodiment, the preset timeout period refers to the maximum allowed processing time for each business state, which is the core time threshold for determining whether a business has timed out.

[0085] In this embodiment, the current timer refers to a timing component created based on the timeout timestamp for the target service identifier, used to monitor whether the service status is updated within the preset timeout period.

[0086] In this embodiment, subsequent change event objects refer to other change event objects that appear successively in the event sub-stream corresponding to the target service identifier after the first change event object arrives, reflecting the subsequent updates of the service status.

[0087] In this embodiment, the new current timer refers to the timing component recreated based on the new timeout timestamp after the business status is updated, which is used to continuously monitor the processing time of the updated business status.

[0088] For example, this method can check the corresponding ValueState type state storage variable when the first change event object (BillEvent) corresponding to a business identifier (bizNo) arrives; at this time, the variable is empty. Here, ValueState refers to the value state mechanism in the Flink framework, which is the core component for implementing state storage, used to persistently store the state data (such as event state, timeout timestamp) of a single business identifier; in this method, a specific ValueState is defined. <billstate>Type of state storage variable, in which <billstate>Encapsulate custom state objects.

[0089] Then, the event state of the changed event object (such as document acceptance) and the calculated SLA timeout timestamp (the sum of the current timestamp and the preset timeout period for this stage) are stored in the state storage variable. Simultaneously, using TimerService (the timer service mechanism in the Flink framework, used to register and delete timers, and a core tool for implementing timeout detection), a current timer triggered at this timeout timestamp is registered. Here, SLA refers to Service Level Agreement, specifically the preset timeout standard for each business state; it is the same concept as "preset timeout period."

[0090] In addition, when the subsequent change event object corresponding to the business identifier (such as BillEvent with a status of liquidation completion) arrives, the corresponding status storage variable is read again.

[0091] Finally, retrieve the timestamp of the previously registered old timer from the state storage variable, and then delete the old timer using the deleteProcessingTimeTimer() method of TimerService (this is a function provided by TimerService to delete timers, which removes the old timers from the previous business state).

[0092] At this point, since the business has entered the next phase, the timeout monitoring of the previous phase no longer needs to continue. Therefore, the subsequent method can update the state storage variables to the new event state (liquidation completed) and the new SLA timeout timestamp, and register a new current timer based on the new timeout timestamp.

[0093] As an optional implementation, deleting the current timer includes: Read the stored timeout timestamp from the state storage variable corresponding to the target business identifier; Delete the corresponding current timer based on the timeout timestamp.

[0094] As an optional implementation, updating the state storage variable includes: Get the new current timestamp, the event status of subsequent change event objects, and the new preset timeout period; Calculate the new timeout timestamp based on the new preset timeout period and the new current timestamp; The state storage variable corresponding to the target business identifier is updated based on the event status of the subsequent change event object and the new timeout timestamp, and a new current timer is determined based on the new timeout timestamp.

[0095] As an optional implementation, the method further includes: If no subsequent change event object corresponding to the target business identifier is detected within the preset timeout period, and the current timer is triggered, it is determined that the business process corresponding to the target business identifier has an anomaly, and the step of obtaining the abnormal business data corresponding to the target business identifier that has an anomaly is executed.

[0096] In this embodiment, timer triggering refers to the automatic triggering of a timeout alarm action when the current timer corresponding to the target service identifier reaches the preset timeout timestamp. It is a key triggering condition for determining abnormalities in the business process.

[0097] In this embodiment, a business process abnormality refers to a state in which the target business does not detect the arrival of subsequent change event objects within a preset timeout period, and when the timer is triggered, the business process is determined to be in a state that does not conform to the preset time rules.

[0098] For example, if no subsequent change event object corresponding to the target business identifier (bizNo) is detected within the preset SLA (preset timeout period) and the current timer is not deleted, the method can automatically call the onTimer() callback method (the callback method of KeyedProcessFunction in the Flink framework, which is the core execution entry point when the timeout timestamp is reached) through Flink.

[0099] Subsequently, the method verifies the current business status in the onTimer() method, confirms that a business process exception has occurred, and creates an AbnormalBillInfo exception object (i.e., exception business data) containing a business identifier (bizNo) and a timeout status. Then, it sends the exception object to the Kafka server (a mainstream distributed message queue service used to receive and store exception alarm data) through Flink's side output stream (SideOutput, a dedicated data stream mechanism for separating exception data in the Flink framework).

[0100] Finally, the method clears the ValueState type state storage variable corresponding to the business identifier (bizNo) and releases the relevant system resources.

[0101] In the above embodiments, the method can construct a dynamic end-to-end timeout monitoring mechanism by configuring dedicated state storage variables and timers for the target business identifier; at the same time, it updates the state variables and timeout timestamps in real time based on the event arrival node and resets the timer synchronously to achieve accurate tracking of the business process progress rhythm; then, by using the event arrival as the criterion for judging the normal process, it can also avoid the lag of traditional timed polling, thereby identifying anomalies at the first time a timeout occurs, and greatly improving the real-time performance and accuracy of business status monitoring.

[0102] In some embodiments, after step S130, the method further includes: S140. When an abnormality is detected in the business status, obtain the abnormal business data corresponding to the target business identifier that is abnormal. S150. Generate an abnormal business document object based on the abnormal business data; wherein, the abnormal business document object includes at least the target business identifier of the abnormality and the event status of the timeout. S160. Determine the attribution matching based on the abnormal business document object and the preset attribution method, and obtain the matching result; wherein, the attribution method is a configurable attribution method or a large model attribution method; S170. Obtain the cause of the anomaly and the solution to the anomaly based on the matching results; S180. Output business exception alarm information including the abnormal business document object, the cause of the exception, and the solution to the exception.

[0103] In this embodiment, the abnormal business document object refers to a structured data object generated based on the encapsulation of abnormal business data, which at least includes core information such as the target business identifier where the abnormality occurred and the event status where the timeout occurred.

[0104] In this embodiment, attribution method refers to the analytical means used to locate the root cause of business anomalies, specifically divided into two categories: configurable attribution method and large model attribution method.

[0105] In this embodiment, the configurable attribution method refers to an attribution method that locates the cause of an anomaly by matching rules based on a preset anomaly attribution rule base.

[0106] In this embodiment, the large model attribution method refers to the attribution method that relies on the data analysis and reasoning capabilities of AI large models to perform in-depth analysis of abnormal business data in order to locate the root cause.

[0107] In this embodiment, attribution matching refers to the process of matching abnormal business document objects with preset attribution methods to select attribution methods that are suitable for the abnormal scenario.

[0108] In this embodiment, the matching result refers to the attribution method selection result and the corresponding preliminary analysis conclusion after the abnormal business document object and the attribution method have been adapted and matched.

[0109] In this embodiment, business anomaly alarm information refers to the structured alarm notification data that can be directly reached by relevant personnel after integrating the abnormal business document object, the cause of the anomaly, and the solution to the anomaly.

[0110] For example, this method can first rely on the abnormal message receiving mechanism to obtain the aforementioned abnormal business data, and then divide it into three core processing links: abnormal message receiving, attribution, and alarm.

[0111] The attribution process supports configurable attribution methods. Developers can pre-configure an exception attribution rule base consisting of "timeout status + priority + conditional expression + matching pattern (regular expression / AI) + exception reason". This method matches the exception business document object with the rule base to locate the specific exception reason and the corresponding exception solution.

[0112] Finally, this method can integrate key information such as business identifier (bizNo), timeout status, cause of anomaly, and solution to generate standardized business anomaly alarm information, and output it through the alarm platform.

[0113] In the above embodiments, the method can generate standardized abnormal business document objects based on abnormal business data, clearly integrating core abnormal information; at the same time, it combines two attribution methods, configuration-based or large model, to complete accurate attribution matching, quickly locate the root cause of the abnormality and provide corresponding solutions; finally, it outputs complete alarm information containing abnormal documents, causes and solutions, and can also realize full-process management from abnormal identification to abnormal alarms, thereby greatly improving the efficiency of abnormal handling and helping to reduce business risks.

[0114] To make the purpose, technical solution and advantages of this application clearer, the technical solution in this application will be clearly and completely described below.

[0115] In some embodiments, Figure 2 A schematic diagram of the overall architecture of a business data monitoring system is shown, wherein the business data monitoring system includes: 1. Data Source Layer: This layer contains multiple independent MySQL business databases, storing data from different stages of the business process, such as the "Document Database," "Settlement Database," and "Settlement Database." The streaming engine captures changes to the business data and standardizes it into a unified data model. Finally, the data model is partitioned according to business identifiers to achieve logical isolation between business documents.

[0116] 2. Real-time Processing Core Layer: This layer includes a state manager and a timer server. It defines the business state machine and the SLA for each state. For each of the above partitions, it manages the business state and timeouts. It identifies and issues timeout-exceeding business documents.

[0117] 3. Anomaly Result Attribution and Alarm Layer: Receives the aforementioned anomaly data and, through configuration capabilities or large model capabilities, locates the specific cause of the anomaly. Subsequently, it processes the business data containing the anomaly cause into an alarm.

[0118] Figure 3 A schematic diagram of a business data monitoring device is shown. It should be understood that this device is related to... Figure 1 The method executed in the middle corresponds to the steps involved in the aforementioned method. The specific functions and effects of the device can be found in the description above. To avoid repetition, detailed descriptions are omitted here.

[0119] The business data monitoring device includes: The first acquisition unit 210 is used to acquire a unified event stream containing change information for all stages of the business process. Partitioning unit 220 is used to perform partitioning processing on the unified event stream based on the business identifier to obtain the event sub-stream corresponding to each business identifier; The monitoring unit 230 is used to monitor the service status of the event sub-stream corresponding to each service identifier.

[0120] In some embodiments, the first acquisition unit 210 includes: The first acquisition subunit 211 is used to acquire the original data change events of each business database in the preset data source; The filtering subunit 212 is used to filter each original data change event to obtain valid business information for each business database; wherein, the valid business information includes at least key business information, which includes at least user identifier, business identifier, business status and event occurrence time; Standardization subunit 213 is used to standardize each valid business information to obtain the change event object corresponding to each business database. Merging subunit 214 is used to merge all change event objects to obtain a unified event flow containing change information for all stages of the business process.

[0121] In some embodiments, the first acquisition subunit 211 is specifically used to acquire log data of each business database in a preset data source through a pre-deployed source connector; The first acquisition subunit 211 is also used to parse the log data of each business database to obtain the log parsing data of each business database. The first acquisition subunit 211 is also used to obtain the operation data of each business database based on log parsing data; The first acquisition subunit 211 is also used to generate the original data change events for each business database based on the operation data.

[0122] In some embodiments, the partitioning unit 220 includes: The second acquisition subunit 221 is used to acquire the business identifier in each change event object in the unified event flow; The partitioning subunit 222 is used to perform data partitioning processing on the unified event stream according to the business identifier to obtain multiple event substreams; wherein, an event substream contains only all change event objects corresponding to a business identifier.

[0123] In some embodiments, the monitoring unit 230 includes: Define subunit 231 to define a state storage variable for each service identifier; wherein, the state storage variable is used to store the event state and timeout timestamp of the change event object corresponding to the service identifier; The monitoring subunit 232 is used to monitor the business status corresponding to each business identifier based on the change event object and state storage variable in the event substream.

[0124] In some embodiments, the monitoring subunit 232 is specifically used to detect the event substream corresponding to the target service identifier currently being monitored, and when the arrival of the first change event object is detected and the state storage variable corresponding to the target service identifier is empty, to obtain the current timestamp, the event status of the first change event object and the preset timeout period. The monitoring subunit 232 is also used to calculate the timeout timestamp based on the preset timeout period and the current timestamp; The monitoring subunit 232 is also used to store the event status and timeout timestamp into the status storage variable corresponding to the target service identifier, and to determine the current timer based on the timeout timestamp; The monitoring subunit 232 is also used to determine that the business process corresponding to the target business identifier has not experienced any abnormality when the subsequent change event object corresponding to the target business identifier is detected to arrive within the preset timeout period. The monitoring subunit 232 is also used to delete the current timer, update the state storage variables, and determine the new current timer; The monitoring subunit 232 is further configured to, before the business process corresponding to the target business identifier has ended, within a new preset timeout period determined based on the updated state storage variable, if a newly arrived subsequent change event object is detected again, delete the current timer, update the state storage variable, and determine a new current timer.

[0125] In some embodiments, the business data monitoring device further includes: The second acquisition unit 240 is used to determine that the business process corresponding to the target business identifier has an anomaly when no subsequent change event object corresponding to the target business identifier is detected within a preset timeout period and the current timer is triggered, and to acquire the abnormal business data corresponding to the target business identifier that has an anomaly.

[0126] In some embodiments, the business data monitoring device further includes: The second acquisition unit 240 is also used to acquire abnormal business data corresponding to the target business identifier that has an abnormality when an abnormality is detected in the business status. The generation unit 250 is used to generate abnormal business document objects based on abnormal business data; wherein, the abnormal business document object includes at least the target business identifier where the abnormality occurred and the event status where the timeout occurred. Matching unit 260 is used to determine attribution matching based on the abnormal business document object and the preset attribution method, and obtain the matching result; wherein, the attribution method is a configurable attribution method or a large model attribution method; The third acquisition unit 270 is used to acquire the cause of the anomaly and the solution to the anomaly based on the matching result; Output unit 280 is used to output business exception alarm information, including the abnormal business document object, the cause of the exception, and the solution to the exception.

[0127] like Figure 4 As shown, some embodiments of this application provide a method for monitoring and alerting anomalies in financial data. This method includes: S301. Obtain a unified event flow from multi-source heterogeneous financial databases to obtain change information for all stages of financial business processes.

[0128] S302. Based on the financial business identifier, the unified event stream is partitioned to obtain multiple event sub-streams that correspond one-to-one with the financial business identifier.

[0129] S303. Monitor the financial business status of each event sub-stream, and when a timeout anomaly is detected in the financial business status, obtain the abnormal business data corresponding to the target business identifier of the timeout anomaly; the financial business status includes at least acceptance, clearing, and settlement.

[0130] S304. Output timeout alarm information based on abnormal business data.

[0131] In the above embodiments, the method can realize real-time tracking of each branch business based on the entire financial business process; at the same time, by focusing on the status of key links such as acceptance, clearing, and settlement, it can quickly identify timeout anomalies and simultaneously complete the location of abnormal data and alarm output, thereby achieving efficient and accurate monitoring of financial business anomalies.

[0132] like Figure 5 As shown, this application provides an electronic device 400, which includes a processor 401 and a memory 402. The processor 401 and the memory 402 are interconnected and communicate with each other through a communication bus 403 and / or other forms of connection mechanism (not shown). The memory 402 stores a computer program that can be executed by the processor 401. When the computing device is running, the processor 401 executes the computer program to perform the method in any of the aforementioned optional implementations.

[0133] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method in any of the aforementioned optional implementations.

[0134] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] This application provides a computer program product, which includes a computer program that, when run by a processor, executes the method in any of the aforementioned optional implementations.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.< / billstate> < / billstate> < / billstate>

Claims

1. A business data monitoring method, characterized in that, include: Obtain a unified event stream containing change information for all stages of the business process; The unified event stream is partitioned based on the service identifier to obtain the event sub-stream corresponding to each service identifier; The service status is monitored for each event substream corresponding to the service identifier.

2. The business data monitoring method according to claim 1, characterized in that, The acquisition of the unified event stream containing change information for all stages of the business process includes: Retrieve raw data change events for each business database in the preset data source; Each of the original data change events is filtered to obtain valid business information for each business database; wherein, the valid business information includes at least key business information, which includes at least user identifier, business identifier, business status, and event occurrence time; Each piece of valid business information is standardized to obtain a change event object corresponding to each business database. All the aforementioned change event objects are merged to obtain a unified event flow containing change information for all stages of the business process.

3. The business data monitoring method according to claim 2, characterized in that, The step of obtaining the original data change events of each business database in the preset data source includes: Log data from each business database in a preset data source is obtained through a pre-deployed source connector; The log data of each of the business databases is parsed to obtain the log parsing data of each of the business databases; The operation data for each of the business databases is obtained based on the parsed log data. The original data change event for each of the business databases is generated based on the operation data.

4. The business data monitoring method according to claim 1, characterized in that, The step of partitioning the unified event stream based on service identifiers to obtain event sub-streams corresponding to each service identifier includes: Obtain the business identifier from each change event object in the unified event stream; The unified event stream is partitioned according to the business identifier to obtain multiple event sub-streams; wherein, each event sub-stream contains only all change event objects corresponding to a single business identifier.

5. The business data monitoring method according to claim 1, characterized in that, The step of monitoring the service status of the event sub-stream corresponding to each service identifier includes: Define a state storage variable for each of the service identifiers; wherein, the state storage variable is used to store the event state and timeout timestamp of the change event object corresponding to the service identifier; Based on the change event object in the event sub-stream and the state storage variable, the business status corresponding to each business identifier is monitored.

6. The business data monitoring method according to claim 5, characterized in that, The monitoring of the business status corresponding to each business identifier based on the change event object in the event sub-stream and the state storage variable includes: The event sub-stream corresponding to the currently monitored target service identifier is detected. When the arrival of the first change event object is detected and the state storage variable corresponding to the target service identifier is empty, the current timestamp, the event state of the first change event object, and the preset timeout period are obtained. Calculate the timeout timestamp based on the preset timeout period and the current timestamp; The event status and the timeout timestamp are stored in the status storage variable corresponding to the target service identifier, and the current timer is determined based on the timeout timestamp; When a subsequent change event object corresponding to the target service identifier is detected to arrive within the preset timeout period, it is determined that the business process corresponding to the target service identifier has not experienced any abnormalities. Delete the current timer, update the state storage variable, and determine a new current timer; If a new subsequent change event object is detected again within a new preset timeout period determined based on the updated state storage variable before the business process corresponding to the target business identifier has ended, then the steps of deleting the current timer, updating the state storage variable, and determining a new current timer are executed.

7. The business data monitoring method according to claim 6, characterized in that, The method further includes: If no subsequent change event object corresponding to the target service identifier is detected within the preset timeout period, and the current timer is triggered, it is determined that the business process corresponding to the target service identifier has an anomaly, and the abnormal business data corresponding to the abnormal target service identifier is obtained.

8. The business data monitoring method according to claim 1, characterized in that, The method further includes: When an anomaly is detected in the business status, obtain the abnormal business data corresponding to the target business identifier that caused the anomaly; An abnormal business document object is generated based on the abnormal business data; wherein, the abnormal business document object includes at least the identifier of the target business that caused the abnormality and the event status of the timeout. The attribution matching is determined based on the abnormal business document object and the preset attribution method to obtain the matching result; wherein, the attribution method is a configurable attribution method or a large model attribution method; Based on the matching results, obtain the cause of the anomaly and the solution to the anomaly; The output includes the abnormal business document object, the cause of the abnormality, and the solution to the abnormality, as well as business abnormality alarm information.

9. A business data monitoring device, characterized in that, The business data monitoring device includes: The first acquisition unit is used to acquire a unified event stream containing change information for all stages of the business process. A partitioning unit is used to perform partitioning processing on the unified event stream based on service identifiers to obtain an event sub-stream corresponding to each service identifier; The monitoring unit is used to monitor the service status of the event sub-stream corresponding to each service identifier.

10. A method for monitoring and alerting anomalies in financial data, characterized in that, include: A unified event flow that obtains change information for all stages of financial business processes from multi-source heterogeneous financial databases; The unified event stream is partitioned based on the financial business identifier to obtain multiple event sub-streams that correspond one-to-one with the financial business identifier. For each event sub-stream, the financial business status is monitored, and when a timeout anomaly is detected in the financial business status, the abnormal business data corresponding to the target business identifier of the timeout anomaly is obtained; the financial business status includes at least acceptance, clearing, and settlement. Output timeout alarm information based on the abnormal business data.

11. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the business data monitoring method according to any one of claims 1 to 8.

12. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, performs the business data monitoring method according to any one of claims 1 to 8.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs the business data monitoring method according to any one of claims 1 to 8.

Images