A data processing method and device, electronic equipment and readable storage medium

Abstract

The application discloses a data processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of computers. The method uses different adapters to respectively acquire change event notifications of corresponding storage systems in a source storage system cluster. The source storage system cluster comprises a plurality of heterogeneous storage systems. Feature information in the change event notifications is extracted, and the change event notifications are converted into standard event notifications by using the feature information. Data flow detection is performed on the standard event notifications. In the case that the standard event notifications trigger event flow, data flow instructions are generated. The target data corresponding to the change event notifications is processed by using the data flow instructions. The application has the technical effect that the whole data flow is realized without human intervention, and the source storage system cluster with heterogeneous storage systems can be adapted, and automatic and efficient data flow can be realized.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Data storage is no longer limited to a single system; increasingly, data is distributed across heterogeneous storage systems. To realize the value of data, it often needs to flow between these heterogeneous systems. However, due to differences in interface standards, data formats, and security mechanisms, free and efficient data flow is difficult between different storage systems. Currently, cross-system data flow largely relies on manual triggering or simple scheduled tasks, resulting in poor real-time performance and low adaptability.

[0003] Therefore, how to enable data to flow between heterogeneous storage systems is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This application provides a data processing method, apparatus, electronic device, and readable storage medium that enables data flow without human intervention throughout the entire process, is adaptable to source storage system clusters with heterogeneous storage systems, and can achieve automatic and efficient data flow.

[0005] A data processing method, comprising: Using different adapters, change event notifications for the corresponding storage systems in the source storage system cluster are obtained respectively; wherein, the source storage system cluster includes multiple heterogeneous storage systems; Extract the feature information from the change event notification, and use the feature information to convert the change event notification into a standard event notification; Data flow detection is performed on the standard event notification, and a data flow instruction is generated when the standard event notification triggers event flow. The data flow instruction is used to process the target data corresponding to the change event notification.

[0006] Preferably, data flow detection is performed on the standard event notification, and if the standard event notification triggers event flow, a data flow instruction is generated, including: The standard event notification is matched and detected against triggering conditions in multiple data flow models. If the standard event notification matches the target triggering conditions, determine the triggering event flow; The data flow model corresponding to the target triggering condition is determined as the target data flow model; The data flow instructions are generated using the execution actions and target configuration of the target data flow model.

[0007] Preferably, establishing the data flow model includes: Receive configuration policy information; Extract trigger condition information, execution action information, and flow configuration information from the configuration strategy information; The data flow model is constructed using the triggering condition information, the execution action information, and the flow configuration information.

[0008] Preferably, the standard event notification is matched and detected against triggering conditions in multiple data flow models, including: Determine whether the standard event notification includes the first event notification and the second event notification in the target triggering condition; wherein the first event notification and the second event notification are different; If so, then determine the trigger event flow; If not, then it is determined that no event flow was triggered.

[0009] Preferably, the data flow instruction is used to perform flow processing on the target data corresponding to the change event notification, including: Call the read interface of the adapter corresponding to the data flow instruction; The target data is obtained using the read interface. The target data is divided into blocks, and the check value of each block is calculated. Transmit the block and the check value to the target path corresponding to the data flow instruction; After transmission is complete, the target data under the target path is verified using the verification value.

[0010] Preferably, it further includes: Detect key nodes in the data flow to trigger status events at those key nodes; The aforementioned status events provide multi-dimensional traceability services on the visual monitoring and auditing platform.

[0011] Preferably, it further includes: The status events are pushed to the distributed log storage system in real time; The status events are cached and forwarded using the event stream platform in the distributed log storage system. The distributed log storage system is used to search the database and index and store the status events.

[0012] A data processing apparatus, comprising: The access module is used to obtain change event notifications for corresponding storage systems in the source storage system cluster using different adapters; wherein, the source storage system cluster includes multiple heterogeneous storage systems; A standardization module is used to extract feature information from the change event notification and use the feature information to convert the change event notification into a standard event notification. An automation module is used to detect data flow in the standard event notification and generate a data flow instruction when the standard event notification triggers event flow. The data flow module is used to process the target data corresponding to the change event notification by using the data flow instruction.

[0013] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above data processing methods.

[0014] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0016] The data processing method provided in this application is used to obtain change event notifications of corresponding storage systems in the source storage system cluster using different adapters. The source storage system cluster includes multiple heterogeneous storage systems. Feature information is extracted from the change event notifications and converted into standard event notifications using the feature information. Data flow detection is performed on the standard event notifications. If the standard event notifications trigger event flow, a data flow instruction is generated. The data flow instruction is used to process the target data corresponding to the change event notifications.

[0017] To adapt to heterogeneous storage systems and achieve efficient and automated data flow processing, this application addresses the situation where data flow is required, as the storage systems themselves often undergo data changes, generating change event notifications. Therefore, this application utilizes adapters for each storage system to obtain change event notifications from the corresponding storage system within the source storage system cluster. This source storage system cluster comprises multiple heterogeneous storage systems. To enable a single data flow detection method to handle data flow across multiple heterogeneous storage systems, after obtaining the change event notification, feature extraction is performed on the change event characteristics, and a standard event notification is derived based on the extracted feature information. Then, data flow detection is performed on the standard event notification, and a data flow instruction is generated when the standard event notification triggers event flow. Finally, using the data flow instruction, the target data corresponding to the change event notification can be processed.

[0018] This application enables data flow without human intervention throughout the entire process, is adaptable to source storage system clusters with heterogeneous storage systems, and achieves the technical effect of automatic and efficient data flow. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A schematic diagram illustrating the implementation flow of a data processing method provided in this application embodiment; Figure 2 This is a schematic diagram of the technical architecture of a data processing method provided in an embodiment of this application; Figure 3 This application provides a schematic diagram of a specific implementation process for a data processing method. Figure 4 A schematic diagram illustrating a strategy-driven flow provided in an embodiment of this application; Figure 5 This is a schematic diagram of data traceability provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 8 This is a schematic diagram of the specific structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0022] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0023] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Please refer to Figure 1 This data processing method includes the following steps.

[0025] S101. Using different adapters, obtain change event notifications for the corresponding storage systems in the source storage system cluster.

[0026] The source storage system cluster includes multiple heterogeneous storage systems.

[0027] The source storage system cluster can include multiple storage systems, which can be heterogeneous, meaning the storage systems within the source storage system cluster can be different storage systems such as S3, Azure Blob, and local file systems.

[0028] Different storage systems have different adapters. In this embodiment, change event notifications for the corresponding storage system in the source storage system cluster can be obtained through the adapters of different storage systems. For ease of management, these adapters of different storage systems can also be grouped into an adapter layer.

[0029] At the adapter layer, each adapter receives real-time notifications of changes such as object creation, modification, deletion, and associated business events (e.g., task completion) by connecting to the native API of the corresponding storage system.

[0030] That is, the change event notification in this application embodiment is a change notification that sends a change message to the outside world when a change occurs in the corresponding storage system.

[0031] S102. Extract the feature information from the change event notification and use the feature information to convert the change event notification into a standard event notification.

[0032] Upon receiving a change event notification, considering the variations in content and format across different storage systems, a standardized expression—a standard event notification—can be converted to achieve unified detection. This standard event notification, as used in this paper, is a unified, fixed format that includes a tuple of specified dimension information, such as a four-tuple containing event type, object identifier, business association, and occurrence time. Of course, in practical applications, the required dimension information can be expanded and adjusted based on the specific trigger settings of the data flow.

[0033] Among them, the feature information can be the information required to build a standard event notification, that is, the key information in the event, such as the event type, associated object ID, timestamp, and business identifier.

[0034] Specifically, the change event notification of the corresponding storage system can be analyzed based on its specific cause and content structure to obtain characteristic information. Then, the characteristic information is filled into a standardized internal event template, thus generating a standard event notification.

[0035] In one specific embodiment of this application, standardized internal event template management includes: Receive standard dimension configuration information specified by the user; The standard dimensions are determined from the standard dimension configuration information; By combining the standard dimensions, we can obtain a variety of standardized internal event templates.

[0036] In other words, standardized internal event templates can be created based on user configurations, and the standard dimensions involved in the templates can be determined according to the actual user configurations to adapt to changes in different scenario requirements.

[0037] S103. Perform data flow detection on standard event notifications, and generate a data flow instruction if the standard event notification triggers event flow.

[0038] In this embodiment, the triggering conditions for data flow can be analyzed in advance to summarize the triggering patterns / rules. Then, standard event notifications can be detected based on these rules to determine whether time flow has been triggered. If the triggering event flow is identified, a data flow instruction can be generated.

[0039] The data flow instruction specifies which data should flow from which location to which location, how it should flow, and the timing of the flow. Specifically, the data flow instruction can include source storage location parameters, target storage write parameters, data filtering range parameters, transmission control parameters, trigger scheduling parameters, and landing verification receipt parameters, which are used to drive the continuous, directional, and verifiable normal flow of data between storage nodes.

[0040] In one specific embodiment of this application, data flow detection is performed on standard event notifications, and when the standard event notification triggers event flow, a data flow instruction is generated, including: Match and detect standard event notifications against trigger conditions in multiple data flow models; Determine the flow of the triggering event when the standard event notification matches the target triggering conditions; The data flow model corresponding to the target triggering condition is determined as the target data flow model; Data flow instructions are generated by utilizing the execution actions and target configuration of the target data flow model.

[0041] The data flow model includes a three-in-one flow strategy: triggering conditions, execution actions, and target configuration. The triggering conditions can be defined in conjunction with multiple events, such as the completion event of a data analysis task and the creation event of the result data object. The execution actions include data migration, replication, and other types. The target configuration covers parameters such as the target storage system address, authentication information, and storage path.

[0042] In practical applications, the data flow model is the rule model obtained through parsing, which is the code extracted from the customer's configured strategy.

[0043] For example, regarding a medical image archiving and collaboration strategy, the strategy description is as follows: Once a patient's image examination is completed (generating a DICOM file) and the diagnostic report has been initially approved, the images and report are automatically packaged, archived to long-term cold storage, and a copy is synchronized to a designated storage bucket at the collaborating research hospital. The corresponding data flow model's triggering condition is: the `object_created` event from the image archiving and communication system (`object_type='dicom_study'`) and the `event_type='report_reviewed'` and `review_status='approved'` event from the HIS system occur in association (associated through the same `patient_study_uid`); the execution actions are: 1) Data packaging and compression; 2) Replication to cold storage (e.g., AWS Glacier); 3) Replication to the partner's S3 storage bucket; target configuration: Target 1: AWS Glacier, Vault name: medical-archive; Target 2: Collaborating hospital S3, path: s3: / / partner-hospital / clinical-study / {date} / . Finally, this data flow model, represented as a CEP model, can be represented by the following code: select from pattern [ every ( e1=ObjectEvent(object_type='dicom_study', event_type='object_created') and e2=BizEvent(event_type='report_reviewed', review_status='approved') where e1.patient_study_uid = e2.patient_study_uid ) ].

[0044] For the content production and multi-platform distribution strategy, the strategy description is as follows: When an article in the content management system (CMS) is marked as published and its corresponding cover image is uploaded, the article text and image are automatically distributed to the website's web server and mobile CDN, respectively, and a PDF copy is generated and archived in the knowledge base. The corresponding data flow model trigger conditions are: an event from the CMS with event_type='cms_content_updated', status='published', and content_type='article', occurring in association with an object_created event from the same content_id with object_type='cover_image'; The actions performed are: 1) Publish the article HTML to the web server; 2) Upload the cover image to the CDN; 3) Call the PDF generation service and store the PDF in the knowledge base; Target configuration: Target 1: Internal web server (via SFTP). Target 2: Cloud service provider CDN (such as Alibaba Cloud OSS). Target 3: Enterprise knowledge base system (such as Confluence API or SharePoint).

[0045] In this embodiment, multiple data flow models can be established to standardize the data flow required under different circumstances. When detecting whether a standard event notification triggers event flow, the standard event notification can be matched one by one with the triggering conditions in the multiple data flow models to determine whether event flow is triggered. Specifically, when a standard event notification matches a target triggering condition, event flow is determined to be triggered. Then, the data flow model corresponding to the target triggering condition can be determined as the target data flow model. Finally, by understanding the execution actions and target configuration of this data flow model, data flow instructions can be generated.

[0046] The establishment of a data flow model includes: Receive configuration policy information; Extract trigger condition information, execution action information, and flow configuration information from configuration policy information; A data flow model is constructed using trigger condition information, execution action information, and flow configuration information.

[0047] In other words, in this embodiment, a data flow model can be constructed based on configuration policy information. That is, trigger condition information, execution action information, and flow configuration information are extracted from the configuration policy information, and then the data flow model can be constructed.

[0048] In practical applications, rule matching and action triggering can be achieved based on Complex Event Processing (CEP) technology. The data flow policy defined by the user through the policy configuration management interface is a three-in-one combined model, specifically including the following three core components: Triggering conditions: Defined by CEP (Complex Event Processing) rules. These rules describe the complex event patterns that will automatically trigger data flow. Execution action: The specific data operation that the system will perform after the aforementioned triggering conditions are met; Target configuration: Specifies the target storage system to which the data will flow and its specific location.

[0049] For example, a specific data flow model is triggered when the `task_complete` event (business_id) with `loan_risk_assessment` and the `object_created` event (object_type) with `risk_data` occur simultaneously, and these two events share the same associated ID (i.e., `loan_id`, loan number). The execution action is to select a data replication action, which can be further configured with more granular execution modes, such as full replication + incremental synchronization, i.e., a full replication on the first attempt, followed by synchronization of only changed data. The target is configured as "Enterprise Public Cloud S3 Storage", with the access address being `s3.amazonaws.com / loan-risk-bucket`. The target path supports variables, such as `{loan_id} / {timestamp} / `.

[0050] In other words, a complete user policy can be described as a data flow model that executes specific data actions, such as copying, to a specified target storage path when [complex event conditions defined by CEP rules] occur.

[0051] In one specific embodiment of this application, matching and detecting standard event notifications with triggering conditions in multiple data flow models includes: Determine whether the standard event notification includes the first event notification and the second event notification in the target triggering condition; wherein the first event notification and the second event notification are different; If so, then determine the trigger event flow; If not, then it is determined that no event flow was triggered.

[0052] In other words, in practical applications, time-based event triggering can also be configured with multiple event linkages. Specifically, if the received standard event notification simultaneously includes both the first event notification and the second event, the conditions for multiple event linkage are met, thus triggering event flow. Of course, in practical applications, the triggering conditions can also be set with more complex judgment logic, such as determining whether to trigger event flow based on the specific type of the subsequent event B after event A has occurred.

[0053] S104. Use data flow instructions to process the target data corresponding to the change event notification.

[0054] Once a data flow instruction is generated, it can be executed to process the target data, that is, to move the target data from one storage system to another.

[0055] The data processing method provided in this application includes: using different adapters to obtain change event notifications for corresponding storage systems in a source storage system cluster; wherein the source storage system cluster includes multiple heterogeneous storage systems; extracting feature information from the change event notifications and converting the change event notifications into standard event notifications using the feature information; performing data flow detection on the standard event notifications, and generating a data flow instruction when the standard event notifications trigger event flow; and using the data flow instruction to perform flow processing on the target data corresponding to the change event notifications.

[0056] To adapt to heterogeneous storage systems and achieve efficient and automated data flow processing, this application addresses the situation where data flow is required, as the storage systems themselves often undergo data changes, generating change event notifications. Therefore, this application utilizes adapters for each storage system to obtain change event notifications from the corresponding storage system within the source storage system cluster. This source storage system cluster comprises multiple heterogeneous storage systems. To enable a single data flow detection method to handle data flow across multiple heterogeneous storage systems, after obtaining the change event notification, feature extraction is performed on the change event characteristics, and a standard event notification is derived based on the extracted feature information. Then, data flow detection is performed on the standard event notification, and a data flow instruction is generated when the standard event notification triggers event flow. Finally, using the data flow instruction, the target data corresponding to the change event notification can be processed.

[0057] This application enables data flow without human intervention throughout the entire process, is adaptable to source storage system clusters with heterogeneous storage systems, and achieves the technical effect of automatic and efficient data flow.

[0058] It should be noted that, based on the above embodiments, the embodiments of this application also provide corresponding improvement schemes. In the preferred / improved embodiments, the same or corresponding steps as in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other; however, these will not be elaborated upon in the preferred / improved embodiments herein.

[0059] In one specific embodiment of this application, data flow instructions are used to process the target data corresponding to a change event notification, including: Call the read interface of the adapter corresponding to the data flow command; Use the read interface to obtain the target data; The target data is divided into blocks, and the check value of each block is calculated. Transmit the data blocks and checksums to the target path corresponding to the data flow instructions; After transmission is complete, the target data under the target path is verified using the checksum.

[0060] In this embodiment, during the execution of data flow instructions to achieve data flow, the target data can also be transmitted in chunks and verified during transmission. That is, during data transmission, a chunked transmission and verification mechanism is employed to support breakpoint resumption and ensure data integrity.

[0061] Among them, resuming interrupted downloads depends on the storage protocols of each vendor. For example, using the S3 protocol, resuming interrupted downloads can be achieved in the S3 adapter at the access layer.

[0062] In one specific embodiment of this application, the status of each node in the data flow can also be monitored to achieve status traceability. The specific implementation process includes: Detect key nodes in the data flow to trigger status events at those key nodes; By leveraging status events, multi-dimensional traceability services are provided on the visual monitoring and auditing platform.

[0063] In this embodiment, an event-driven state tracking system can be established to achieve end-to-end state management of data flow. State events are triggered at key nodes in the data flow (transmission start, transmission in progress, transmission completion, data verification, target confirmation, transmission failure), and each state event carries complete metadata (including task ID, source address, target address, data volume, timestamp, status code, and any exception information). A visual monitoring and auditing platform provides multi-dimensional traceability capabilities, supporting queries by task ID, time range, data source, and other conditions. State information is displayed in dashboards, trend charts, and end-to-end flowcharts, enabling operational monitoring and audit traceability.

[0064] In one specific embodiment of this application, it further includes: Push status events to the distributed log storage system in real time; The event stream platform in the distributed log storage system is used to cache and forward status events; The database is searched using a distributed log storage system, and the status events are indexed and stored.

[0065] That is, status events can be pushed to a distributed log storage system (using a Kafka + Elasticsearch architecture) in real time. Kafka implements caching and forwarding of high-concurrency status events, while Elasticsearch indexes and stores the log data.

[0066] To help those skilled in the art better understand the data processing method provided in the embodiments of this application, the data processing method will be described in detail below with reference to specific application scenarios.

[0067] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the technical architecture of a data processing method according to an embodiment of this application. The technical architecture includes four parts: an access layer, a core service layer, an application service layer, and a storage layer. The access layer includes dedicated adapters for different storage systems to enable access to multi-source events; the core service layer consists of a unified event bus, a policy-driven flow engine, and a state tracking system, and is the core control module for data flow; the application service layer provides a policy configuration management interface and a visual monitoring and auditing platform to support user operation and maintenance management; and the storage layer covers various source storage clusters and target storage clusters, serving as the data storage carrier.

[0068] Please refer to Figure 3 This allows for the construction of an abstract, unified event bus, serving as a hub for the aggregation and distribution of multi-source events. First, an adapter layer is formed for different storage systems such as S3, Azure Blob, and local file systems. This layer includes various adapters, each of which receives real-time notifications of object creation, modification, deletion, and associated business events (such as task completion) by connecting to the native API of its corresponding storage system. Second, the adapter layer transmits the received heterogeneous events to an event standardization conversion module. This module extracts key information from the events (such as event type, associated object ID, timestamp, and business identifier) ​​and converts them into standardized internal events consisting of a four-tuple: event type, object identifier, business association, and occurrence time. Finally, the internal event bus distributes these standardized events in real-time to the policy-driven flow engine and state tracking system.

[0069] Please refer to Figure 4The strategy-driven flow engine, as the control core of data flow, employs Complex Event Processing (CEP) technology to achieve rule matching and action triggering. First, it provides a visual strategy definition function through the strategy configuration management interface, supporting users to configure a three-in-one flow strategy encompassing "trigger conditions - execution actions - target configuration." Trigger conditions support multi-event association definitions (e.g., "data analysis task completion event + result data object creation event"), execution actions include data migration and replication, and target configuration covers parameters such as target storage system address, authentication information, and storage path. Second, the strategy engine parses user-defined strategies into CEP rule models, receiving standardized event streams pushed by the unified event bus in real time and performing pattern matching. When an event combination satisfying CEP rules is detected, a data flow instruction is automatically generated, invoking the data transmission interface. During data transmission, a chunked transmission and verification mechanism is employed, supporting breakpoint resumption to ensure data integrity.

[0070] Please refer to Figure 5 An event-driven state tracking system is established to achieve end-to-end state management of data flow. State events are triggered at key nodes in the data flow (transmission start, transmission in progress, transmission completion, data verification, target confirmation, and transmission failure). Each state event carries complete metadata (including task ID, source address, target address, data volume, timestamp, status code, and any exception information). State events are pushed in real-time to a distributed log storage system (using a Kafka + Elasticsearch architecture). Kafka handles high-concurrency state event caching and forwarding, while Elasticsearch indexes and stores the log data. Simultaneously, a visual monitoring and auditing platform provides multi-dimensional traceability capabilities, supporting queries by task ID, time range, data source, and other conditions. State information is displayed in dashboards, trend charts, and end-to-end flowcharts, enabling operational monitoring and audit traceability.

[0071] The specific implementation steps of this invention will be described in detail below, taking the risk assessment scenario in the financial industry as an example.

[0072] Regarding system deployment and initialization, each layer of the structure can be deployed in the following manner.

[0073] Access Layer Deployment: Deploy the S3 adapter, Azure Blob adapter, and local file system adapter. The S3 adapter connects to the enterprise's public cloud S3 storage via the AWS SDK, the Azure Blob adapter connects to third-party Azure Blob storage via the Azure Storage SDK, and the local file system adapter connects to the local storage of the enterprise's core business systems via a file system listener interface. Configure the connection parameters for each adapter (such as access keys, endpoints, listener paths, etc.) and set the event receiving frequency to real-time push.

[0074] Core service layer deployment: Deploy a unified event bus, configure the standardized event format as JSON, and define the four-tuple fields as "event_type", "object_id", "business_id", and "timestamp"; deploy a policy-driven flow engine, integrate the CEP rule engine component (using Esper CEP), and configure the data transmission thread pool size to 10; deploy a state tracking system, build a 3-node Kafka cluster for event caching, a 2-node Elasticsearch cluster for log storage, and set the log retention period to 90 days.

[0075] Application service layer deployment: Deployment strategy configuration management interface and visual monitoring and auditing platform, configure the communication interface between the platform and the core service layer, and set key indicator thresholds (such as triggering an alarm when the transmission failure rate exceeds 5%).

[0076] During the policy configuration phase, administrators define data flow policies through the policy configuration management interface: Triggering conditions: Set the CEP rule to: the task_complete event (task completion event) of business_id=loan_risk_assessment (loan risk assessment business identifier) ​​+ the object_created event (object creation event) of object_type=risk_data (risk data object type), and the associated IDs (loan_id, loan number) of the two events are consistent.

[0077] Execution action: Select the data replication action and configure the replication mode as full replication + incremental synchronization.

[0078] Target configuration: The target storage system is enterprise public cloud S3 storage, the access address is configured as s3.amazonaws.com / loan-risk-bucket, the access key is securely injected through the key management system, and the target path is {loan_id} / {timestamp} / .

[0079] After saving the policy, the policy engine automatically parses it into CEP rule statements: select from pattern[every (e1=TaskEvent(business_id='loan_risk_assessment', event_type='task_complete') and e2=ObjectEvent(object_type='risk_data', event_type='object_created') where e1.loan_id = e2.loan_id)].

[0080] During the data flow execution phase: Event generation and access: The enterprise's internal core business system completes the raw data processing of a customer's (loan_id=202405001) loan risk, generates a risk assessment report (object_id=risk_202405001.pdf), the local file system adapter listens for the file creation event, generates the raw event and pushes it to the unified event bus; at the same time, the risk assessment task management system completes the assessment task calculation for this customer and pushes the task_complete event to the unified event bus through the Azure Blob adapter.

[0081] Event Standardization Conversion: The standardization conversion module of the unified event bus receives two raw events, extracts key information, and converts them into standardized events. The standardized event for task_complete is: {"event_type":"task_complete","object_id":"task_202405001","business_id":"loan_risk_assessment","timestamp":"2024-05-20T10:30:00","loan_id":"202405001"}; the standardized event for object_created is: {"event_type":"object_created","object_id":"risk_202405001.pdf","business_id":"loan_risk_assessment","timestamp":"2024-05-20T10:30:10","loan_id":"202405001"}.

[0082] CEP rule matching: The CEP module of the strategy-driven flow engine receives two standardized events in real time, matches them with preset rules, determines that the triggering conditions are met, and generates a data flow task (task ID=flow_202405001).

[0083] Data transmission execution: The flow engine calls the read interface of the local file system adapter to obtain the risk_202405001.pdf file, transmits it in 10MB blocks, and calculates the MD5 checksum of each block. Through the upload interface of the S3 adapter, the block data is uploaded sequentially to the target path s3.amazonaws.com / loan-risk-bucket / 202405001 / 20240520103010 / . During the transmission process, a transmission status event is triggered after each block upload is completed, carrying information such as the number of blocks transmitted and the number of blocks remaining.

[0084] Data verification and confirmation: After all chunks are uploaded, the streaming engine triggers data verification, comparing the MD5 value of the local file with the MD5 value of the file stored in S3. If the verification passes, a verification success event is triggered. After receiving the file upload completion notification, the S3 adapter pushes a confirmation event from the target end to the status tracking system.

[0085] Status Monitoring and Auditing: Operations personnel can query the status of the entire data flow using the visual monitoring and auditing platform by inputting task ID=flow_202405001. The platform displays the status as follows: 10:30:15 Transmission Start → 10:30:20 Transmission in Progress (3 / 5 blocks) → 10:30:30 Transmission Completed → 10:30:32 Verification Successful → 10:30:35 Target Confirmation. Simultaneously, the platform displays the transmission rate changes in a line graph and lists the transmission details of each block in tabular form. Auditors can query all data flow tasks within the time range of 2024-05-20 10:00-11:00 and export an audit report containing task details and status records.

[0086] In case of anomaly handling, if the third segment fails to transmit due to network fluctuations during the above transmission process, the streaming engine immediately triggers a transmission failure status event, recording the anomaly information as network timeout, error code: 504; simultaneously, it initiates the breakpoint resume mechanism, retrying to transmit the third segment after 5 seconds, and continuing the transmission of subsequent segments after successful transmission. Anomaly events are pushed to the monitoring platform in real time, triggering a yellow alarm. Operations personnel can locate the failed segment and the cause of the anomaly through the traceability view.

[0087] As can be seen, the data processing method provided in this application embodiment has the following technical effects: Enhancing the real-time performance and automation of data flow: A unified event bus enables real-time access and standardized transformation of multi-source events. Combined with the CEP rule engine to identify complex event patterns, it achieves automatic triggering of data flow without manual intervention. Compared to traditional scheduled task transmission, response latency is reduced from hours to seconds, meeting the needs of dynamic business scenarios. For example, in risk assessment scenarios in the financial industry, risk data can be automatically synchronized to the analysis system within one minute of generation, improving the timeliness of risk decision-making.

[0088] Enhanced adaptability and scalability of heterogeneous systems: Adopting an adapter layer architecture, dedicated adapters are developed for different storage systems. When adding a new storage system, only the corresponding adapter needs to be developed for integration, without modifying the core service logic. Testing shows that when adding Alibaba Cloud OSS storage, adapter development and deployment can be completed by only 2 people per day, improving adaptation efficiency by over 80% and solving the problem of "reconstruction with every new system addition" in traditional solutions.

[0089] Achieving end-to-end traceability and controllability of data flow: Establishing a full-link status tracking mechanism, with real-time recording and visualization of key node status, reducing fault location time from hours to minutes. Simultaneously, structured log storage supports multi-dimensional audit queries, meeting compliance requirements in industries such as finance and healthcare. For example, in medical data flow scenarios, the transmission path, processing nodes, and operators of patient medical records can be traced, complying with HIPAA compliance standards.

[0090] Ensuring data transmission reliability and integrity: By employing chunked transmission, breakpoint resumption, and MD5 verification mechanisms, the data transmission success rate is increased from 92% in traditional solutions to 99.9%. Even in cross-regional transmission scenarios with unstable networks, it ensures that data is not lost or tampered with, providing reliable protection for cross-system data sharing.

[0091] The embodiments of this application also provide a data processing apparatus. For a description of the features in the embodiments corresponding to the data processing apparatus, please refer to the relevant descriptions in the embodiments corresponding to the data processing method, which will not be repeated here.

[0092] Please refer to Figure 6 The device includes: Access module 101 is used to obtain change event notifications of corresponding storage systems in the source storage system cluster using different adapters; wherein, the source storage system cluster includes multiple heterogeneous storage systems; Standardization module 102 is used to extract feature information from change event notifications and use the feature information to convert change event notifications into standard event notifications. Automation module 103 is used to detect data flow in standard event notifications and generate data flow instructions when the standard event notification triggers event flow. The data flow module 104 is used to process the target data corresponding to the change event notification by using data flow instructions.

[0093] Using the apparatus provided in the embodiments of this application, the apparatus utilizes different adapters to obtain change event notifications for corresponding storage systems in a source storage system cluster; wherein, the source storage system cluster includes multiple heterogeneous storage systems; extracts feature information from the change event notifications, and uses the feature information to convert the change event notifications into standard event notifications; performs data flow detection on the standard event notifications, and generates a data flow instruction when the standard event notifications trigger event flow; and uses the data flow instruction to perform flow processing on the target data corresponding to the change event notifications.

[0094] To adapt to heterogeneous storage systems and achieve efficient and automated data flow processing, this application addresses the situation where data flow is required, as the storage systems themselves often undergo data changes, generating change event notifications. Therefore, this application utilizes adapters for each storage system to obtain change event notifications from the corresponding storage system within the source storage system cluster. This source storage system cluster comprises multiple heterogeneous storage systems. To enable a single data flow detection method to handle data flow across multiple heterogeneous storage systems, after obtaining the change event notification, feature extraction is performed on the change event characteristics, and a standard event notification is derived based on the extracted feature information. Then, data flow detection is performed on the standard event notification, and a data flow instruction is generated when the standard event notification triggers event flow. Finally, using the data flow instruction, the target data corresponding to the change event notification can be processed.

[0095] This application enables data flow without human intervention throughout the entire process, is adaptable to source storage system clusters with heterogeneous storage systems, and achieves the technical effect of automatic and efficient data flow.

[0096] In one specific embodiment of this application, an automation module is specifically used to match and detect standard event notifications with triggering conditions in multiple data flow models; Determine the flow of the triggering event when the standard event notification matches the target triggering conditions; The data flow model corresponding to the target triggering condition is determined as the target data flow model; Data flow instructions are generated by utilizing the execution actions and target configuration of the target data flow model.

[0097] In one specific embodiment of this application, the model management module is used to establish a data flow model, including: Receive configuration policy information; Extract trigger condition information, execution action information, and flow configuration information from configuration policy information; A data flow model is constructed using trigger condition information, execution action information, and flow configuration information.

[0098] In one specific embodiment of this application, the automation module is specifically used to determine whether the standard event notification includes a first event notification and a second event notification in the target triggering condition; wherein the first event notification and the second event notification are different; If so, then determine the trigger event flow; If not, then it is determined that no event flow was triggered.

[0099] In one specific embodiment of this application, the data flow module is specifically used to call the read interface of the adapter corresponding to the data flow instruction; Use the read interface to obtain the target data; The target data is divided into blocks, and the check value of each block is calculated. Transmit the data blocks and checksums to the target path corresponding to the data flow instructions; After transmission is complete, the target data under the target path is verified using the checksum.

[0100] In one specific embodiment of this application, it further includes: The status tracing module is used to detect key nodes in the data flow in order to trigger status events at those key nodes. By leveraging status events, multi-dimensional traceability services are provided on the visual monitoring and auditing platform.

[0101] In one specific embodiment of this application, the state tracing module is also used to push state events to the distributed log storage system in real time; The event stream platform in the distributed log storage system is used to cache and forward status events; The database is searched using a distributed log storage system, and the status events are indexed and stored.

[0102] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the data processing method described above can be referred to each other.

[0103] See Figure 7 As shown, the electronic device includes: Memory 332 is used to store computer programs; The processor 322 is used to implement the steps of the data processing method in the above method embodiments when executing a computer program.

[0104] For details, please refer to Figure 8 , Figure 8 This is a schematic diagram of the specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0105] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0106] The steps in the data processing method described above can be implemented by the structure of an electronic device.

[0107] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below corresponds to the data processing method described above. This application also provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments at runtime.

[0108] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0109] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0110] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0111] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0112] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only intended to help understand the methods and core ideas of this application. It should be noted that those skilled in the art can make several improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Using different adapters, change event notifications for the corresponding storage systems in the source storage system cluster are obtained respectively; wherein, the source storage system cluster includes multiple heterogeneous storage systems; Extract the feature information from the change event notification, and use the feature information to convert the change event notification into a standard event notification; Data flow detection is performed on the standard event notification, and a data flow instruction is generated when the standard event notification triggers event flow. The data flow instruction is used to process the target data corresponding to the change event notification.

2. The method according to claim 1, characterized in that, Data flow detection is performed on the standard event notification. If the standard event notification triggers event flow, a data flow instruction is generated, including: The standard event notification is matched and detected against triggering conditions in multiple data flow models. If the standard event notification matches the target triggering conditions, determine the triggering event flow; The data flow model corresponding to the target triggering condition is determined as the target data flow model; The data flow instructions are generated using the execution actions and target configuration of the target data flow model.

3. The method according to claim 2, characterized in that, Establishing the data flow model includes: Receive configuration policy information; Extract trigger condition information, execution action information, and flow configuration information from the configuration strategy information; The data flow model is constructed using the triggering condition information, the execution action information, and the flow configuration information.

4. The method according to claim 2, characterized in that, Matching the standard event notification with triggering conditions in multiple data flow models includes: Determine whether the standard event notification includes the first event notification and the second event notification in the target triggering condition; wherein the first event notification and the second event notification are different; If so, then determine the trigger event flow; If not, then it is determined that no event flow was triggered.

5. The method according to claim 1, characterized in that, Using the data flow instruction, the target data corresponding to the change event notification is processed by flow, including: Call the read interface of the adapter corresponding to the data flow instruction; The target data is obtained using the read interface. The target data is divided into blocks, and the check value of each block is calculated. Transmit the block and the check value to the target path corresponding to the data flow instruction; After transmission is complete, the target data under the target path is verified using the verification value.

6. The method according to any one of claims 1 to 5, characterized in that, Also includes: Detect key nodes in the data flow to trigger status events at those key nodes; The aforementioned status events provide multi-dimensional traceability services on the visual monitoring and auditing platform.

7. The method according to claim 6, characterized in that, Also includes: The status events are pushed to the distributed log storage system in real time; The status events are cached and forwarded using the event stream platform in the distributed log storage system. The distributed log storage system is used to search the database and index and store the status events.

8. A data processing apparatus, characterized in that, include: The access module is used to obtain change event notifications for corresponding storage systems in the source storage system cluster using different adapters; wherein, the source storage system cluster includes multiple heterogeneous storage systems; A standardization module is used to extract feature information from the change event notification and use the feature information to convert the change event notification into a standard event notification. An automation module is used to detect data flow in the standard event notification and generate a data flow instruction when the standard event notification triggers event flow. The data flow module is used to process the target data corresponding to the change event notification by using the data flow instruction.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 7.

Images