Adaptive data scheduling method and device based on event driving, equipment and medium

Abstract

The invention relates to the field of artificial intelligence, can be applied to business system platforms of finance, medical health and the like, and discloses an event-driven adaptive data scheduling method, device and equipment and a medium. When a change event of a specified type is captured, metadata before and after a source system is changed are analyzed, a corresponding dynamic mapping rule is generated according to a current mapping strategy, and a global logic view is updated; when a query instruction is received, the scheduler based on reinforcement learning generates an optimal scheduling execution plan according to the current system state information under the guidance of the current optimal execution strategy; executing an optimal scheduling execution plan based on the updated global logic view and collecting execution performance data; and updating the current mapping strategy and the current optimal execution strategy for the next round of source system event change processing. The data flow change is automatically sensed, adaptive data scheduling optimization is carried out, and the robustness of data federal scheduling is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an event-driven adaptive data scheduling method, apparatus, device, and medium. Background Technology

[0002] Zero-ETL architecture is a cloud-native data integration method that aims to achieve direct data processing and analysis by eliminating data movement and transformation steps in the traditional ETL (extract, transform, load) process, thereby improving data processing efficiency and thus having wide applications in many fields.

[0003] For example, in the healthcare industry, the Zero-ETL architecture supports the efficient integration and analysis of patient data, medical records, and operational information. By directly synchronizing electronic medical record systems or hospital management data to the analysis platform, medical institutions can quickly query and generate real-time reports. Similarly, in the financial industry, there are many data analysis needs, such as risk management, market risk assessment, and operational risk assessment. The Zero-ETL architecture can help financial institutions unify multi-source data (such as user behavior and market data) for data analysis, thereby reducing redundant ETL tasks, optimizing costs, and accelerating report generation.

[0004] However, in the current multi-source heterogeneous production environment, the existing Zero-ETL architecture usually relies on predefined schema mapping. Once the source changes, the virtual view will become invalid or data will be lost. It is also difficult to adapt to complex cross-source query scheduling. Therefore, how to achieve automatic change detection and adaptive optimization of data federation scheduling is still an urgent problem to be solved. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of this invention is to provide an event-driven adaptive data scheduling method, apparatus, device and medium that can be applied to the financial field, medical field or other related fields. Its main purpose is to realize automatic perception of data flow changes and adaptive data scheduling optimization, thereby improving the robustness of data federation scheduling in multi-source heterogeneous scenarios.

[0006] The technical solution of the present invention is as follows: The first aspect of this invention provides an event-driven adaptive data scheduling method, comprising: Listen for change events in the source system and publish the corresponding pending events to the internal event bus when a change event of a specified type is captured. When a pending event is received from the internal event bus, the metadata of the source system before and after the change is analyzed, and corresponding dynamic mapping rules are generated according to the current mapping strategy and the global logical view is updated. When a query command is received, the reinforcement learning-based scheduler generates the optimal scheduling execution plan based on the current system state information, guided by the current optimal execution strategy. Execute the optimal scheduling plan based on the updated global logical view and collect execution performance data; The current mapping strategy and the current optimal execution strategy are updated based on the execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing.

[0007] A second aspect of the present invention provides an event-driven adaptive data scheduling device, comprising: The event listening module is used to listen for change events in the source system and publish the corresponding pending events to the internal event bus when a change event of a specified type is captured. The dynamic mapping module is used to analyze the metadata of the source system before and after the change when it receives the pending event pushed by the internal event bus, generate the corresponding dynamic mapping rules according to the current mapping strategy and update the global logical view. The scheduling plan generation module is used to generate the optimal scheduling execution plan based on the current system state information, under the guidance of the current optimal execution strategy, when a query command is received. The scheduling and execution module is used to execute the optimal scheduling and execution plan based on the updated global logical view and collect execution performance data. The strategy update module is used to update the current mapping strategy and the current optimal execution strategy based on the execution performance data, and obtain a new mapping strategy and execution strategy for the next round of source system event change processing.

[0008] A third aspect of the present invention provides a computer device including at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the event-driven adaptive data scheduling method described above.

[0009] A fourth aspect of the present invention provides a non-volatile computer-readable storage medium storing computer-executable instructions, which, when executed by one or more processors, cause the one or more processors to perform the above-described event-driven adaptive data scheduling method.

[0010] Beneficial Effects: This invention discloses an event-driven adaptive data scheduling method, apparatus, device, and medium. Compared to existing technologies, this invention listens for change events in the source system and publishes corresponding pending events to the internal event bus when a specified type of change event is captured. Upon receiving the pending events pushed by the internal event bus, the metadata of the source system before and after the change is analyzed, and corresponding dynamic mapping rules are generated according to the current mapping strategy, and the global logical view is updated. When a query command is received, a reinforcement learning-based scheduler generates an optimal scheduling execution plan based on the current optimal execution strategy and the current system state information. The optimal scheduling execution plan is executed based on the updated global logical view, and execution performance data is collected. The current mapping strategy and the current optimal execution strategy are updated based on the execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing. By listening for change events to generate dynamic mapping rules to adaptively update the global logical view, and combining a reinforcement learning-based scheduler to continuously optimize the scheduling execution strategy, this invention achieves automatic perception of data flow changes and adaptive data scheduling optimization, improving the robustness of data federation scheduling. Attached Figure Description

[0011] To more clearly illustrate the solutions in this invention, the accompanying drawings used in the description of the embodiments of this invention will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0012] Figure 1 A schematic diagram of an application environment for the event-driven adaptive data scheduling method provided in an embodiment of the present invention; Figure 2 A flowchart of an event-driven adaptive data scheduling method provided in an embodiment of the present invention; Figure 3 A schematic diagram of the functional modules of the event-driven adaptive data scheduling device provided in an embodiment of the present invention; Figure 4 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments of the invention are described below in conjunction with the accompanying drawings.

[0014] The event-driven adaptive data scheduling method provided in this invention can be applied to, for example... Figure 1 In the application environment, it includes a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0015] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (for example only).

[0016] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0017] Server 105 can be a server providing various services, such as a backend server supporting the content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. Server 105 can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"), such as high management difficulty and weak business scalability. Server 105 can also be a server for a distributed system or a server combined with blockchain.

[0018] It should be noted that the event-driven adaptive data scheduling method provided in this application embodiment can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the event-driven adaptive data scheduling device provided in this embodiment can also be located in the first terminal device 101, the second terminal device 102, or the third terminal device 103. Alternatively, the event-driven adaptive data scheduling method provided in this embodiment can generally be executed by the server 105. Correspondingly, the event-driven adaptive data scheduling device provided in this embodiment can generally be located in the server 105.

[0019] It should be understood that the number of terminal devices, networks, and servers listed above is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be used.

[0020] like Figure 2 As shown, the event-driven adaptive data scheduling method provided in this embodiment of the invention specifically includes the following steps: S201. Listen for change events in the source system and publish the corresponding pending events to the internal event bus when a change event of a specified type is captured.

[0021] In this embodiment, the source system includes various data sources such as relational databases, non-relational databases, message queues (e.g., Kafka, RabbitMQ), and API service interfaces. An event capture module monitors the operational status of the source system in real time, listening for change events. When a specified type of change event is detected, the event to be processed is published through an internal event bus (e.g., a Kafka-based distributed event bus), ensuring that event information can be quickly subscribed to and processed by subsequent modules. Specific event types can include structured events (e.g., CREATE TABLE, ALTER TABLEADD COLUMN, JSON Schema Evolution); data events (e.g., a change in the amount of data in a source table); and operational events (e.g., a new data source being registered and launched, or a source system entering maintenance mode). By achieving real-time and comprehensive awareness of changes across multiple source systems, all change events are reported without omission or delay, providing a prerequisite for subsequent adaptive processing to flexibly adapt to changes in data flow.

[0022] For example, an e-commerce platform's data federation system connects to a MySQL product database, a Kafka user behavior log queue, and a MongoDB user review database. The development team adds a field `environmental_score` (environmental score) to the `products` table in the product database. At this point, the event capture module, through MySQL's Binlog listening function, captures the `ALTER TABLE products ADD COLUMN environmental_scoreFLOAT` operation executed on the `products` table in the product database. This event is a structured event and meets the specified type requirements. The event capture module then extracts information such as the event timestamp "2024-08-15 14:30:22", the source system identifier "MySQL_Product_DB", the event type "STRUCTURAL_CHANGE", and the change content "added field environmental_score, data type FLOAT", encapsulates it into a JSON-formatted event to be processed, and publishes it to the internal event bus.

[0023] S202. When receiving the pending event pushed by the internal event bus, analyze the metadata of the source system before and after the change, generate the corresponding dynamic mapping rules according to the current mapping strategy, and update the global logical view.

[0024] In this embodiment, a metadata self-learning engine deployed in the data federation engine server adaptively updates the global logical view based on change events to achieve automatic data adaptation that detects changes. Specifically, when the metadata self-learning engine receives a pending event pushed by the internal event bus, it obtains and analyzes the metadata of the source system before and after the change to confirm the data flow changes brought about by the current pending event. It then calls the latest optimized mapping strategy in the strategy knowledge base as the current mapping strategy, which includes field type matching rules, semantic association mapping rules, and view update priority rules, such as preset rules like "integer type fields are preferentially mapped to numeric attributes in the global view" and "field names containing the semantics of 'age' are associated with the 'age' attribute." The current mapping strategy generates corresponding dynamic mapping rules from the analysis results of data flow changes. These dynamic mapping rules are used to map changes in the data flow to the global logical view. The global logical view is a unified data access interface provided by the data federation system, containing the logical relationships and unified field descriptions of data from various source systems. Therefore, after the dynamic mapping rules are generated, the global logical view is updated accordingly. For example, newly added fields are mapped to the corresponding logical entities based on semantic association results, and changes in field types are synchronously updated in the field type definitions in the global view. This ensures that the global logical view remains consistent with changes in the source systems, achieving dynamic schema adaptation without the need for manual configuration of mapping rules. This avoids view failure or data loss caused by changes in the source systems, reduces operation and maintenance costs, and ensures the continuity and consistency of data access.

[0025] For example, continuing with the e-commerce platform example above, after the metadata self-learning engine receives a pending event indicating the addition of a new field to the products table, it retrieves a snapshot of the table's metadata before the change (containing information such as fields id, name, price, category, etc., and their corresponding mapping relationships) from the metadata cache. Then, it connects to the MySQL product database and extracts 100 sample data entries containing the environmental_score field. Analysis reveals that this field's data type is FLOAT, the field name semantically corresponds to "environmental rating," and the current mapping strategy contains a rule that "maps product-related numeric fields to the global view 'product' entity." Therefore, a dynamic mapping rule is generated: "Map MySQL_Product_DB.products.environmental_score (FLOAT) to the global logical view 'product.environmental rating' (numeric)." The metadata self-learning engine then updates the global logical view according to this dynamic mapping rule, adding the "environmental rating" attribute under the "product" logical entity, completing view synchronization. Without the administrator's knowledge, downstream data analysts can then directly use statements like SELECT * FROM product WHERE environmental rating > 4.5 in federated queries, thus seamlessly adapting to this change event.

[0026] S203. When a query command is received, the reinforcement learning-based scheduler generates the optimal scheduling execution plan based on the current system state information, guided by the current optimal execution strategy.

[0027] In this embodiment, when a user initiates a query command through SQL query statements, API calls, or other means, the query command is processed by a scheduler deployed in the scheduling execution server. This scheduler is based on reinforcement learning. Specifically, when the scheduler performs reinforcement learning, it uses the current state of the system (such as the response latency of each data source, the number of available connections, the complexity of the query, and the amount of data) as the state space; it uses possible execution actions (such as whether to access data source A or B first, whether to use broadcast JOIN or redistributed JOIN, and what the concurrency level should be) as the action space; and it uses negative values ​​of indicators such as query response time and resource consumption as rewards. By continuously collecting feedback on query execution, the scheduler continuously learns which execution actions should be taken in which system state to obtain the maximum long-term cumulative reward, and thus finds the corresponding optimal execution strategy. For example, when a data source becomes slow at night, the scheduler, which has obtained the optimal execution strategy through reinforcement learning, will automatically learn to route queries involving that source to its replicas at night, or adjust the execution order to avoid bottlenecks, thus realizing the self-evolution of query scheduling.

[0028] Therefore, when a query command is received, the reinforcement learning-based scheduler can, under the guidance of the current optimal execution strategy, select the optimal action combination from the action space based on the current system state information corresponding to the query command, forming a scheduling execution plan that includes data source access order, data reading method, related operation execution steps, number of concurrent threads, etc., thereby realizing dynamic optimization of the scheduling strategy, flexibly adjusting the execution plan according to the real-time system state, avoiding performance bottlenecks caused by fixed execution plans, and improving the response speed and success rate of complex queries.

[0029] For example, a client initiates a query command `SELECT * FROM product WHERE environmental rating > 4.5`. After parsing, the data federation engine server determines that the associated data source is MySQL_Product_DB. The scheduler collects the real-time status of this data source as follows: response latency 15ms, available connections 20, current load rate 35%, network bandwidth utilization 40%, and the query involves approximately 50,000 data entries. Based on the current optimal execution strategy ("simple queries on a single data source use a high-concurrency read mode, with concurrency set to 8"), the scheduler generates a scheduling execution plan: "1. Establish 8 concurrent connections to MySQL_Product_DB; 2. Filter data according to the condition 'environmental rating > 4.5', and read the product data that meets the condition in batches; 3. Aggregate and integrate the read data; 4. Return the integrated query results to the client."

[0030] S204. Execute the optimal scheduling execution plan based on the updated global logical view and collect execution performance data.

[0031] In this embodiment, after the scheduler generates the optimal scheduling execution plan, it executes the optimal plan based on the updated global logical view. Specifically, based on the mapping relationship between the global logical view and the source system, the physical storage location of each query field in the optimal scheduling execution plan (e.g., the MySQL_Product_DB.products.environmental_score field corresponding to "Product.Environmental Score"), data reading interface (e.g., JDBC interface, API address), and data format conversion rules are determined. Subsequently, the corresponding data query operations are executed sequentially according to the steps of the execution plan. These operations include connecting to the source system, obtaining raw data according to preset concurrency and data reading methods, performing data processing operations such as filtering, association, and aggregation, encapsulating the processed data in the unified format of the global logical view, generating query results, and returning them to the client device. This ensures that the query execution is consistent with the changed view of the source system, guaranteeing the accuracy and completeness of the data query results.

[0032] Meanwhile, the system collects performance data in real time during execution. The collected performance metrics can include query response time (total time from receiving the query command to returning the result), data read latency (time from initiating a data request to receiving the first piece of data), resource utilization (CPU utilization, memory utilization, network bandwidth utilization), concurrent connection success rate, data processing throughput, query execution success rate, etc. This provides objective and comprehensive feedback for subsequent strategy updates, enabling the system to automatically sense data changes and perform scheduling self-evolution.

[0033] For example, based on the updated global logical view, the physical location of the "Environmental Score" field corresponding to MySQL_Product_DB.products.environmental_score is parsed, and eight concurrent connections to this data source are established via JDBC interface. Data meeting the condition "Environmental Score > 4.5" is read in batches according to the execution plan, totaling 3200 product data entries. After aggregation and integration, the data is returned to the client. Simultaneously, performance data is recorded in real time, such as query response time 80ms, data read latency 18ms, CPU utilization 45%, memory usage 30%, network bandwidth usage 55%, concurrent connection success rate 100%, data processing throughput 40 records / ms, and query execution success rate 100%. This data is then tagged and stored in a performance data warehouse.

[0034] S205. Update the current mapping strategy and the current optimal execution strategy based on the execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing.

[0035] In this embodiment, a global feedback loop feeds the collected execution performance data back to the metadata self-learning engine and the scheduling optimization module. The metadata self-learning engine uses this performance feedback data to verify and correct its automatically generated mapping rules (for example, if a query based on a new mapping fails frequently, the rule needs to be adjusted), thereby updating the current mapping strategy. The scheduling optimization module uses this data for reinforcement learning to update the optimal execution strategy, thus obtaining a new mapping strategy and execution strategy that overwrites the original current mapping strategy and the current optimal execution strategy. This serves as the default strategy for the next round of source system event change processing, achieving continuous self-optimization of the strategy. By constructing a self-evolving closed loop and continuously optimizing the mapping and scheduling strategies based on performance feedback, the system dynamically evolves with business scenarios and the data ecosystem, continuously improving the reliability, efficiency, and adaptability of data access.

[0036] In one embodiment, step S201 includes: Change events of different types of source systems are monitored through a pre-configured monitoring mechanism; When a change event is captured, it is confirmed whether the event type is the specified type. If so, the captured change event is standardized to obtain the corresponding event to be processed. The event to be processed is published to the internal event bus.

[0037] In this embodiment, differentiated listening mechanisms are pre-configured for different types of source systems to ensure comprehensive capture of change events. Specifically, for database-type source systems (such as MySQL and Oracle), a Binlog listening mechanism is used. This involves subscribing to the database's Binlog logs in real time, parsing DDL (Data Definition Language) and DML (Data Manipulation Language) operations in the logs to obtain corresponding events. For message queue-type source systems (such as Kafka and RabbitMQ), a topic subscription listening mechanism is used. This involves subscribing to the corresponding message topics of each source system to receive message data in real time to capture events such as newly added fields and changes in field types. For API interface-type source systems, a timed polling and status verification listening mechanism is used. This involves configuring the polling period and requesting API interface calls to obtain the status code, response header, and response body returned by the interface, comparing them with historical data, and thus capturing change events.

[0038] Furthermore, after capturing a change event, it is confirmed whether the event type is a specified type. This is done by extracting the core characteristic information of the change event, including the event triggering operation type, source system type, and event impact scope, and then matching it with a preset list of specified event types. This list includes structural events, data events, and operational events. If the captured core characteristics of the event match any type in the specified event type list, it is determined to be a specified event type; otherwise, it is determined to be a non-specified event type and is ignored without processing.

[0039] For change events determined to be of a specified type, standardized processing is performed. The event information is encapsulated according to a preset event data structure (including fields such as event ID, event type, source system identifier, event occurrence timestamp, event details, and scope of impact) to obtain the corresponding event to be processed, ensuring the integrity and consistency of event information. A differentiated monitoring mechanism covers multiple types of source systems, such as databases, message queues, and API interfaces. Combined with event type validation and standardized processing, the comprehensiveness and accuracy of change event capture are ensured.

[0040] After standardizing and encapsulating the events to be processed, they are published to the internal event bus. This internal event bus can be a Kafka-based distributed event bus, supporting high-concurrency event transmission and message delivery. A reliable event bus delivery mechanism ensures no event loss or duplication, allowing event subscribers to obtain events in real time, providing data input for subsequent metadata analysis and mapping rule generation.

[0041] In one embodiment, step S202 includes: Subscribe to the internal event bus and extract sample data from the modified source system when receiving pending events pushed by the internal event bus; The sample data is compared with the metadata snapshot of the source system before the change to identify the differences in the fields; Based on the current mapping strategy, the difference fields are semantically associated and matched with the global logical view to generate corresponding dynamic mapping rules; The difference fields are mapped to the global logical view according to the dynamic mapping rules, and the global logical view is updated.

[0042] In this embodiment, the metadata self-learning engine subscribes to change event topics on the internal event bus to receive pending events pushed by the internal event bus in a timely manner. Upon receiving a pending event, it determines the connection parameters of the source system to be accessed based on the source system identifier, event type, and scope of impact information in the pending event. It then connects to the changed source system via JDBC, API calls, or other methods, and extracts a small amount of sample data. The specific sample size is configurable; for example, it extracts 100 data entries by default to ensure coverage of valid data for newly added or changed fields.

[0043] Simultaneously, a metadata snapshot of the source system prior to the change is obtained. Specifically, the most recent metadata snapshot of the source system can be extracted from the metadata cache. This snapshot includes information such as table structure, field names, data types, field semantic descriptions, and mapping relationships with the global logical view. The extracted sample data is then compared and analyzed with the metadata snapshot prior to the change. This includes comparing the field name list to identify newly added fields, comparing field attributes and semantic descriptions, etc., to confirm the corresponding differing fields (such as newly added fields, changes to field attributes, modifications to field names or semantics, etc.). By comparing the extracted sample data with the metadata snapshot, the differing fields from the source system change can be accurately identified.

[0044] Then, based on the current mapping strategy, the differing fields are semantically associated and matched with the global logical view. The current mapping strategy is the latest optimized mapping strategy, such as the one optimized and saved based on feedback from the previous query. It includes field type matching rules, semantic association mapping rules, and view update priority rules. The mapping strategy confirms the change rules for mapping the currently changed fields to the global logical view, thereby integrating and generating corresponding dynamic mapping rules. Based on these dynamic mapping rules, the differing fields are mapped to the global logical view, and the change events from the source system are reflected accordingly in the global logical view. For example, for the dynamic mapping rule of adding a new field event, a target attribute is added under the corresponding entity in the global logical view, and a mapping record between the source field and this attribute is added in the mapping relationship layer. For the dynamic mapping rule of changing attribute fields, the type, length, and other attribute information of the target attribute are updated accordingly, thereby dynamically updating the global logical view. This achieves schema-adaptive dynamic mapping, ensuring seamless downstream data access.

[0045] In one embodiment, the step of semantically associating and matching the difference field with the global logical view according to the current mapping strategy to generate corresponding dynamic mapping rules includes: Perform type analysis on the difference fields to confirm their data types; The difference fields are semantically matched with the global logical view to obtain the corresponding semantic association results; Based on the data type of the difference field and the semantic association result, a corresponding dynamic mapping rule is generated according to the current mapping strategy.

[0046] In this embodiment, when generating dynamic mapping rules, the difference fields are analyzed by combining the source system type and sample data characteristics to accurately determine the data type of the difference fields. For example, the corresponding field type metadata can be read from the metadata snapshot, and the actual values ​​in the sample data can be extracted for verification to ensure accurate type determination and confirm the data type of the difference fields, including integer, character, numeric, boolean, date, etc., providing a foundation for subsequent rule generation.

[0047] Furthermore, the system performs semantic association matching between the difference fields and the global logical view. The global logical view contains preset business entities and associated attribute sets. The attribute sets store attribute information for each entity, such as attribute name, semantic description, synonym set, and business scenario description. During semantic association matching, dictionary matching and semantic similarity methods can be used. For example, the name of the difference field is first precisely matched with the attribute names and synonym sets in the attribute information. If a completely identical entry exists, the corresponding entity attribute is directly associated, generating a precise matching result. For difference letters not matched by dictionary matching, the difference field is compared with the semantic description and business scenario description in the attribute information to calculate similarity. Entity attributes with similarity greater than a preset similarity are selected, thus generating semantic association results. This clarifies the target entity and target attribute corresponding to the difference field, ensuring the accuracy of the association between the difference field and the global logical view.

[0048] Based on the rules contained in the current mapping strategy, the data types and semantic association results of the difference fields are matched to generate corresponding dynamic mapping rules. For example, if the difference field `user_age` is an integer type and semantically equivalent to the customer's age in the existing logical view, and the mapping strategy includes a rule that "when the field name contains the semantics of 'age', it is associated with the 'age' attribute," then after matching, a dynamic mapping rule is automatically generated to map the difference field to the age attribute of the customer entity in the global logical view. Through refined field type analysis, a dual semantic association matching mechanism, and a standardized rule generation process, the accuracy and adaptability of dynamic mapping rules are further improved, avoiding mapping errors caused by semantic ambiguity and ensuring the reliability of data mapping.

[0049] In one embodiment, step S203 includes: Receive query commands initiated by users and parse them to obtain the corresponding query requirements; Collect status data from associated data sources based on the query requirements, and use this as the current system status information; The reinforcement learning-based scheduler invokes the current optimal execution strategy, selects the action sequence under the current system state information from the preset action space, and generates the optimal scheduling execution plan.

[0050] In this embodiment, users can initiate query commands through various methods, including SQL query statements, API calls, etc. Upon receiving a query command, it is first parsed to obtain the corresponding query requirements, including the query target (such as obtaining a product list, counting user order volume, cross-source association query of user behavior and order data), the set of data sources involved, data processing logic, etc., to provide a basis for subsequent system status information collection and scheduling plan generation.

[0051] Next, based on the set of associated data sources in the query requirements, the status data of the corresponding associated data sources is collected, including the response latency, number of available connections, current load rate, query complexity, and data volume of each data source. The collected status data serves as the current system status information for this query task, providing real-time and reliable status input for subsequent scheduling decisions. When generating the optimal scheduling execution plan, the reinforcement learning-based scheduler calls the current optimal execution strategy. This current optimal execution strategy is obtained by the scheduler through reinforcement learning training using historical status data and historical execution data. For example, the scheduler used in the initial deployment after reinforcement learning has not undergone strategy optimization based on execution performance data, so the current optimal execution strategy is the optimal strategy learned. Alternatively, it can be the latest execution strategy updated based on feedback from actual query results. For example, after updating the global logical view based on event-driven processing and performing a data query, the update of the global logical view may cause a performance degradation of the previously learned optimal execution strategy. Therefore, the current mapping strategy and the current optimal execution strategy are updated based on execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing. The current optimal execution strategy called in the next round is the latest updated and stored execution strategy. Guided by the current optimal execution strategy, the scheduler uses the current system state information as input and selects the action sequence with the highest cumulative reward from a pre-defined action space. The action space includes different types of executable actions, such as access type actions (e.g., access order, access method), connection configuration actions (e.g., concurrent connections, connection timeout), data processing strategies (e.g., JOIN algorithm selection, data sharding method), and resource allocation actions (e.g., CPU core allocation, memory usage limit, data transfer bandwidth limit). For example, guided by the current optimal execution strategy, the scheduler selects the action sequence corresponding to the query requirement in the current system state as "parallel access to MySQL and MongoDB data sources, concurrent connections set to 8, and using the broadcast JOIN algorithm." Based on the selection results, the scheduler generates the optimal scheduling execution plan after forming the corresponding action sequence. The scheduling execution plan includes execution step numbers, action descriptions, parameter configurations, dependencies, and expected execution time, ensuring the reliability of the scheduling execution. This embodiment accurately extracts the core query requirements by parsing the query command, and generates the optimal scheduling execution plan by combining the current system status information with the reinforcement learning scheduler, making the scheduling execution plan more in line with the real-time system status, and further improving the targeting and efficiency of query execution.

[0052] In one embodiment, step S204 includes: Based on the updated global logical view, the optimal scheduling execution plan is parsed to confirm the data location information associated with each action in the scheduling execution plan; Based on the data location information, each action is executed sequentially to obtain the corresponding data query results and collect the execution performance data during the execution process.

[0053] In this embodiment, after the scheduler generates the optimal scheduling execution plan, the data location information in the execution plan is parsed based on the updated global logical view. That is, according to the mapping relationship between the global logical view and the source system, each action in the scheduling execution plan is parsed to determine the physical storage location and access method of the source system corresponding to each query field, thereby generating a corresponding list of data location information, which is associated with each action of the scheduling execution plan to ensure that the scheduling execution module can quickly locate the target data and use the correct access method when executing each action.

[0054] During the execution of each action in the execution plan, each action is executed sequentially according to the step number of the scheduling execution plan based on the data location information. A connection with the source system is established based on the connection parameters in the data location information. After reading data from the source system, operations such as JOIN, aggregation, and sorting are performed according to the requirements of the data processing actions to generate the final query results. The corresponding actions are then executed after parsing the data location information through a global logical view, ensuring that the target data for each execution action can be accurately traced. Preferably, a multi-threaded execution framework can be used. If parallel actions exist in the execution plan, they can be executed synchronously to improve execution efficiency.

[0055] In addition, during execution, various methods such as data point monitoring, system resource monitoring, and network monitoring are used to collect execution performance data in real time, including the execution time of each action, total query response time, data reading time, data processing time, network transmission time, CPU utilization, peak memory usage, disk I / O rate, network bandwidth utilization, connection pool utilization, query success rate, data integrity, data accuracy, number of retries, etc., to provide complete performance feedback for subsequent policy updates, so as to realize a data federation scheduling system that can automatically learn, adapt, and optimize.

[0056] In one embodiment, step S205 includes: The effectiveness of the dynamic mapping rules is evaluated based on the execution performance data, and the current mapping strategy is updated based on the effectiveness evaluation results. The execution performance data is converted into a reward feedback signal, and the current optimal execution strategy is updated based on the reward feedback signal. The updated mapping and execution strategies are stored in the strategy knowledge base and used as the current mapping and optimal execution strategies for the next round of source system event change processing.

[0057] In this embodiment, the current mapping strategy and the current optimal execution strategy are updated based on execution performance data. Specifically, the effectiveness of the dynamic mapping rules is evaluated based on the execution performance data, considering metrics such as package adaptation success rate, data retrieval accuracy, query performance impact, and stability. Each evaluation metric is assigned a corresponding weight, and a weighted sum is used to obtain the effectiveness evaluation result. If the effectiveness evaluation result is higher than a specified score, the current dynamic mapping rule is deemed effective, and the relevant strategy details are retained and optimized. Otherwise, the current dynamic mapping rule is deemed unqualified, and the reasons for failure (such as semantic association errors or type conversion incompatibility) are analyzed before adjusting the mapping rule generation logic to update the current mapping strategy. This multi-dimensional scoring mechanism enables scientific effectiveness evaluation of mapping rules, allowing for targeted optimization or adjustment of rules and improving the reliability of the mapping strategy.

[0058] For optimizing the execution strategy, the execution performance data is converted into a reward feedback signal for reinforcement learning. Specifically, the reward value can be calculated using a reward function. For example, if the query response time is less than or equal to a preset optimal threshold, resource utilization is less than or equal to a preset upper limit, and the query success rate is 100%, a positive reward (e.g., +5) is given; if the query response time exceeds the preset threshold but is less than or equal to 150% of the threshold, and resource utilization is normal, a neutral reward (e.g., 0) is given; if the query response time exceeds 150% of the threshold or the query fails, a negative reward (e.g., -3 to -10, graded according to severity) is given. After receiving the reward feedback signal, the scheduler combines the current system state and execution actions, updates the reinforcement learning model parameters using the policy gradient descent algorithm, optimizes the action selection logic, generates a new optimal execution strategy, and achieves precise optimization of the execution strategy.

[0059] The updated mapping and execution strategies are then stored in the strategy knowledge base. The strategy knowledge base also triggers a synchronization mechanism to synchronize the updated mapping and execution strategies to the local cache of the metadata self-learning engine and the scheduler. This ensures that when the source system event changes in the next round, the metadata self-learning engine and the scheduler can quickly obtain the latest current mapping strategy and the current optimal execution strategy. This ensures that the updated strategy takes effect safely and efficiently, improves the closed-loop mechanism of system self-evolution, and guarantees the continuity and stability of strategy optimization.

[0060] It should be noted that there is no necessary order between the above steps. Those skilled in the art will understand from the description of the embodiments of the present invention that the above steps may have different execution orders in different embodiments, that is, they may be executed in parallel or in turn, etc.

[0061] Further reference Figure 3 As a response to the above Figure 2 The present invention provides an embodiment of an event-driven adaptive data scheduling device, which, in accordance with the implementation of the method shown, provides an embodiment of such a device. Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0062] like Figure 3 As shown, the event-driven adaptive data scheduling device 30 described in this embodiment includes: The event listening module 301 is used to listen for change events in the source system and publish the corresponding pending events to the internal event bus when a change event of a specified type is captured. The dynamic mapping module 302 is used to analyze the metadata of the source system before and after the change when it receives the pending event pushed by the internal event bus, generate the corresponding dynamic mapping rules according to the current mapping strategy and update the global logical view. The scheduling plan generation module 303 is used to generate the optimal scheduling execution plan based on the current system state information when a query command is received, under the guidance of the current optimal execution strategy. The scheduling execution module 304 is used to execute the optimal scheduling execution plan based on the updated global logical view and collect execution performance data; The strategy update module 305 is used to update the current mapping strategy and the current optimal execution strategy based on the execution performance data, and obtain a new mapping strategy and execution strategy for the next round of source system event change processing.

[0063] The module referred to in this invention is a series of computer program instruction segments that can perform specific functions. It is more suitable than a program for describing the event-driven adaptive data scheduling execution process. For specific implementation methods of each module, please refer to the corresponding method embodiments above, which will not be repeated here.

[0064] In one embodiment, the event listening module 301 includes: The listening unit is used to listen for change events in different types of source systems through a pre-configured listening mechanism; The event confirmation unit is used to confirm whether the event type is a specified type when a change event is captured. If so, the captured change event is standardized to obtain the corresponding event to be processed. An event publishing unit is used to publish the event to be processed to the internal event bus.

[0065] In one embodiment, the dynamic mapping module 302 includes: The data extraction unit is used to subscribe to the internal event bus and extract sample data from the modified source system when it receives the pending event pushed by the internal event bus. The data comparison unit is used to compare the sample data with the metadata snapshot of the source system before the change to confirm the difference fields; The rule generation unit is used to perform semantic association matching between the difference field and the global logical view according to the current mapping strategy, and generate corresponding dynamic mapping rules. The dynamic mapping unit is used to map the difference fields to the global logical view according to the dynamic mapping rules, and to update the global logical view.

[0066] In one embodiment, the rule generation unit includes: A data type analysis unit is used to perform type analysis on the difference fields to confirm the data type of the difference fields; The semantic association unit is used to perform semantic association matching between the difference field and the global logical view to obtain the corresponding semantic association result; The generation unit generates corresponding dynamic mapping rules based on the data type of the difference field and the semantic association result according to the current mapping strategy.

[0067] In one embodiment, the scheduling plan generation module 303 includes: The query receiving unit is used to receive query instructions initiated by users and parse them to obtain the corresponding query requirements; The status acquisition unit is used to collect status data from the associated data source according to the query requirements, as the current system status information. The plan generation unit is used by the reinforcement learning-based scheduler to invoke the current optimal execution strategy, select the action sequence under the current system state information from the preset action space, and generate the optimal scheduling execution plan.

[0068] In one embodiment, the scheduling execution module 304 includes: The plan parsing unit is used to parse the optimal scheduling execution plan based on the updated global logical view and confirm the data location information associated with each action in the scheduling execution plan; The planning and execution unit is used to execute each action sequentially according to the data positioning information, obtain the corresponding data query results, and collect execution performance data during the execution process.

[0069] In one embodiment, the policy update module 305 includes: The mapping strategy update unit is used to evaluate the effectiveness of the dynamic mapping rules based on the execution performance data, and update the current mapping strategy based on the effectiveness evaluation results. An execution strategy update unit is used to convert the execution performance data into a reward feedback signal and update the current optimal execution strategy according to the reward feedback signal; The policy storage unit is used to store the updated mapping policy and execution policy in the policy knowledge base, which will serve as the current mapping policy and the current optimal execution policy for the next round of source system event change processing.

[0070] In the above embodiments, this invention discloses an event-driven adaptive data scheduling device. It monitors change events in the source system and, upon capturing a change event of a specified type, publishes a corresponding pending event to the internal event bus. Upon receiving the pending event pushed by the internal event bus, it analyzes the metadata of the source system before and after the change, generates corresponding dynamic mapping rules according to the current mapping strategy, and updates the global logical view. When a query command is received, a reinforcement learning-based scheduler, guided by the current optimal execution strategy, generates an optimal scheduling execution plan based on the current system state information. It executes the optimal scheduling execution plan based on the updated global logical view and collects execution performance data. Based on the execution performance data, it updates the current mapping strategy and the current optimal execution strategy to obtain a new mapping strategy and execution strategy for the next round of source system event change processing. By monitoring change events to generate dynamic mapping rules to adaptively update the global logical view, and combining this with a reinforcement learning-based scheduler to continuously optimize the scheduling execution strategy, it achieves automatic perception of data flow changes and adaptive data scheduling optimization, improving the robustness of data federation scheduling.

[0071] Another embodiment of the present invention provides a computer device, such as... Figure 4 As shown, the computer device 40 includes: One or more processors 401 and memory 402, Figure 4 The following section uses a processor 401 as an example. The processor 401 and the memory 402 can be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0072] Processor 401 performs various control logic functions for computer device 40. It can be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), microcontroller, ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. Furthermore, processor 401 can also be any conventional processor, microprocessor, or state machine. Processor 401 can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP, and / or any other such configuration.

[0073] Memory 402, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions corresponding to the event-driven adaptive data scheduling method in the embodiments of the present invention. Processor 401 executes various functional applications and data processing of computer device 40 by running the non-volatile software programs, instructions, and units stored in memory 402, thereby implementing the event-driven adaptive data scheduling method in the above method embodiments.

[0074] The memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device 40. Furthermore, the memory 402 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 402 may optionally include memory remotely located relative to the processor 401, and these remote memories may be connected to the computer device 40 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. One or more units stored in the memory 402, when executed by one or more processors 401, perform the steps of the event-driven adaptive data scheduling method in any of the above method embodiments.

[0075] In the above embodiments, the present invention discloses a computer device that monitors change events in a source system and, upon capturing a change event of a specified type, publishes a corresponding pending event to an internal event bus. Upon receiving a pending event pushed by the internal event bus, the device analyzes the metadata of the source system before and after the change, generates corresponding dynamic mapping rules according to the current mapping strategy, and updates the global logical view. When a query command is received, a reinforcement learning-based scheduler, guided by the current optimal execution strategy, generates an optimal scheduling execution plan based on the current system state information. The optimal scheduling execution plan is executed based on the updated global logical view, and execution performance data is collected. The current mapping strategy and the current optimal execution strategy are updated based on the execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing. By monitoring change events to generate dynamic mapping rules to adaptively update the global logical view, and combining this with a reinforcement learning-based scheduler to continuously optimize the scheduling execution strategy, the device achieves automatic perception of data flow changes and adaptive data scheduling optimization, improving the robustness of data federation scheduling.

[0076] This invention provides a non-volatile computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by one or more processors, they perform the steps of the event-driven adaptive data scheduling method described in any of the above method embodiments.

[0077] In the above embodiments, the present invention discloses a non-volatile computer-readable storage medium that monitors change events in the source system and publishes corresponding pending events to the internal event bus when a specified type of change event is captured. Upon receiving the pending events pushed by the internal event bus, the metadata of the source system before and after the change is analyzed, and corresponding dynamic mapping rules are generated according to the current mapping strategy, and the global logical view is updated. When a query command is received, a reinforcement learning-based scheduler generates an optimal scheduling execution plan based on the current optimal execution strategy and the current system state information. The optimal scheduling execution plan is executed based on the updated global logical view, and execution performance data is collected. The current mapping strategy and the current optimal execution strategy are updated based on the execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing. By monitoring change events to generate dynamic mapping rules to adaptively update the global logical view, and combining this with a reinforcement learning-based scheduler to continuously optimize the scheduling execution strategy, the system achieves automatic perception of data flow changes and adaptive data scheduling optimization, improving the robustness of data federation scheduling.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0079] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0080] In summary, the event-driven adaptive data scheduling method, apparatus, device, and medium disclosed in this invention include the following steps: monitoring change events in the source system; publishing corresponding pending events to an internal event bus when a specified type of change event is captured; upon receiving a pending event pushed by the internal event bus, analyzing the metadata of the source system before and after the change, generating corresponding dynamic mapping rules according to the current mapping strategy, and updating the global logical view; upon receiving a query command, a reinforcement learning-based scheduler generates an optimal scheduling execution plan based on the current optimal execution strategy and current system state information; executing the optimal scheduling execution plan based on the updated global logical view and collecting execution performance data; updating the current mapping strategy and the current optimal execution strategy based on the execution performance data to obtain new mapping and execution strategies for the next round of source system event change processing. By monitoring change events to generate dynamic mapping rules to adaptively update the global logical view, and combining this with a reinforcement learning-based scheduler to continuously optimize the scheduling execution strategy, the method achieves automatic perception of data flow changes and adaptive data scheduling optimization, improving the robustness of data federation scheduling.

[0081] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The computer program can be stored in a non-volatile, computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, floppy disk, flash memory, optical storage, etc.

[0082] It should be noted that any software tools or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. It should be understood that the application of this invention is not limited to the examples described above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An event-driven adaptive data scheduling method, characterized in that, include: Listen for change events in the source system and publish the corresponding pending events to the internal event bus when a change event of a specified type is captured. When a pending event is received from the internal event bus, the metadata of the source system before and after the change is analyzed, and corresponding dynamic mapping rules are generated according to the current mapping strategy and the global logical view is updated. When a query command is received, the reinforcement learning-based scheduler generates the optimal scheduling execution plan based on the current system state information, guided by the current optimal execution strategy. Execute the optimal scheduling plan based on the updated global logical view and collect execution performance data; The current mapping strategy and the current optimal execution strategy are updated based on the execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing.

2. The event-driven adaptive data scheduling method according to claim 1, characterized in that, The process of listening for change events in the source system and publishing corresponding pending events to the internal event bus when a change event of a specified type is captured includes: Change events of different types of source systems are monitored through a pre-configured monitoring mechanism; When a change event is captured, it is confirmed whether the event type is the specified type. If so, the captured change event is standardized to obtain the corresponding event to be processed. The event to be processed is published to the internal event bus.

3. The event-driven adaptive data scheduling method according to claim 1, characterized in that, Upon receiving a pending event pushed by the internal event bus, the process involves analyzing the metadata of the source system before and after the change, generating corresponding dynamic mapping rules according to the current mapping strategy, and updating the global logical view, including: Subscribe to the internal event bus and extract sample data from the modified source system when receiving pending events pushed by the internal event bus; The sample data is compared with the metadata snapshot of the source system before the change to identify the differences in the fields; Based on the current mapping strategy, the difference fields are semantically associated and matched with the global logical view to generate corresponding dynamic mapping rules; The difference fields are mapped to the global logical view according to the dynamic mapping rules, and the global logical view is updated.

4. The event-driven adaptive data scheduling method according to claim 3, characterized in that, The step of semantically associating and matching the difference fields with the global logical view according to the current mapping strategy to generate corresponding dynamic mapping rules includes: Perform type analysis on the difference fields to confirm their data types; The difference fields are semantically matched with the global logical view to obtain the corresponding semantic association results; Based on the data type of the difference field and the semantic association result, a corresponding dynamic mapping rule is generated according to the current mapping strategy.

5. The event-driven adaptive data scheduling method according to claim 1, characterized in that, When a query command is received, the reinforcement learning-based scheduler, guided by the current optimal execution strategy, generates an optimal scheduling execution plan based on the current system state information, including: Receive query commands initiated by users and parse them to obtain the corresponding query requirements; Collect status data from associated data sources based on the query requirements, and use this as the current system status information; The reinforcement learning-based scheduler invokes the current optimal execution strategy, selects the action sequence under the current system state information from the preset action space, and generates the optimal scheduling execution plan.

6. The event-driven adaptive data scheduling method according to claim 1, characterized in that, The process of executing the optimal scheduling plan based on the updated global logical view and collecting execution performance data includes: Based on the updated global logical view, the optimal scheduling execution plan is parsed to confirm the data location information associated with each action in the scheduling execution plan; Based on the data location information, each action is executed sequentially to obtain the corresponding data query results and collect the execution performance data during the execution process.

7. The event-driven adaptive data scheduling method according to claim 1, characterized in that, The step of updating the current mapping strategy and the current optimal execution strategy based on the execution performance data to obtain a new mapping strategy and execution strategy for the next round of source system event change processing includes: The effectiveness of the dynamic mapping rules is evaluated based on the execution performance data, and the current mapping strategy is updated based on the effectiveness evaluation results. The execution performance data is converted into a reward feedback signal, and the current optimal execution strategy is updated based on the reward feedback signal. The updated mapping and execution strategies are stored in the strategy knowledge base and used as the current mapping and optimal execution strategies for the next round of source system event change processing.

8. An event-driven adaptive data scheduling device, characterized in that, include: The event listening module is used to listen for change events in the source system and publish the corresponding pending events to the internal event bus when a change event of a specified type is captured. The dynamic mapping module is used to analyze the metadata of the source system before and after the change when it receives the pending event pushed by the internal event bus, generate the corresponding dynamic mapping rules according to the current mapping strategy and update the global logical view. The scheduling plan generation module is used to generate the optimal scheduling execution plan based on the current system state information, under the guidance of the current optimal execution strategy, when a query command is received. The scheduling and execution module is used to execute the optimal scheduling and execution plan based on the updated global logical view and collect execution performance data. The strategy update module is used to update the current mapping strategy and the current optimal execution strategy based on the execution performance data, and obtain a new mapping strategy and execution strategy for the next round of source system event change processing.

9. A computer device, characterized in that, Includes at least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the event-driven adaptive data scheduling method according to any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the event-driven adaptive data scheduling method according to any one of claims 1-7.

Images