Real-time synchronous data decision-making method and system based on visual configuration

Abstract

The invention relates to a real-time synchronous data decision-making method based on visual configuration, and the method is characterized in that the method comprises the steps: deploying a data collector cluster in a target system; configuring a collection task, generating a configuration file, and pushing the configuration file to a database collector cluster; synchronizing data in real time, storing synchronization site information to an acquisition platform by using a data acquisition unit cluster, and controlling starting and pausing of an acquisition task by using a synchronization site through the acquisition platform; a subscription service is deployed in a target system to consume data on the topic, and the subscription service realizes conversion of middle-layer source data in a source database through a calling interface; the management background of the target system uses visual data configuration to automatically complete label generation on the business data according to the formulated label rule so as to complete matching of decision information, and the scheme simplifies the data synchronization process, improves the timeliness of developing corresponding and updating programs, and also improves the flexibility and availability of the data at the same time.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a decision-making method and system for real-time synchronous data based on visualization configuration. Background Art

[0002] Most existing cross-system synchronization technologies rely on fixed program logic for publishing and subscription services. For example, they use SQL Server database synchronization, direct database replication, log shipping, and trigger mechanisms. However, these solutions lack flexibility and efficiency, and lack a unified monitoring mechanism. This leads to a complex implementation process when connecting and adjusting systems, and increases the risk of errors. Summary of the Invention

[0003] To address the shortcomings of the existing technology, the present invention is based on real-time synchronization technology with visual configuration, which simplifies the process of multi-system data synchronization and reduces the workload and docking difficulty in the joint development process; avoids the development of independent publishing services for each data source end system, reduces the waste of development resources, and reduces the difficulty of system maintenance; through unified data conversion module processing, the database layer source data is converted into business data, which improves the flexibility and availability of data.

[0004] A first aspect of the present invention provides a decision-making method for real-time synchronization data based on visualization configuration, characterized by comprising:

[0005] S1 deploys a data collector cluster on the target system;

[0006] S2 configures the collection task, generates a configuration file, and pushes the configuration file to the database collector cluster;

[0007] S3 synchronizes data in real time, using a data collector cluster to save synchronization site information to the acquisition platform, which uses synchronization sites to control the start and pause of acquisition tasks;

[0008] S4 deploys a subscription service in the target system to consume data on the topic. The subscription service converts the middle-level source data in the source database by calling an interface.

[0009] The S5 management backend uses visual data configuration to automatically generate tags for the business data according to tag rules to complete the matching of decision information.

[0010] Exemplarily, the S2 configuration collection task includes:

[0011] Set the read permission of the data collector cluster in the source database;

[0012] Configure table and field requirements on the collection platform;

[0013] Use Kafka middleware to configure and apply for the Topic address and name for the collection task;

[0014] Exemplarily, the S3 real-time synchronization data uses the database to save the synchronization site information to the collection platform, including: the database collector cluster uses MySQL binlog to complete real-time synchronization of data after receiving the collection command from the collection platform.

[0015] Exemplarily, the S2 pushes the configuration file to the database collector cluster deployed in the target system, including collecting the customized tables and fields in the configuration file through interface calls.

[0016] Exemplarily, the subscription service of S4 realizes the conversion of the layer source data in the source database by calling the interface, including: converting the layer source data into the business data required by the scenario according to the calling rules, and writing the obtained business data into the cache database.

[0017] Exemplarily, the management backend of the S5 target system uses visual data configuration to automatically generate labels for the business data according to the established labeling rules, including: analyzing the business data according to different dimensions and different classification methods and generating target portraits.

[0018] A second aspect of the present invention provides a decision-making system for real-time synchronized data based on visual configuration, characterized by comprising:

[0019] Cluster deployment module, used to deploy data collector clusters;

[0020] Configuration collection module, used to configure collection tasks, generate configuration files, and push the configuration files to the database collector cluster;

[0021] The data synchronization module is used to synchronize data in real time and save the synchronization location information to the collection platform using the data collector cluster. The collection platform uses the synchronization location to control the start and pause of the collection task;

[0022] The data conversion module is used to deploy subscription services to consume data on topics. The subscription service converts the middle-level source data in the source database by calling an interface.

[0023] The label generation and matching module is used to automatically generate labels for the business data according to label rules through the management background using visual data configuration to complete the matching of decision information.

[0024] The present invention also provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

[0025] The present invention also provides an electronic device, characterized in that it includes a processor and a memory;

[0026] The memory is used to store computer programs;

[0027] The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

[0028] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement the steps of the above method when executed by a processor.

[0029] The beneficial effects of the present invention are as follows: (1) through the automated configuration process and real-time data synchronization technology, the time for manual configuration and data synchronization is reduced, and the efficiency of data synchronization is improved; (2) users can easily configure the data source through the visual interface without in-depth understanding of the database and network configuration, which simplifies the data access process; (3) through configuration files and visual management, it can quickly adapt to changes in data sources and adjustments to business needs, thereby improving the flexibility and scalability of the system; (4) since the need for customized development is reduced, the development and maintenance costs of the system are reduced accordingly; (5) the data collector cluster can monitor changes in data sources in real time and synchronize the changes to the target system in real time, supporting real-time data processing and analysis; (6) through the control of synchronization sites, the consistency of data during the collection process can be ensured, and it can be continued from the correct position even after the collection task is paused or restarted; (7) by converting database layer source data into business layer data, it supports rapid business decision-making and data analysis, thereby improving the efficiency and accuracy of business decision-making; (8) the unified management platform can conveniently manage and monitor the synchronization tasks and synchronization status of each system, and once an anomaly or failure occurs, it can quickly locate and solve the problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The figure is a flowchart of a decision-making method for real-time synchronization of data based on visual configuration according to the present invention.

[0031] Figure 2 This is a schematic diagram of the structure of the decision-making system for real-time synchronous data based on visual configuration of the present invention. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0033] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are used only to distinguish the description and should not be understood as indicating or implying relative importance.

[0034] The first aspect of the present invention provides a decision-making method for real-time synchronization data based on visual configuration, such as Figure 1 As shown, including:

[0035] S1 deploys a data collector cluster on the target system;

[0036] Specifically, you first need to deploy a database collector cluster in the target system to collect data from the data source system. The collector cluster must have read permissions to access the data source system database, and the network policy to the data source system database deployment server needs to be opened. Specifically, it can be understood that the enterprise has deployed a group of data collectors (such as Canal or Debezium clusters) in its data center. These collectors are responsible for real-time monitoring and capturing change events of source databases (such as MySQL, Oracle, etc.). The data collector cluster is configured with high availability and load balancing to ensure the continuity of data collection in the event of any single point of failure. The data collector cluster is used to collect data from the data source system, and at the same time, the collector cluster is set with read permissions to access the data source system database, and the network policy to the data source system database deployment server needs to be opened.

[0037] S2 configures the collection task, generates a configuration file, and pushes the configuration file to the database collector cluster;

[0038] Specifically, the company's IT team uses a proprietary visual task management service to configure data collection tasks, including specifying the database tables and fields to be synchronized. Once configured, the system automatically generates configuration files and pushes these configurations to the deployed data collector cluster via API calls, enabling the collectors to execute tasks according to the new configurations.

[0039] More specifically, configure a user name and password with permission to read the data source system database on the collection platform (a self-developed collection task management visualization service) interface. After configuration, you can pull the table structure of the data source system database, configure the required tables and fields on the interface based on business needs, and finally configure the topic address and name applied for on the collection bus / data bus (based on the data flow middleware Kafka). This step aims to customize all data requirements that the target system needs to collect from the data source system and generate a configuration file. The generated configuration file is pushed to the database collector (Canal) cluster deployed on the target system through an interface call. The database collector cluster collects customized tables and fields according to the configuration file to meet dynamically adjusted business needs.

[0040] S3 synchronizes data in real time, using a data collector cluster to save synchronization site information to the acquisition platform, which uses synchronization sites to control the start and pause of acquisition tasks;

[0041] Specifically, the data collector cluster captures database changes in real time and sends them to a message queue (such as Kafka) over the network, saving the location of the changes to the collection platform. The collection platform records the location of each data synchronization, such as the binlog file name and location, to ensure that synchronization can resume from the last stopped point in the event of a system restart or failover.

[0042] S4 deploys a subscription service in the target system to consume data on the topic. The subscription service converts the middle-level source data in the source database by calling an interface.

[0043] Specifically, enterprises deploy subscription services on target systems (such as data warehouses or data analysis platforms). These services are responsible for consuming data from message queues such as Kafka. Subscription services call predefined interfaces and convert database-layer source data obtained from message queues into business-layer data for subsequent processing according to calling rules. These calling rules may include unified conversion of data formats, data cleaning and deduplication, data mapping, data trimming and aggregation, etc. After receiving the startup command from the collection platform, the database collector cluster of the target system begins real-time synchronization of MySQL's binlog and saves the binlog synchronization points to the collection platform. The collection platform can use the points to stop and restart the collection service at any time to ensure that start and stop operations do not affect data accuracy.

[0044] The target system writes the processed data to databases such as MySQL, Elasticsearch, and Redis according to different business usage scenarios.

[0045] The S5 management backend uses visual data configuration to automatically generate tags for the business data according to tag rules to complete the matching of decision information.

[0046] Specifically, the company's human resources department uses the target system's backend's visual configuration tools to define talent tagging rules based on business needs. The backend automatically generates tags for synchronized business data based on these rules, matching employee skills, experience, and other information with job requirements to support talent decision-making and resource optimization.

[0047] The present invention also provides a decision-making system for real-time synchronized data based on visual configuration, which is characterized by comprising:

[0048] Cluster deployment module, used to deploy data collector clusters;

[0049] Configuration collection module, used to configure collection tasks, generate configuration files, and push the configuration files to the database collector cluster;

[0050] The data synchronization module is used to synchronize data in real time and save the synchronization location information to the collection platform using the data collector cluster. The collection platform uses the synchronization location to control the start and pause of the collection task;

[0051] The data conversion module is used to deploy subscription services to consume data on topics. The subscription service converts the middle-level source data in the source database by calling an interface.

[0052] The label generation and matching module is used to automatically generate labels for the business data according to label rules through the management background using visual data configuration to complete the matching of decision information.

[0053] By using this system, the above-mentioned processing method can be executed and the corresponding technical effects can be achieved.

[0054] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all steps of the method in the above embodiment. The computer-readable storage medium stores a computer program that implements all steps of the method in the above embodiment when executed by a processor.

[0055] An embodiment of the present invention also provides an electronic device for executing the above-mentioned method. As an implementation device of the method, the electronic device has at least a processor and a memory, and in particular, the memory stores the data and related computer programs required for executing the method, and the processor calls the data and programs in the memory to execute all the steps of the implementation method and obtains the corresponding technical effect.

[0056] Preferably, the electronic device may include a bus architecture, which may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are all well known in the art and, therefore, will not be described further herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter can be the same component, namely a transceiver, which provides a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory can be used to store data used by the processor when performing operations.

[0057] Additionally, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operating control) employed may include a microprocessor or other processor device and / or logic device, which receives input and controls the operation of various components of the electronic device. The memory may be one or more of a cache, flash memory, a hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices, and may store the aforementioned data and information. It may also store programs for executing the relevant information, and the processor may execute the programs stored in the memory to implement information storage or processing. The input unit is used to provide input to the processor, and may, for example, be a keypad or touch input device. The power supply is used to provide power to the electronic device. The display is used to display objects such as images and text, and may, for example, be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, similar to the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide audio output via the speaker and receive audio input from the microphone, thereby implementing common telecommunications functions. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processing unit, enabling local recording via the microphone and playback of stored audio via the speaker.

[0058] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or boxes.

[0060] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system that is implemented in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The present invention is described in detail below. ...

[0062] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention are intended to be covered by the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A decision-making method for real-time synchronization data based on visual configuration, characterized in that: include: S1 deploys a data collector cluster on the target system; S2 configures the collection task, generates a configuration file, and pushes the configuration file to the database collector cluster; S3 synchronizes data in real time, using a data collector cluster to save synchronization site information to the acquisition platform, which uses synchronization sites to control the start and pause of acquisition tasks; S4 deploys a subscription service in the target system to consume data on the topic. The subscription service converts the middle-level source data in the source database by calling an interface. The S5 management backend uses visual data configuration to store the business data according to the tag specifications Then the label generation is automatically completed to complete the matching of decision information.

2. The method according to claim 1, characterized in that The S2 configuration collection task includes: Set the read permission of the data collector cluster in the source database; Configure table and field requirements on the collection platform; Use Kafka middleware to configure and apply for the Topic address and name for the collection task.

3. The method according to claim 1, characterized in that The S2 pushes the configuration file to the database collector cluster deployed in the target system, including collecting the customized tables and fields in the configuration file through interface calls.

4. The method according to claim 1, wherein The S3 real-time synchronization data uses a data collector cluster to save the synchronization site information to the collection platform, including: the database collector cluster uses MySQL binlog to complete real-time data synchronization after receiving the collection command from the collection platform.

5. The method according to claim 1, wherein The subscription service of S4 realizes the conversion of the layer source data in the source database by calling the interface, including: converting the layer source data into the business data required by the scene according to the calling rules, and writing the obtained business data into the cache database.

6. The method according to claim 1, characterized in that The S5 management backend uses visual data configuration to automatically generate labels for the business data according to label rules, including: analyzing the business data according to different dimensions and different classification methods and generating target portraits.

7. A decision-making system based on real-time synchronous data of visual configuration, characterized in that: include: Cluster deployment module, used to deploy data collector clusters; Configuration collection module, used to configure collection tasks, generate configuration files, and push the configuration files to the database collector cluster; The data synchronization module is used to synchronize data in real time and save the synchronization location information to the collection platform using the data collector cluster. The collection platform uses the synchronization location to control the start and pause of the collection task; The data conversion module is used to deploy subscription services to consume data on topics. The subscription service converts the middle-level source data in the source database by calling an interface. The label generation and matching module is used to automatically generate labels for the business data according to label rules through the management background using visual data configuration to complete the matching of decision information.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which implements the method according to any one of claims 1 to 6 when executed by a processor.

9. An electronic device, characterized in that: including processor and memory; The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.

10. A computer program product comprising a computer program and / or instructions, characterized in that When the computer program and / or instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images