Visual data processing method

Abstract

The invention discloses a visual data processing method, and relates to the technical field of data processing, and the method comprises the steps: selecting a task triggering component according to an input data type, determining a target data flow according to the task triggering component, and improving the real-time efficiency of data processing by rapidly identifying data characteristics and starting a corresponding process. On the basis of the target data stream, at least one processing operation is carried out on the input data through the processing component to obtain the processed data, and the required component can be quickly expanded through component design, so that the data processing flexibility is improved, the code process is simplified, and the data processing efficiency is improved. And outputting the processed data to a target data storage system or an external system through an output component according to the direction of the target data stream. The characteristics of different storage systems or external interfaces are adapted through the output assembly, it is ensured that processing results are distributed in an optimal path, data transmission delay is reduced through accurate directional output, and the real-time efficiency of data processing is further improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method for processing visual data. Background Art

[0002] With the exponential growth of data scale and the continuous improvement of real-time requirements in business scenarios, traditional data processing methods face severe challenges: the hard-coded development model relies on professional developers to write customized code or stream computing frameworks, which requires support and long-term maintenance from professional big data teams.

[0003] Traditional data visualization processing tools have problems with rigid data links and delayed responses, which results in low real-time efficiency of visualization data processing.

[0004] Therefore, how to improve the real-time efficiency of visual data processing is a problem that needs to be solved urgently. Summary of the Invention

[0005] The main purpose of this application is to provide a visual data processing method, aiming to solve the technical problem of low real-time efficiency of visual data processing.

[0006] To achieve the above objectives, the present application proposes a visual data processing method, which includes:

[0007] Selecting a task trigger component according to an input data type, and determining a target data flow according to the task trigger component;

[0008] Based on the target data stream, performing at least one processing operation on the input data by a processing component to obtain processed data;

[0009] According to the direction of the target data flow, the processed data is output to the target data storage system or external system through the output component.

[0010] In one embodiment, the target data stream includes a batch data stream and a real-time data stream, and the steps of selecting a task trigger component according to the input data type and determining the target data stream according to the task trigger component include:

[0011] If the input data type is batch data, the scheduled task component is selected as the task trigger component;

[0012] Through the scheduled task component, data is read from the target data source according to the preset batch size to generate batch data streams;

[0013] If the input data type is real-time data, the monitoring component is selected as the task triggering component;

[0014] The monitoring component parses the log change events of the target data source, captures incremental data in real time and generates a real-time data stream.

[0015] In one embodiment, the step of parsing the log change events of the target data source by the monitoring component, capturing incremental data in real time, and generating a real-time data stream includes:

[0016] Based on the monitoring component, log change events of the target data source are parsed in incremental synchronization mode, stock and incremental synchronization mode, or savepoint mode.

[0017] Based on the log change event of the target data source, dynamically allocate the number of listening threads and obtain the target listening thread;

[0018] Incremental data is captured in real time based on the target listening thread and a real-time data stream is generated.

[0019] In one embodiment, the processing component includes a basic processing component and a convergence pipeline component. Based on the target data stream, the processing component performs at least one processing operation on the input data to obtain processed data, including:

[0020] If the input data is homologous data, performing at least one processing operation on the homologous data by a basic processing component based on the target data stream to obtain processed data;

[0021] If the input data is multi-source heterogeneous data, based on the target data stream, the multi-source heterogeneous data is merged and processed by the convergence pipeline component to obtain standard data;

[0022] At least one processing operation is performed on the standard data by a basic processing component to obtain processed data.

[0023] In one embodiment, the basic processing component includes at least one of a data conversion component, a data filtering component, and a data aggregation component. The step of performing at least one processing operation on the homologous data by the basic processing component based on the target data stream to obtain processed data includes at least one of the following:

[0024] Based on the target data stream, configuring a field mapping strategy through the data conversion component, performing field mapping on the input data according to the field mapping strategy, and obtaining processed data;

[0025] Based on the target data stream, configuring data filtering rules through the data filtering component, filtering the input data according to the data filtering rules to obtain processed data;

[0026] Based on the target data stream, an aggregation dimension is configured through the data aggregation component, and the input data is dimensionally unified according to the aggregation dimension to obtain processed data.

[0027] In one embodiment, the processing component includes a convergence pipeline component, and the step of performing at least one processing operation on the input data by the processing component based on the target data stream to obtain processed data includes:

[0028] Based on the target data stream, dynamically monitor the input data through the convergence pipeline component to obtain monitoring results, wherein the monitoring results include data accumulation amount and buffering time;

[0029] If the data accumulation amount reaches a preset accumulation threshold or the buffer time reaches a preset overtime, the input data is transferred in batches to the downstream component;

[0030] If the configuration is a multi-node environment, a distributed queue is used to share the cache, and a distributed lock is used to perform concurrency control on the input data;

[0031] If the configuration is a single-node environment, the distributed queue is switched to a local queue, and the distributed lock is downgraded to a local lock.

[0032] In one embodiment, the step of outputting the processed data to a target data storage system or an external system via an output component according to the direction of the target data stream includes:

[0033] If the target data stream is directed to a target data storage system, the processed data is output to the target data storage system through a table partitioning strategy or a dynamic indexing rule according to the characteristics of the target data storage system;

[0034] If the target data flow is directed to an external system, the processed data is output to the external system via a message sending component or an interface pushing component.

[0035] In one embodiment, before the steps of selecting a task triggering mode according to the input data type and determining the target data stream according to the task triggering mode, the method further includes:

[0036] In response to the user's drag command, the target components are connected according to the program design logic to form an initial data flow;

[0037] Based on the initial data flow, each target component is configured according to business requirements to obtain a target data flow.

[0038] In one embodiment, the step of configuring each target component according to business requirements based on the initial data flow to obtain the target data flow includes:

[0039] Based on the initial data flow, configuring each of the target components according to business requirements to obtain configuration information;

[0040] Broadcast the configuration information to the distributed cluster for real-time update, and the cluster nodes parse the component mapping relationship and load it into the execution engine;

[0041] Select the execution node through the scheduling strategy, obtain the target data flow, and generate the version record;

[0042] If the update fails, roll back to the previous stable version based on the version record.

[0043] In one embodiment, the steps of selecting an execution node through a scheduling policy, obtaining a target data stream, and generating a version record include:

[0044] Select at least one of full-node scheduling, designated-node scheduling, or dynamic node scheduling through a scheduling strategy to determine a scheduling node;

[0045] Periodically send heartbeat signals to the scheduling node to monitor the health status of the scheduling node;

[0046] The data flow task execution status is updated according to the health status of the scheduling node, the target data flow is obtained, and a version record is generated.

[0047] One or more technical solutions proposed in this application have at least the following technical effects:

[0048] Select a task trigger component based on the input data type, and determine the target data flow based on the task trigger component. By quickly identifying data characteristics and starting the corresponding process, the real-time efficiency of data processing is improved. Based on the target data flow, the processing component performs at least one processing operation on the input data to obtain processed data. The componentized operation improves the flexibility of data processing, simplifies the code flow, and improves the efficiency of data processing. According to the direction of the target data flow, the processed data is output to the target data storage system or external system through the output component. The output component adapts to the characteristics of different storage systems or external interfaces to ensure that the processing results are distributed along the optimal path. The precise directional output reduces data transmission delay and further improves the real-time efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] Figure 1 This is a flowchart of the first embodiment of the visual data processing method of the present application;

[0052] Figure 2 This is a flow chart of the second embodiment of the visual data processing method of the present application;

[0053] Figure 3 This is a schematic diagram of the batch processing interface operation of data in the embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the data stream processing interface operation in an embodiment of the present application;

[0055] Figure 5 This is a schematic diagram of the interface operation of combining data batch processing and stream processing in an embodiment of the present application;

[0056] Figure 6 This is a flowchart of the third embodiment of the visual data processing method of the present application;

[0057] Figure 7 This is a schematic diagram of multiple nodes converging to one node in an embodiment of the present application;

[0058] Figure 8 This is a schematic diagram of parallel processing in an embodiment of the present application;

[0059] Figure 9 This is a flowchart of a fourth embodiment of the visual data processing method of the present application;

[0060] Figure 10 This is a schematic diagram of parallel processing in an embodiment of the present application;

[0061] Figure 11 This is a schematic diagram of multiple nodes failing over in an embodiment of the present application;

[0062] Figure 12 This is a schematic diagram of the overall back-end architecture of the embodiment of this application.

[0063] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0065] In order to better understand the technical solution of this application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0066] Traditional data synchronization and cleansing methods have significant limitations. The hard-coded model relies on professional developers writing customized code, resulting in long development cycles (2 to 5 man-days per process), repetitive work (similar scenarios require repeated development), high maintenance costs (data structure changes require code refactoring), and technical barriers to entry for non-developers. While stream computing frameworks (such as FlinkCDC) support real-time processing, they require the support of a professional big data team (increasing labor costs by 30% to 50%), are complex to develop (requiring in-depth knowledge of distributed systems principles), difficult to debug (difficult to locate real-time streaming issues), and require long-term maintenance of the cluster infrastructure, consuming large amounts of resources. ETL tools (such as DataX / Canal), which combine configuration files with customized code, still face challenges such as complex configuration (requiring over 100 lines of configuration for a single task), poor scalability (relying on Java code development for special needs), weak visualization capabilities (lack of a process design interface), and difficult version management (configuration changes cannot be visually compared). These issues result in low real-time efficiency in visual data processing. Therefore, improving the real-time efficiency of visual data processing is an urgent issue that needs to be addressed.

[0067] The present application provides a solution that selects a task trigger component based on the input data type, and determines the target data stream based on the task trigger component, thereby improving the real-time efficiency of data processing by quickly identifying data characteristics and starting the corresponding process. Based on the target data stream, at least one processing operation is performed on the input data through the processing component to obtain processed data. The flexibility of data processing is improved through componentized operations, and the efficiency of data processing is improved while simplifying the code process. According to the direction of the target data stream, the processed data is output to the target data storage system or external system through the output component. By adapting the characteristics of different storage systems or external interfaces through the output component, it is ensured that the processing results are distributed along the optimal path. The precise directional output reduces data transmission delay and further improves the real-time efficiency of data processing.

[0068] It should be noted that this application can be applied to the data flow engine. The visual drag-and-drop interface hides the technical complexity behind a friendly operation page. Users can complete the data flow construction through intuitive graphical operations, greatly reducing the usage threshold. The data flow engine is responsible for parsing the flowchart constructed by the user, converting it into an executable task sequence, and handling data transfer between components. The engine automatically parallelizes and executes components in parallel to achieve operations such as data synchronization, cleaning, conversion, and aggregation. The process is the business implementation.

[0069] Based on this, the embodiment of the present application provides a method for processing visual data. Figure 1 , Figure 1 This is a flowchart of the first embodiment of the visualization data processing method of this application.

[0070] In this embodiment, the visual data processing method includes steps S10 to S30:

[0071] Step S10: selecting a task trigger component according to the input data type, and determining a target data flow according to the task trigger component.

[0072] It should be noted that the input data type can be understood as the format or category of the original data entering the system, such as structured data (such as a database table) or semi-structured data (such as JSON / XML). For example, data can be obtained from a database, message queue, or IoT device in real time or in batches. The task trigger component can select the corresponding data processing link, i.e., the target data flow, by matching the data type. The target data flow can be understood as the finalized data processing path plan, which can include a series of ordered processing components and data flow rules.

[0073] Step S20: Based on the target data stream, at least one processing operation is performed on the input data by the processing component to obtain processed data.

[0074] It should be noted that the processing component can be understood as a functional module that performs specific data processing operations, such as a data conversion component, a data filtering component, etc., wherein the data conversion component can convert data formats with different structures into a preset standard data format, and the data filtering component can filter out data that does not conform to the preset rules. It should be noted that multiple processing components can be dynamically combined according to business needs to form a pipeline. The processed data can be understood as the result after processing by the processing component, and the result can include newly generated features and structured data. For example, the processed data can be obtained by the conversion or calculation behavior performed by the processing component on the input data, such as deduplication, normalization, aggregation statistics and other operations.

[0075] Step S30: output the processed data to a target data storage system or an external system through an output component according to the direction of the target data flow.

[0076] It should be noted that the output component can push processed data to a target data storage system or an external system. Exemplary output components include a write component and a message sending component. The target data storage system can be understood as the data storage system of the final system configuration. Data storage systems include but are not limited to relational databases (MySQL), NoSQL databases (MongoDB), and data lakes (HDFS). External systems can be understood as external services that interact with the current system, such as third-party applications.

[0077] In this embodiment, a task trigger component is selected according to the input data type, and the target data flow is determined based on the task trigger component. By quickly identifying the data characteristics and starting the corresponding process, the real-time efficiency of data processing is improved. Based on the target data flow, at least one processing operation is performed on the input data through the processing component to obtain processed data. The component design facilitates the operator to quickly configure the required card capabilities. In terms of business code, various components can be quickly expanded according to demand, which improves the flexibility of data processing, simplifies the code process, and improves the efficiency of data processing. According to the direction of the target data flow, the processed data is output to the target data storage system or external system through the output component. The output component adapts to the characteristics of different storage systems or external interfaces to ensure that the processing results are distributed along the optimal path. The precise directional output reduces data transmission delay and further improves the real-time efficiency of data processing.

[0078] Reference Figure 2 , Figure 2 This is a flow chart of the second embodiment of the visualization data processing method of this application, based on the above Figure 1 The first embodiment shown here proposes a second embodiment of the visualization data processing method of the present application.

[0079] In the second embodiment, step S10 includes:

[0080] Step S101: If the input data type is batch data, a scheduled task component is selected as a task trigger component.

[0081] It should be noted that batch data can be understood as structured or semi-structured data acquired in batches at a fixed period or threshold. The scheduled task component can be understood as an automated module that schedules data reading based on preset time intervals or trigger conditions.

[0082] Step S102 : Reading data from a target data source according to a preset batch size through a scheduled task component to generate a batch data stream.

[0083] It should be noted that the batch size can refer to the amount of data processed by a single batch task, and the batch data stream can be understood as an ordered data set formed by batch processing. Figure 3 As shown in the figure, through the scheduled task component, when the scheduled task is triggered, the downstream components are executed, such as querying the Doris database and obtaining the query results. Data is queried according to the preset batch size. Each batch of data is executed and written to the MySQL database. At the same time, logs are printed in parallel and data is output for viewing. When there is no data, the task is terminated and waits for the next scheduled task to be triggered.

[0084] Step S103: If the input data type is real-time data, a monitoring component is selected as the task triggering component.

[0085] It should be noted that real-time data can be understood as dynamic data that is continuously generated and requires immediate response, and the monitoring component can be understood as a module that parses data log changes in real time and triggers processing.

[0086] Step S104: parsing the log change events of the target data source through the monitoring component, capturing incremental data in real time and generating a real-time data stream.

[0087] It should be noted that log change events can be understood as event notifications generated when the data source status changes, and incremental data can be understood as newly added data when the data changes. For example, Figure 4 As shown, the monitoring component (the corresponding backend technology can be Debezium) monitors the log files (such as Binlog) of databases such as MySQL, obtains the parsed data, and continuously transmits the data downstream. When the data changes (such as adding or deleting operations), the monitoring will be triggered to obtain the data. For example, Figure 5 As shown, log change events can be generated by batch data streams or real-time data streams.

[0088] In this embodiment, the processing link is dynamically selected according to the input data type. Through the scheduled task component, data is read from the target data source according to the preset batch quantity to generate a batch data stream. The log change events of the target data source are parsed through the monitoring component, and incremental data is captured in real time to generate a real-time data stream. This reduces the configuration complexity and provides a complete automated process from input to output, greatly improving the data processing efficiency.

[0089] In one embodiment, step S104 includes: based on the listening component, parsing the log change events of the target data source through the incremental synchronization mode, the inventory and incremental synchronization mode or the save point mode; based on the log change events of the target data source, dynamically allocating the number of listening threads and obtaining the target listening thread; based on the target listening thread, capturing incremental data in real time and generating a real-time data stream.

[0090] It should be noted that the incremental synchronization mode can be understood as only capturing and synchronizing incremental changes of the data source (such as add, update, and delete operations), without repeatedly processing historical data. The stock and incremental synchronization mode can be understood as first synchronizing the entire historical data (stock), and then continuously synchronizing incremental changes. The save point mode can be understood as periodically recording the progress of data processing (such as offsets and timestamps), and supporting continued processing from the save point during fault recovery. For example, the number of listening threads can be automatically adjusted according to the system load (such as CPU / memory usage, queue backlog). The listening component is implemented through the Debezium engine (DebeziumEngine), and the listening information is configured by double-clicking the listening component. Multiple selections are supported for both data sources and tables. Whether the status is enabled can be configured. Specifically, the startup strategy can include incremental synchronization mode (INCREASE incremental synchronization), stock and incremental synchronization mode (STOCK_AND_INCREASE), and save point mode (LAST_STORAGE). LAST_STORAGE can specifically start from the latest save point, and if there is no save point, an error will be reported. AUTO starts from the latest save point, and if there is no save point, it starts according to the stock and incremental synchronization mode (default mode). CUSTOM starts from the specified save point, and if there is no save point, an error will be reported. Exemplarily, the monitoring component can also include a recovery mode (RECOVERY), which is used to repair the library table structure. At the same time, the monitoring operation type can be supported. Exemplarily, the following operation types can be supported, such as stock, create, update, and delete operations, and multiple selections can also be supported. Specifically, the page supports configuring the number of threads. If no thread count is configured, a default thread count based on the number of tables is selected. For example, if you select a single table, the thread count is 1; if you select more than two tables, the thread count is 2; if you select more than five tables, the thread count is 3, and so on. Note that a single listener component is limited to returning a maximum of four threads. Savepoints are supported to prevent system outages, allowing tasks to be resumed after a restart or resynchronized from a specified savepoint. The underlying implementation supports DebeziumEngine's offset.storage policy, allowing savepoints to be stored in Redis and MySQL (both support periodic deletion of historical savepoints). Upon project startup, the listener component first reads the savepoint from Redis or MySQL according to the startup policy and then performs the relevant recovery operations. This component also supports monitoring various databases (including MySQL, Oracle, Doris, PolarDB, SqlServer, PostgreSQL, Db2, and all databases that include the MySQL protocol). The listener component also supports custom time zones, implementing Debezium's database.connectionTimeZone method to support operations in overseas markets.

[0091] In this embodiment, by dynamically selecting incremental synchronization, stock and incremental synchronization, or save point mode based on the monitoring component to parse the log change events of the target data source, the system can flexibly adapt to the data processing needs of different business scenarios. The incremental mode reduces the repeated processing of the full data, the stock + incremental mode ensures the integrity of the cold start data, and the save point mode realizes breakpoint resumption and fault recovery. Combined with the dynamic allocation of the number of listening threads, the system can automatically adjust resource usage according to the real-time load (such as expanding threads at high concurrency and reducing them at low load) to avoid resource waste or processing delays. The save point mechanism ensures data reliability in abnormal scenarios, and ultimately achieves multi-dimensional optimization of resource utilization, real-time performance, and fault tolerance.

[0092] Reference Figure 6 , Figure 6 This is a flow chart of the third embodiment of the visualization data processing method of this application, based on the above Figure 2 The second embodiment shown here proposes the third embodiment of the visualization data processing method of the present application.

[0093] In the third embodiment, step S20 includes:

[0094] Step S201: If the input data is homologous data, at least one processing operation is performed on the homologous data by a basic processing component based on a target data stream to obtain processed data.

[0095] It should be noted that homologous data can be understood as original data from the same data source or the same data structure. The basic processing component can be understood as a set of functional modules for basic processing of data, which may include at least one of a data conversion component, a data filtering component, and a data aggregation component. It should be noted that step S201 includes at least one of the following: based on the target data flow, configuring the field mapping strategy through the data conversion component, performing field mapping on the input data according to the field mapping strategy, and obtaining processed data; based on the target data flow, configuring the data filtering rules through the data filtering component, filtering the input data according to the data filtering rules, and obtaining processed data; based on the target data flow, configuring the aggregation dimension through the data aggregation component, unifying the dimensions of the input data according to the aggregation dimension, and obtaining processed data.

[0096] For example, a field mapping strategy can be used to define the correspondence and conversion rules between input and output fields. Data filtering rules can be used to filter logical conditions for data (such as SQL WHERE clauses and regular expression matching). Aggregation dimensions can be understood as the fields used to group data (such as calculating daily sales by the "date" dimension).

[0097] Step S202: If the input data is multi-source heterogeneous data, based on the target data stream, the multi-source heterogeneous data is merged and processed by the convergence pipeline component to obtain standard data.

[0098] It should be noted that multi-source heterogeneous data can be understood as data from different data sources or with different structures (such as MySQL order table + MongoDB log document + CSV file). The convergence pipeline component can be used in three scenarios: multi-source data merging, data flow control, and data synchronization. In this embodiment, it is mainly used for multi-source data merging, such as Figure 7 The convergence pipeline component can support receiving data input from multiple upstream data sources at the same time. For example, data fusion can be achieved through data standardization (such as unified timestamp format) and field alignment (such as filling missing fields).

[0099] Step S203: Perform at least one processing operation on the standard data through the basic processing component to obtain processed data.

[0100] For example, Figure 4 As shown, after merging and processing multi-source heterogeneous data through the convergence pipeline component to obtain standard data, at least one processing operation (such as data conversion or data filtering, etc.) can be performed on the standard data through the basic processing component according to business needs.

[0101] In this embodiment, if the input data is from the same source, basic processing components can be combined according to business needs to perform basic operations on the data. Basic processing components can be flexibly combined to meet diverse processing needs. Rules can be defined through a configuration interface, eliminating the need for code development and simplifying the operational process. If the input data is from multiple sources and heterogeneous data, it can be standardized through a convergence pipeline to achieve data fusion and support the processing of multiple data sources. Through automated standard processing processes, manual intervention is reduced, and the real-time and efficiency of data processing are improved.

[0102] In one embodiment, based on the above examples and the above implementation methods, it should be noted that the processing component includes a convergence pipeline component, and the step S20 includes: based on the target data flow, dynamically monitoring the input data through the convergence pipeline component to obtain monitoring results, and the monitoring results include data accumulation and buffering time; if the data accumulation reaches a preset accumulation threshold or the buffering time reaches a preset excess time, the input data is batch-delivered to the downstream component; if it is configured as a multi-node environment, a distributed queue is used to share the cache, and a distributed lock is used to perform concurrent control of the input data; if it is configured as a single-node environment, the distributed queue is switched to a local queue, and the distributed lock is downgraded to a local lock.

[0103] It should be noted that the input data is dynamically monitored through the intelligent batch delivery mechanism, and the monitoring results include data accumulation and buffering time. The data accumulation can be understood as the number of data items accumulated in the cache per unit time, and the buffering time can be understood as the maximum stay time of data in the cache waiting for processing. Correspondingly, the preset accumulation threshold can be understood as the critical data volume that triggers batch delivery, and the preset exceedance time can be understood as the maximum stay time allowed for data in the cache. The multi-node environment can be understood as a distributed deployment scenario, such as Figure 8 As shown, it contains multiple independent processing nodes. The distributed queue shared cache can be understood as an area for temporarily storing data in a distributed queue for shared access by multiple nodes. Distributed locks can be a mechanism for coordinating concurrent access across nodes. The single-node environment can be understood as a stand-alone deployment scenario with only one processing node. The local queue can be understood as a cache queue in the memory of a single machine. It can be understood that the local lock can be understood as a synchronization mechanism between threads on a single machine. For example, Redis distributed queues can be used to implement multi-node shared cache, and automatically switch to JDK local queues in a single-node environment. JDK local queues can be understood as thread-safe queues implemented based on the JDK standard library. By setting a preset cumulative threshold (default 10,000) and a collaborative working mechanism for controlling the timeout (default 2000ms), the data is triggered to be transmitted downstream when any condition is met. Distributed locks are used to implement concurrency control in multi-node deployments, and automatically downgrade to JDK local locks in single-node environments.

[0104] In this implementation, the input data is dynamically monitored through the aggregation pipeline component, balancing real-time performance and throughput to avoid system overload. Multi-node distributed locks ensure data consistency, and cross-node transmission overhead is reduced through shared cache. Single-node local locks simplify implementation, and local queues reduce network latency. By automatically adapting to stand-alone or cluster environments, resource utilization is improved. The intelligent batch delivery mechanism significantly improves the batch processing efficiency of downstream data writing components. When the accumulated data reaches the set threshold, batch delivery is performed, and cached data is automatically delivered when a timeout is triggered to ensure data real-time performance. Traffic shaping is achieved through dynamic buffer pools, maximizing batch processing efficiency while ensuring data timeliness, solving the contradiction between data aggregation and real-time processing in a big data environment.

[0105] In one embodiment, based on the above examples and the above implementation methods, it should be noted that step S30 includes: if the target data flow is directed to the target data storage system, according to the characteristics of the target data storage system, the processed data is output to the target data storage system through the table partitioning strategy or dynamic indexing rules; if the target data flow is directed to an external system, the processed data is output to the external system through a message sending component or an interface push component.

[0106] It's important to note that sharding strategies can distribute data across multiple tables based on specific rules, such as horizontal sharding (sharding by user ID hash) or vertical sharding (sharding by field). Dynamic indexing rules can be understood as automatically creating or adjusting indexes based on data characteristics (such as frequently queried fields or write frequencies), for example, indexing timestamp fields to accelerate range queries. For example, data queries can be implemented using the query component, which is implemented underlyingly through ShardingSphere. When querying MySQL, if a sharding strategy is configured, the corresponding table data is automatically queried based on the sharding strategy. For example, for tables like order_202401, order_202402, and order_202403, a single table name, "order," is sufficient, eliminating the need to repeatedly query multiple tables. Similarly, when writing data to tables like order_202401, order_202402, and order_202403 in MySQL, a single table name, "order," is sufficient. This is implemented underlyingly through ShardingSphere and the configured sharding strategy. It should be noted that all writing components (such as writing to MySQL, writing to Doris, writing to StarRocks, etc.) only need to select the specified table, and do not need to write any update or insert and delete SQL statements. After the underlying encapsulation, it can be automatically executed according to the operation identifier passed in from the upstream (such as update UPDEAT, delete DELETE, add INSERT). The component configuration panel supports custom SQL. It should be noted that the priority of user-defined query, update and delete statements is higher than the system default settings.

[0107] For example, the write component (Elastic) supports dynamic indexing. When the user configures the index name as order_${createTime.toString('yyyyMM')}, the corresponding index name will be automatically generated based on the current data's creation time (createTime), such as order_202401. Then, the write operation is executed, and different index files are written based on strategies such as day or month, data field hashing, etc. The underlying support is provided by the InlineExpressionParserFactory component provided by ShardingSphere. The interface push component can provide standardized access protocols for external systems.

[0108] It should be noted that in order to ensure that the message consumption component does not consume data repeatedly, an idempotent process for message consumption has been designed. This process adds a unique message ID to all message transmissions, and then designs an idempotent table. For example, a default setting of 10 tables can be used, and then expanded according to business volume to increase data storage capacity and solve the problem of how much data a single table can handle, in order to meet the needs of big data. The message ID can be set as a unique key, and the database's unique index can be used to ensure the idempotence of the message. When a message is consumed, the message ID in the request header is first obtained. The message ID is hashed and then modulo 10 is calculated to determine which idempotent table to insert it into. If the insertion is successful, it indicates that the message is consumed for the first time, and normal business processing is carried out. If the insertion fails, it indicates that the message has already been consumed and is ignored.

[0109] In this embodiment, data storage pressure is dispersed according to the table partitioning strategy, query performance is optimized through dynamic indexing rules, query efficiency and storage scalability are improved, efficient data writing and retrieval are ensured, and compatibility with heterogeneous systems is achieved through data interfaces, which reduces manual configuration while achieving reliable data transmission. Furthermore, the flexibility of data output is improved to adapt to the needs of multiple scenarios.

[0110] Reference Figure 9 , Figure 9 This is a flow chart of the fourth embodiment of the visualization data processing method of this application, based on the above Figure 6 The third embodiment shown provides the fourth embodiment of the visualization data processing method of the present application.

[0111] In the fourth embodiment, before step S10, the following steps are included:

[0112] Step S010 , in response to a user drag command, connecting target components according to program design logic to form an initial data flow.

[0113] It should be noted that a drag-and-drop instruction can be understood as an operation in which a user selects and connects data processing components by dragging the mouse through a graphical interface (such as a visual programming tool). Programming logic can be understood as the system's built-in rules for parsing drag-and-drop operations and automatically connecting components to form a basic data flow. The target component can be understood as a data processing component (task trigger component, processing component, or output component) selected based on business needs. The initial data flow can be understood as the preliminary data path generated based on the user's drag-and-drop operation, which includes component connection relationships but is not bound to specific parameters.

[0114] Step S020 , based on the initial data flow, each target component is configured according to business requirements to obtain a target data flow.

[0115] For example, you can double-click the target component card to configure it. For example, you can set runtime parameters or rules for the component, such as data filtering conditions, output format, number of concurrent threads, etc. The target data flow can be understood as the complete data processing link formed after the configuration is completed, which is executable. For example, Figure 10 As shown in the figure, when you double-click the real-time data monitoring component in the canvas, the edit page corresponding to this component card pops up on the right. On the configuration page, you can configure the name, data source, operation type, data volume, number of threads, startup strategy, etc.

[0116] In this embodiment, through the drag-and-drop component design, any combination allows parallel execution, and one-to-many efficiency is improved to complete the synchronization, cleaning, and conversion processing of different data sources. The entire process is zero-code operation, and the visual operation greatly reduces the development cost, solving the problems of long development cycle and high technical threshold of traditional methods.

[0117] In one embodiment, step S020 includes: based on the initial data stream, configuring each target component according to business needs to obtain configuration information; broadcasting the configuration information to the distributed cluster for real-time updating, and the cluster node parsing the component mapping relationship and loading it into the execution engine; selecting the execution node through the scheduling strategy, obtaining the target data stream, and generating a version record; if the update fails, returning to the previous stable version based on the version record.

[0118] Exemplarily, the core of the data flow is the data flow engine (FlowEngine), which is used to manage the life cycle of the data flow, including loading, running, stopping, deleting, etc. When the data flow server is started for the first time, all the data flow configurations in the database will be loaded into this FlowEngine, or when the page is configured with the data flow to be executed, it will switch to the publishing page, and the description of the published data flow version and the number of nodes used for the data flow operation will be entered when publishing. You can choose to run several service instances in the data flow server cluster. At the same time, FlowEngine also supports running in all or a specified number of data flow server clusters. When the configuration is completed, click the publish confirmation button. The request will first reach the configuration service, and the configuration service will verify the parameters to verify the correctness of the data flow configuration. The data flow configuration information of this version (complete JSON) is then stored in the publishing record table to facilitate subsequent rollback operations. At the same time, the message queue (MQ) operation is published to the data flow cluster. After the data flow cluster receives the data flow configuration MQ, it performs preprocessing (parses component information) and maps the parsed component information to different component implementation strategies. After the second verification is completed without any abnormalities, the data flow is loaded into the data flow engine for execution.

[0119] It should be noted that the steps of selecting an execution node through a scheduling strategy, obtaining a target data stream, and generating a version record include: determining a scheduling node by selecting at least one of full-node scheduling, designated node scheduling, or dynamic node scheduling through a scheduling strategy; periodically issuing a heartbeat signal to the scheduling node to monitor the health status of the scheduling node; updating the data stream task execution status according to the health status of the scheduling node, obtaining a target data stream, and generating a version record.

[0120] For example, Figure 11 As shown, data flow nodes can be scheduled for execution through the scheduler. If all nodes are selected during configuration, all nodes in the data flow cluster are scheduled to start the data flow. If a specific data flow service node is selected, the data flow will only be run if the server's Internet Protocol (IP) matches. If the number of nodes is specified, for example, two, the scheduler automatically schedules two central processing units (CPUs) and the two nodes with the lowest memory usage to start the data flow. The scheduler periodically verifies whether the scheduled nodes are still running. If one of the two scheduled nodes fails and there is no heartbeat, it automatically finds a node with the lowest CPU and memory usage to run as a replacement. If the number of nodes is specified to be two, the scheduler finds that the data flow cluster has only one node. When new service instances are subsequently added to the cluster, the scheduler detects the newly registered instance and automatically increases the number of running data flows to two, thus ensuring system reliability. It is understood that when the data flow support page is notified to click the stop operation, the process is similar to the publishing process. When the configuration service receives the stop operation, it initially verifies whether the data flow is still running. If it is running normally, it sends an MQ notification to all data flow engines in the data flow cluster to shut down the data flow. For example, when writing to components such as Elasticsearch, Doris, and StarRocks, a multi-copy, multi-node approach is used to improve data writing stability. For example, the Doris production environment is often configured with three or more BE storage nodes. When the Doris component receives data for writing, it polls a node from the configured BE node address for the write operation. When a network anomaly such as a write failure occurs, the node automatically shifts and automatically pushes data to the next available BE node, then removes the faulty node. The faulty node enters a self-test state and uses a scheduled task to regularly ping every 10 seconds to check whether the network has recovered. After recovery, it rejoins the cluster.

[0121] It should be noted that all components use Java CgLib dynamic proxy mode (Java polymorphism, using strategy mode and responsibility chain mode to connect the data transmission of each component) to monitor the status of the component. The overall backend architecture is as follows Figure 12As shown. Prometheus is used to collect component operating indicators, such as running time, throughput, and failure counts, for display in real-time monitoring pages. Retry operations are added to all components in this proxy mode implementation class. When a downstream node encounters an exception, three retries are performed by default (configurable). If the failure still occurs after the retries, an alarm notification is sent to an external system (such as Feishu, Enterprise WeChat, or DingTalk Robot) during the component's exception handling attempt. Log collection during data flow runtime can be achieved through log capture tools (such as Promtail+Loki), making it easier to view operation and execution information on the data flow platform later.

[0122] In this embodiment, through the dynamic configuration broadcast mechanism, the system can synchronize component parameters in real time in the distributed cluster, ensure configuration consistency between multiple nodes, and avoid data processing errors caused by inconsistent parameters; the multi-mode support of full-node scheduling, designated node scheduling and dynamic node scheduling enables the system to cope with high-throughput scenarios (full-node parallelism) and accurately adapt to specific business needs (such as designating dedicated nodes to process sensitive data). At the same time, the dynamic scheduling strategy can automatically optimize resource allocation according to real-time load and improve resource utilization. The heartbeat signal and health status monitoring mechanism ensure the high availability of the scheduling node. When a node anomaly is detected, the task can be quickly migrated to a healthy node to avoid data processing interruptions caused by a single point of failure. The version record function provides atomic rollback capabilities for configuration changes. If the new configuration causes data processing anomalies (such as SQL syntax errors or index failures), the system can return to the previous stable version with one click, significantly reducing operation and maintenance risks.

[0123] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the visualization data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0124] The flow chart and block diagram in the accompanying drawings illustrate the method according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the part of this module, program segment or code comprises one or more executable instructions for realizing the logical function of regulation. It should also be noted that in some alternative implementations, the function annotated in the box can also occur in a sequence different from that annotated in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or operation of regulation, or can be realized by a combination of special hardware and computer instructions.

[0125] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0126] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A visual data processing method, characterized in that: The visual data processing method comprises: Selecting a task trigger component according to an input data type, and determining a target data flow according to the task trigger component; Based on the target data stream, performing at least one processing operation on the input data by a processing component to obtain processed data; According to the direction of the target data flow, the processed data is output to the target data storage system or external system through the output component.

2. The method according to claim 1, wherein The target data stream includes a batch data stream and a real-time data stream. The step of selecting a task trigger component according to the input data type and determining the target data stream according to the task trigger component includes: If the input data type is batch data, the scheduled task component is selected as the task trigger component; Through the scheduled task component, data is read from the target data source according to the preset batch size to generate batch data streams; If the input data type is real-time data, the monitoring component is selected as the task triggering component; The monitoring component parses the log change events of the target data source, captures incremental data in real time and generates a real-time data stream.

3. The method according to claim 2, wherein The step of parsing the log change events of the target data source by the monitoring component, capturing incremental data in real time and generating a real-time data stream includes: Based on the monitoring component, log change events of the target data source are parsed in incremental synchronization mode, stock and incremental synchronization mode, or savepoint mode. Based on the log change event of the target data source, dynamically allocate the number of listening threads and obtain the target listening thread; Incremental data is captured in real time based on the target listening thread and a real-time data stream is generated.

4. The method according to claim 1, wherein The processing component includes a basic processing component and a convergence pipeline component. Based on the target data stream, the processing component performs at least one processing operation on the input data to obtain processed data, including: If the input data is homologous data, performing at least one processing operation on the homologous data by a basic processing component based on the target data stream to obtain processed data; If the input data is multi-source heterogeneous data, based on the target data stream, the multi-source heterogeneous data is merged and processed by the convergence pipeline component to obtain standard data; At least one processing operation is performed on the standard data by a basic processing component to obtain processed data.

5. The method according to claim 4, wherein The basic processing component includes at least one of a data conversion component, a data filtering component, and a data aggregation component. The step of performing at least one processing operation on the homologous data by the basic processing component based on the target data stream to obtain processed data includes at least one of the following: Based on the target data stream, configuring a field mapping strategy through the data conversion component, performing field mapping on the input data according to the field mapping strategy, and obtaining processed data; Based on the target data stream, configuring data filtering rules through the data filtering component, filtering the input data according to the data filtering rules to obtain processed data; Based on the target data stream, an aggregation dimension is configured through the data aggregation component, and the input data is dimensionally unified according to the aggregation dimension to obtain processed data.

6. The method according to claim 4, wherein The processing component includes a convergence pipeline component. The step of performing at least one processing operation on the input data by the processing component based on the target data stream to obtain processed data includes: Based on the target data stream, dynamically monitor the input data through the convergence pipeline component to obtain monitoring results, wherein the monitoring results include data accumulation amount and buffering time; If the data accumulation amount reaches a preset accumulation threshold or the buffer time reaches a preset overtime, the input data is transferred in batches to the downstream component; If the configuration is a multi-node environment, a distributed queue is used to share the cache, and a distributed lock is used to perform concurrency control on the input data; If the configuration is a single-node environment, the distributed queue is switched to a local queue, and the distributed lock is downgraded to a local lock.

7. The method according to claim 1, wherein The step of outputting the processed data to a target data storage system or an external system through an output component according to the direction of the target data stream includes: If the target data stream is directed to a target data storage system, the processed data is output to the target data storage system through a table partitioning strategy or a dynamic indexing rule according to the characteristics of the target data storage system; If the target data flow is directed to an external system, the processed data is output to the external system via a message sending component or an interface pushing component.

8. The method according to any one of claims 1 to 7, characterized in that Before the step of selecting a task trigger mode according to the input data type and determining the target data stream according to the task trigger mode, the method further includes: In response to the user's drag command, the target components are connected according to the program design logic to form an initial data flow; Based on the initial data flow, each target component is configured according to business requirements to obtain a target data flow.

9. The method according to claim 8, wherein The step of configuring each target component based on the initial data flow according to business requirements to obtain a target data flow includes: Based on the initial data flow, configuring each of the target components according to business requirements to obtain configuration information; Broadcast the configuration information to the distributed cluster for real-time update, and the cluster nodes parse the component mapping relationship and load it into the execution engine; Select the execution node through the scheduling strategy, obtain the target data flow, and generate the version record; If the update fails, roll back to the previous stable version based on the version record.

10. The method according to claim 9, wherein The steps of selecting an execution node through a scheduling strategy, obtaining a target data stream, and generating a version record include: Select at least one of full-node scheduling, designated-node scheduling, or dynamic node scheduling through a scheduling strategy to determine a scheduling node; Periodically send heartbeat signals to the scheduling node to monitor the health status of the scheduling node; The data flow task execution status is updated according to the health status of the scheduling node, the target data flow is obtained, and a version record is generated.

Images