A data integration processing method and apparatus

Abstract

The application discloses a kind of data integration processing method and device, it is related to computer technical field.The specific embodiment of the method includes: generating target task according to the configuration information related to data integration, loading the plug-in corresponding to configuration information;Determine the task type of target task, through the execution mode that plug-in is pre-configured for task type, according to one or more of data source information and data file information, execute data integration operation;In response to execution result is successful, determine the link where target task is located, in the case where link is last link, determine that data integration is completed;In the case where link is not last link, according to configuration information generates next task and executes.The embodiment skips the matching process of file format, connects the detection, segmentation, synchronous of source data file through an event-driven framework, and provides failure retry strategy, further enhances the robustness and scalability of system.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data integration processing method and apparatus. Background Technology

[0002] In the data integration field of data middleware platforms, file data sources are a special type of source. In actual project applications, the data files accessed by clients are usually provided by upstream business entities, which are mostly other companies. These entities have different definitions of data file formats and specifications, and clients cannot request formats from upstream suppliers. This results in inconsistent and highly variable data formats for file access. Existing technologies mostly require customized development for specific file formats, leading to a complex and unstable access process. Therefore, a more efficient and stable solution is urgently needed. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data integration processing method and apparatus, which can at least solve the problem in the prior art where customized development for specific formats leads to a complex and unstable access process.

[0004] To achieve the above objectives, according to one aspect of the present invention, a data integration processing method is provided, comprising:

[0005] The target task is generated based on the configuration information for data integration, and the plugin corresponding to the configuration information is loaded; wherein, the configuration information includes data source information and data file information;

[0006] The task type of the target task is determined, and the data integration operation is performed based on one or more of the data source information and data file information, according to the execution method pre-configured for the task type in the plugin.

[0007] In response to a successful execution result, the current stage of the target task is determined. If it is the last stage, data integration is confirmed to be complete. If it is not the last stage, the next task is generated and executed based on the configuration information.

[0008] Optionally, when the target task is a probe task, generating the target task based on the configuration information related to data integration includes:

[0009] In response to the detection of newly added configuration information related to data integration in the database, the configuration information is read from the database to generate a probe task based on the configuration information;

[0010] The step of performing data integration operations based on one or more of the data source information and the data file information includes: detecting in the data source whether there is a data file that matches the data file information.

[0011] Optionally, the method further includes: in response to an execution result of failure, counting the number of failures in executing the probe task, and if the number of failures is less than a preset failure retry threshold, regenerating the target task based on the configuration information.

[0012] Optionally, when the target task is a segmented task, generating the target task based on the configuration information related to data integration includes:

[0013] In response to the success of the probe mission, a segmentation mission is generated based on the configuration information of the relevant data integration.

[0014] The step of performing data integration operations based on one or more of the data source information and the data file information includes:

[0015] Read the data file that matches the data file information from the data source, divide the data file into multiple sub-data files, and store the multiple sub-data files in the storage location of the storage device according to the storage location in the data file information.

[0016] Optionally, when the target task is a synchronous task, generating the target task based on the configuration information related to data integration includes:

[0017] In response to the success of the splitting task, a synchronization task is generated based on the relevant data integration configuration information;

[0018] The step of performing data integration operations based on one or more of the data source information and the data file information includes:

[0019] The system employs a multi-threaded parallel execution approach to read the multiple sub-data files from the storage location of the storage device and write the multiple sub-data files into the data warehouse.

[0020] Optionally, the process of configuring the plugin corresponding to the configuration information includes:

[0021] The file format is obtained from the data file information, the parsing method corresponding to the file format is determined, the original plugin pre-configured for the parsing method is loaded, and the initial information of the original plugin is initialized.

[0022] Based on the data source information and data file information, configure the execution methods for detecting the existence of data files, downloading and checking the integrity of data files, splitting data files, and clearing intermediate cache data.

[0023] The original plugin is updated based on the configuration execution method to obtain a plugin corresponding to the configuration information.

[0024] Optionally, loading the plugin corresponding to the configuration information includes:

[0025] Identify the target task processing service, transmit the target task to the target task processing service, and load the plugin corresponding to the configuration information through the target task processing service.

[0026] Optionally, the scheduler includes an event publisher and an event looper, and the method further includes:

[0027] The target task is submitted to the event publisher, which then publishes the target task to the event queue according to the preset time range and interval frequency.

[0028] The target task is captured from the event queue through the event looper, and the current load of each task processing service is obtained to determine the target task processing service based on the load.

[0029] Optionally, the method further includes one or more of the following:

[0030] The target task is tagged with the scheduler and persisted to the database, and simultaneously written to memory; in response to an exception of the scheduler, other schedulers retrieve the target tasks tagged with the scheduler node that have not been processed from the database and write the target tasks to memory.

[0031] In response to an anomaly in the target task processing service, an anomaly information is transmitted to the scheduler, causing the scheduler to mark the target task as fault-transferred and to re-determine the target task processing service so as to transfer the target task to the new target task processing service.

[0032] To achieve the above objectives, according to another aspect of the present invention, a data integration processing apparatus is provided, comprising:

[0033] The task module is used to generate target tasks based on configuration information related to data integration and load plugins corresponding to the configuration information; wherein, the configuration information includes data source information and data file information;

[0034] The execution module is used to determine the task type of the target task, and perform data integration operations based on one or more of the data source information and data file information, according to the execution method pre-configured for the task type in the plugin.

[0035] The judgment module is used to determine the stage of the target task in response to a successful execution result. If the stage is the last stage, it determines that data integration is complete; if the stage is not the last stage, it generates the next task based on the configuration information and executes it.

[0036] Optionally, when the target task is a reconnaissance task, the task module is configured to:

[0037] In response to the detection of newly added configuration information related to data integration in the database, the configuration information is read from the database to generate a probe task based on the configuration information;

[0038] The execution module is used to: detect whether there is a data file in the data source that matches the data file information.

[0039] Optionally, the device further includes: in response to an execution result of failure, counting the number of failures in executing the probe task, and if the number of failures is less than a preset failure retry threshold, regenerating the target task based on the configuration information.

[0040] Optionally, when the target task is a segmented task, the task module is configured to:

[0041] In response to the success of the probe mission, a segmentation mission is generated based on the configuration information of the relevant data integration.

[0042] The execution module is used for:

[0043] Read the data file that matches the data file information from the data source, divide the data file into multiple sub-data files, and store the multiple sub-data files in the storage location of the storage device according to the storage location in the data file information.

[0044] Optionally, when the target task is a synchronous task, the task module is configured to:

[0045] In response to the success of the splitting task, a synchronization task is generated based on the relevant data integration configuration information;

[0046] The execution module is used for:

[0047] The system employs a multi-threaded parallel execution approach to read the multiple sub-data files from the storage location of the storage device and write the multiple sub-data files into the data warehouse.

[0048] Optionally, the process of configuring the plugin corresponding to the configuration information includes:

[0049] The file format is obtained from the data file information, the parsing method corresponding to the file format is determined, the original plugin pre-configured for the parsing method is loaded, and the initial information of the original plugin is initialized.

[0050] Based on the data source information and data file information, configure the execution methods for detecting the existence of data files, downloading and checking the integrity of data files, and splitting data files; and

[0051] Configure the execution method for clearing intermediate cache data;

[0052] The original plugin is updated based on the configuration execution method to obtain a plugin corresponding to the configuration information.

[0053] Optionally, the task module is configured to: determine a target task processing service, transmit the target task to the target task processing service, and load the plugin corresponding to the configuration information through the target task processing service.

[0054] Optionally, the scheduler includes an event publisher and an event looper, and the apparatus further includes:

[0055] The target task is submitted to the event publisher, which then publishes the target task to the event queue according to the preset time range and interval frequency.

[0056] The target task is captured from the event queue through the event looper, and the current load of each task processing service is obtained to determine the target task processing service based on the load.

[0057] Optionally, the device further includes a flexible expansion / contraction module for:

[0058] The target task is tagged with the scheduler and persisted to the database, and simultaneously written to memory; in response to an exception of the scheduler, other schedulers retrieve the target tasks tagged with the scheduler node that have not been processed from the database and write the target tasks to memory.

[0059] In response to an anomaly in the target task processing service, an anomaly information is transmitted to the scheduler, causing the scheduler to mark the target task as fault-transferred and to re-determine the target task processing service so as to transfer the target task to the new target task processing service.

[0060] To achieve the above objectives, according to another aspect of the present invention, a data integration and processing electronic device is provided.

[0061] The electronic device of this invention includes: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement any of the data integration processing methods described above.

[0062] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the data integration processing methods described above.

[0063] To achieve the above objectives, according to another aspect of the present invention, a computing program product is provided. One such computing program product includes a computer program that, when executed by a processor, implements the data integration processing method provided in the present invention.

[0064] According to the solution provided by the present invention, one embodiment of the invention has the following advantages or beneficial effects: It skips the customized data integration process based on file formats, simplifies the complexity of accessing traditional data files to the Hive data warehouse through multi-threaded parallel extraction and plug-in design, and significantly improves the efficiency of secondary development. The event-driven framework provides comprehensive instance lifecycle management and visualized logs, reducing the difficulty of use for customers and lowering development and maintenance costs. The overall solution not only improves the flexibility and reliability of the system but also significantly enhances the efficiency of data access and resource utilization.

[0065] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0066] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0067] Figure 1 This is a schematic diagram of the main flow of a data integration processing method according to an embodiment of the present invention;

[0068] Figure 2 It is a system architecture diagram;

[0069] Figure 3 A schematic flowchart of a specific data integration processing method according to an embodiment of the present invention;

[0070] Figure 4 This is a flowchart illustrating an optional data integration processing method according to an embodiment of the present invention;

[0071] Figure 5 This is a flowchart illustrating the plug-in design process;

[0072] Figure 6 This is a flowchart illustrating the working process of the plugin;

[0073] Figure 7 This is a schematic diagram of the main modules of a data integration and processing device according to an embodiment of the present invention;

[0074] Figure 8 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0075] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention, such as a mobile device or server. Detailed Implementation

[0076] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0077] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0078] In the data integration field of data middleware platforms, file data sources are a special type of source. This is because the data files accessed by clients are typically provided by upstream business entities, which are mostly other companies. These entities have different definitions of data file formats and specifications, and clients cannot request formats from upstream suppliers. This results in inconsistent and highly variable data formats during file access. Existing technologies mostly involve customized development for specific data formats to achieve the goal of accessing the Hive repository. In the actual file access process, the diversity of client data files is significant, and existing implementations suffer from several problems. These problems make the file data source access process complex and unstable, necessitating a more efficient and stable solution.

[0079] First, existing methods often require developing a front-end service before the existing data integration service to convert the actual data files into a platform-recognizable format. This service, influenced by actual R&D conditions, is often unstable and lacks standardization, increasing the difficulty of troubleshooting and creating a disconnect from core data integration capabilities. Simultaneously, the preprocessing of data by the front-end service lengthens the data processing chain, potentially incurring additional storage and performance overhead, thus increasing costs. Specifically, as a separate product, data integration faces diverse upstream data files when actually sold and deployed to customers. These files vary widely in format and parsing methods. Developing a dedicated front-end parsing service for each format inherently reflects a lack of standardization; furthermore, limited by cost and time constraints, this service cannot be a highly sophisticated, high-throughput platform application, with R&D focus typically concentrated on file format parsing and conversion. When extracting large volumes of data, these front-end services struggle to ensure stable interfacing with the core data integration service at every stage under immense data pressure.

[0080] Secondly, data access via files often suffers from inaccurate data supply times. This means upstream data providers can only guarantee data file uploads within a certain timeframe, not down to a precise point in time. This raises questions about when front-end services should retrieve data and the impact of data retrieval failures on subsequent data integration. Specifically, files stored on upstream system servers mean clients cannot detect whether files have been uploaded. Traditionally, this is addressed through agreements—specifically, agreeing to extract files at a certain time each day. However, such agreements are unreliable. Due to various reasons, the upstream might upload files later than agreed, causing clients to be unable to retrieve data at the agreed time, resulting in extraction failure. Once extraction fails, various strategies arise, such as whether to retry, or whether extraction should continue the following day if today's data files are not yet available.

[0081] Furthermore, the file integration process can only be read using a single thread, resulting in less than ideal integration speed and a lack of optimization methods. This is particularly evident when processing large-scale data, severely impacting the efficiency and performance of data integration.

[0082] See Figure 1 The diagram shows the main flowchart of a data integration processing method provided by an embodiment of the present invention, which includes the following steps:

[0083] S101: Generate a target task based on the configuration information for data integration, and load the plugin corresponding to the configuration information; wherein, the configuration information includes data source information and data file information;

[0084] S102: Determine the task type of the target task, and perform data integration operations based on one or more of the data source information and data file information, according to the execution method pre-configured for the task type in the plugin;

[0085] S103: In response to a successful execution result, determine the current stage of the target task;

[0086] S104: If this is the last step in the process, the data integration is confirmed to be complete.

[0087] S105: If the step is not the last step, generate the next task according to the configuration information and execute it.

[0088] In the above implementation, data integration refers to consolidating data from different sources into a unified system or repository for consistent analysis and processing. The goal of data integration is to eliminate data silos, ensure data consistency and accuracy, thereby supporting better decision-making and business process optimization. This solution abstracts the integration of data files into "tasks," which are categorized into three types: "Detect," "Split," and "Sync." The Detect task is used to detect whether the data files from a specified data source are ready. The Split task is used to split the data files into fragments, supporting subsequent multi-threaded parallel extraction. The Sync task is used to integrate the fragmented data files into HDFS.

[0089] For steps S101-S105, users need to configure the data to be accessed. This configuration information includes data source information, data file information, and column separators for the data files. This information will be persisted to the database. Data file information refers to the metadata of the data file, including the file format (e.g., .txt, .csv), file encoding (e.g., GBK, UTF-8), and the specific storage path of the file on the storage device (e.g., a server). The column separator is a specific character or string used in big data scenarios to distinguish different fields (columns) of each record in the data file. Common separators include commas (,), tabs (\t), vertical bars (|), and dollar signs ($). It should be noted that the file format and encoding in the data source information and data file information, as well as the column separator, can be provided by the upstream provider, but the specific storage path of the file on the storage device (e.g., a server) needs to be set by the developers.

[0090] Column delimiters help programs correctly parse data in a file, ensuring that each field of every record can be accurately identified and processed. For example, if a two-dimensional table's data is stored in a file (using a .txt file as an example), the entire .txt file is a single large string to the program. Therefore, how to split the data into individual rows and each column within each row requires pre-defined delimiters. Currently, the industry standard is to use one row per record, but because column delimiters vary greatly, they need to be specified by the user.

[0091] See Figure 2 The diagram shown illustrates the system architecture of this application, primarily divided into two parts: the Scheduler and the Processor. The Scheduler, also known as the scheduling service, is mainly responsible for initiating file detection and scheduling file splitting tasks. It possesses elastic scaling capabilities and utilizes ZooKeeper for node discovery, ensuring high availability and scalability of the service. The Processor, the task processing service, is responsible for executing the file detection and splitting tasks. It also supports elastic scaling and uses ZooKeeper for node discovery, guaranteeing the efficiency and reliability of task processing.

[0092] The Metrics module implements metric reporting between the two processes, enabling real-time collection and reporting of key performance indicators. This data can be further transmitted to monitoring dashboards or monitoring systems such as Prometheus, helping operations personnel understand service status promptly. Internal communication mechanism: The service uses an RPC (Remote Procedure Call) component based on TCP (Transmission Control Protocol) for communication, ensuring efficient and stable data exchange between different service components.

[0093] The other components are defined as follows: 1) API: The entry point for external requests, providing a REST interface for external systems to initiate tasks, query metrics, query logs, etc. This component is independent of the system's automatic scheduling mechanism, allowing direct interaction between external systems and the system. 2) Looper: The event loop, whose main responsibility is to instantiate daily task instances and fetch and process events. 3) Scheduler: The task scheduler, containing multiple sub-components, responsible for the overall scheduling of file splitting tasks. Its specific responsibilities include, but are not limited to, task allocation, resource management, queue control, data persistence, and failover. 4) DetectTask: A file detection task that checks if a file is ready. 5) SplitTask: A file splitting task that checks if a file is ready and completes the entire process of obtaining, verifying, and splitting data files. 6) SyncTask: A file synchronization task that completes the synchronization operation of data files.

[0094] Example 1, applied to a probe mission. In this application, different missions occupy different stages; each stage emphasizes the specific position or step of a mission within the overall process, such as the probe mission being the first stage.

[0095] 1. The Scheduler checks the database for newly added configuration information according to a preset scanning cycle. If newly added configuration information is detected, it reads the configuration information from the database, generates a probe task Detect based on the configuration information, and submits the probe task Detect to the EventPublisher (a general component).

[0096] 2. The event publisher will publish the probe tasks to the event queue within the configured time range and interval frequency to drive the probe tasks to check whether the files are ready. Since upstream data files are often not placed on time, if no file is detected, the probe is considered to have failed. After a failure, a certain retry strategy is required, including the number of retries and the interval configuration. These time ranges and interval frequencies can be the default configuration or can be customized according to the aforementioned configuration information to schedule the interval for delayed execution and failure retries.

[0097] 3. The Looper, an event loop, captures probe tasks from the event queue and obtains the load information of each task processing processor to determine the target processor for that probe task. This method evenly distributes tasks across different processors to maximize resource utilization and prevent some processors from having too many or too few. In certain special scenarios, the required resources also need to be determined based on configuration information. By comprehensively considering the processor load, resource requirements, and hardware compatibility, the target processor is finally determined. Then, the probe task is dispatched to the target processor via RPC.

[0098] Suppose a probe task needs to process a large file. According to the configuration, this task requires a large amount of memory and a high-performance CPU. The event loop Looper checks the current load, available memory, and CPU performance of each Processor and selects the most suitable Processor to handle the task. If a Processor has a low load but insufficient memory or inadequate CPU performance, it will not be selected. Ultimately, the selected Processor will receive and process the probe task via RPC.

[0099] 4. The target processor begins processing the probe task, loading the plugins corresponding to the configuration information. The plugin configuration process will be discussed later. Figure 4Description shown:

[0100] 1) Based on the parsing method of the detection task, i.e. the parsing method corresponding to the file format in the configuration information, load the plugin corresponding to the parsing method, so that the detection task can be submitted to the thread pool through the plugin. In the thread pool, according to the execution method of detecting whether the data file exists in the data source information and data file information pre-configured in the plugin, the detection operation is performed, that is, to detect whether there is a data file in the data source that matches the data file information.

[0101] 2) If the probe is successful, it indicates that the data file is ready. The probe task execution success status (or information) is transmitted to the scheduler, so that the scheduler generates a split task according to the configuration information and submits the split task to the event publisher.

[0102] However, if the probe fails, the number of failures is counted, and the failure status (or information) and the number of failures are transmitted to the scheduler. The scheduler then compares the number of failures with a preset retry threshold. If the threshold is not reached, a new probe task is generated based on the configuration information and submitted to the event publisher. If the threshold is reached, the process ends directly and a failure is returned.

[0103] Example 2, applied to segmentation tasks.

[0104] 1. When the target Processor detects that the data file is ready, that is, when the execution result is successful, it returns the success status (or information) of the detection task to the Scheduler, so that the Scheduler generates a split task according to the configuration information and submits the split task to the event publisher.

[0105] 2. The event publisher will publish the segmentation tasks to the event queue within the configured time range and interval frequency. The Looper, through which the event loop captures the segmentation tasks from the event queue and obtains the load status of each task processing service Processor, determines the target Processor for handling the segmentation task. This method can evenly distribute tasks among different Processors to maximize resource utilization and prevent some Processors from having too many or too few. In some special scenarios, it is also necessary to determine the required resources based on the configuration information. By comprehensively considering the Processor load, resource requirements, and hardware compatibility, the target Processor is finally determined. Then, the segmentation task is dispatched to the target Processor via RPC. It should be noted that the target Processor here can be different from the target Processor for handling probe tasks.

[0106] 3. The target Processor begins processing the splitting task, loading the plugins corresponding to the configuration information. The plugin configuration process will be discussed later. Figure 4 Description shown:

[0107] 1) Submit the splitting task to the thread pool through the plugin. In the thread pool, the splitting operation is performed according to the execution method of reading and splitting the data file in advance according to the data source information and data file information configured in the plugin. That is, the data file matching the data file information is read from the data source, the data file is split into multiple sub-data files, and the multiple sub-data files are stored in the storage location of the storage device according to the storage location in the data file information.

[0108] Since the file itself is binary data, it cannot be read concurrently and must be read from beginning to end. The file splitting step is precisely to solve this problem. For most two-dimensional table data, the order of each data entry is usually irrelevant (like a deck of cards; as long as all the cards are present, the order doesn't matter), which provides an entry point for concurrent extraction. This solution determines the number of file slices based on the number of concurrent read threads configured by the user and distributes the data evenly among these file slices, one by one, to maximize resource utilization.

[0109] 2) If the split is successful, the scheduler transmits the success status (or information) of the split task to the scheduler. The scheduler then generates a synchronous split task based on its configuration information and submits it to the event publisher. That is, although both probe and split tasks are executed on the processor, the processor reports the task's execution status to the scheduler during execution. Upon receiving the success message, the scheduler determines whether to generate a next task based on the node where the task is located. For example, upon receiving the success status of the probe task, it will submit a corresponding split task to the event publisher.

[0110] Example 3, applied to synchronous tasks.

[0111] 1. When the target Processor detects that the data file has been split, that is, when the execution result is successful, it returns the status (or information) of successful splitting task execution to the Scheduler, so that the Scheduler generates a synchronization task (Sync) according to the configuration information and submits the synchronization task to the event publisher;

[0112] 2. The event publisher will publish synchronization tasks to the event queue within the configured time range and interval frequency. The Looper captures synchronization tasks from the event queue and obtains the load status of each task processing service Processor to determine the target Processor for handling the synchronization task. This method can evenly distribute tasks among different Processors to maximize resource utilization and prevent some Processors from having too many or too few. In some special scenarios, it is also necessary to determine the required resources based on the configuration information. By comprehensively considering the load status of Processors, resource requirements, and hardware compatibility, the target Processor is finally determined. Then, the synchronization task is dispatched to the target Processor via RPC. It should be noted that the target Processor here can be different from the target Processor for handling probe tasks and segmentation tasks.

[0113] 3. The target Processor begins processing the synchronization task, loading the plugins corresponding to the configuration information. The plugin configuration process will be discussed later. Figure 4 Description shown:

[0114] 1) The synchronization task is submitted to the thread pool via a plugin. Within the thread pool, based on the execution method pre-configured in the plugin according to the data file information, multiple threads read the sub-data files split in the previous step in parallel and write them to the HDFS directory corresponding to the Hive table, completing the file access. In other words, based on the storage location in the data file information, multiple threads execute in parallel to read the multiple sub-data files from the storage location on the storage device and write them into the data warehouse.

[0115] 2) If synchronization is successful, the success status (or information) of the synchronization task is transmitted to the scheduler, so that the scheduler can directly end the process if it determines that the synchronization task is the last task.

[0116] In addition, during the execution of the aforementioned detection, segmentation, and synchronization tasks, the Processor generates a separate log file for each task and writes it to the system. Figure 2 The Logger in the system facilitates subsequent tracking. The Logger is the log manager, responsible for outputting and retrieving logs of a specified type, as well as recycling historical logs.

[0117] The method provided in the above embodiments skips the file format matching operation and connects the detection, segmentation, and synchronization of source data files through an event-driven framework. It also provides a failure retry strategy and elastic scaling capabilities, further enhancing the system's robustness and scalability. Specifically, it generates target tasks by configuring information and loading corresponding plugins, and performs data integration operations based on task type and data source information, flexibly handling different types of integration tasks. By executing each step and determining whether the task is completed, the integrity and reliability of the data integration are ensured.

[0118] See Figure 3 The diagram illustrates a specific data integration processing method according to an embodiment of the present invention, including the following steps:

[0119] 1. The upstream provides data source information, file format and file encoding in the data file information, and column separator of the data file. R&D personnel set the specific storage path (or storage location) of the file on the storage device (such as a server), and generate configuration information related to data integration based on this information, and persist the configuration information to the database.

[0120] 2. The Scheduler checks the database for newly added configuration information according to a preset scanning cycle. If new configuration information is detected, it reads the configuration information from the database and generates a probe task (Detect) based on it, then submits the probe task to the EventPublisher. Furthermore, the API serves as the entry point for external requests, providing a REST interface for external systems to initiate tasks, query metrics, query logs, and perform other operations. This component is independent of the system's automatic scheduling mechanism, allowing direct interaction between external systems and the system.

[0121] 3. The event publisher will publish the probe tasks to the event queue within the configured time range and interval frequency to drive the probe tasks to check whether the files are ready.

[0122] 4. The Event Looper captures probe tasks from the event queue and obtains the load status of each task processing service Processor to determine the target Processor for handling the probe task. The probe task is then dispatched to the target Processor via RPC.

[0123] 5. The target processor begins processing the probe task, loading the plugin corresponding to the configuration information, i.e., LoadPlugin. Then, the probe task is submitted to the thread pool via the plugin, i.e., Submit to Thread Pool. The probe is executed in the thread pool (Execute Detect), and if the probe data file is ready, the execution result is transmitted to the scheduler (Scheduler).

[0124] 6. When the scheduler responds to the success of the probe task execution, it generates a split task Split based on the relevant data integration configuration information, and submits the split task Split to the event publisher. Then, it repeats steps 3 to 5, except that when repeating step 5, it executes the splitting Execute Spilt in the thread pool. If the probe data file has been split, the execution result is transmitted to the scheduler.

[0125] 7. When the scheduler responds to the successful execution of the split task, it generates a synchronization task Sync based on the relevant data integration configuration information and submits the synchronization task Sync to the event publisher. Then, it repeats steps 3 to 5, except that when repeating step 5, it executes the read file shard in the thread pool. Through multi-threading, it writes these file shards to HDFS, and then the process ends.

[0126] See Figure 4 The diagram illustrates an optional data integration processing method according to an embodiment of the present invention, comprising the following steps:

[0127] S401: Obtain the file format from the data file information, determine the parsing method corresponding to the file format, load the original plugin pre-configured for the parsing method, and initialize the initial information of the original plugin;

[0128] S402: Based on the data source information and data file information, configure the execution method for detecting the existence of data files, the execution method for downloading and detecting the integrity of data files, and the execution method for splitting data files;

[0129] S403: Configure the execution method for clearing intermediate cache data;

[0130] S404: Update the original plugin based on the configuration execution method to obtain a plugin corresponding to the configuration information.

[0131] In the above implementation, the plugin plays a crucial role in the detection, splitting, and synchronization of data files. Detection, splitting, and synchronization are three different task types, but for any file format, the process must first involve a detection task, then a splitting task, and finally a synchronization task to complete the connection. The purpose of the detection task is to determine whether the source file is ready; the purpose of the splitting task is to extract the source file and divide it into an "intermediate format" to facilitate subsequent concurrent extraction; and the synchronization task is to extract these file slices.

[0132] However, the concept of "readiness" differs depending on the source file format. In most cases, the source file name is not fixed; it is extracted daily, and its name typically changes based on date and other business attributes. Furthermore, a source file is not simply a regular TXT file; it might be a compressed archive, and developers may need to extract one or more files from a specific directory within that archive. Compressed archives themselves come in various formats (such as zip, 7z (a high-compression archive format), tgz, etc.), meaning the reading and parsing process for different source files will also differ.

[0133] The implementation of plugins describes these differences. Through plugin design, file format requirements can be masked, allowing different types of source files to be processed separately. Whenever support for a new source file type is needed, only a corresponding plugin needs to be extended. This is illustrated in the plugin design... Figure 5 The differences are abstracted into the following steps. For each new file format, it's sufficient to clearly describe how these steps are performed under the current file format to complete the adaptation. The plugin implementation only needs to focus on these steps and does not need to be aware of the overall integration process.

[0134] 1. Obtain File Format: Retrieve the file format from the configuration information. It's important to note that "file format" is a broad concept; it can be a specific format (e.g., TXT, CSV) or a specific range of data (e.g., the last 50 lines of data in a file). This is because, although there are agreed-upon data limits, the data provided by the upstream provider often exceeds these agreed-upon limits.

[0135] 2. Plugin Initialization: To simplify the processing flow, the parsing method corresponding to the file format is determined, the pre-configured plugins for the parsing method are loaded, and the initial information of the plugins, such as delimiters and encoding formats, is initialized. Some data files may have different formats but the same parsing method; therefore, this solution skips the file format requirement and directly sets the corresponding plugin based on the parsing method. In this way, this solution can more flexibly handle data files with different formats but the same parsing method.

[0136] Suppose we have two data files: one containing sales data in CSV format and the other in TXT format. Although the formats differ, both files are parsed by reading and parsing data row by row, one column at a time. Therefore, this solution loads the same plugin based on the parsing method (reading and parsing row by row, one column at a time), rather than selecting a different plugin based on the file format (CSV or TXT).

[0137] 3. File Probe: Configure the execution method for probing the existence of data files based on data source and data file information. The execution method describes the source file probing steps and under what circumstances the data source file is considered ready.

[0138] 4. Download and Verification: This involves acquiring the source file and verifying its integrity. Configure the execution methods for downloading and checking the integrity of the data file, as well as the execution methods for splitting the data file. Checking the integrity of the data file includes, for example, using algorithms such as MD5 and SHA-1 to calculate the file's checksum and comparing it with known checksums. It can also involve using public-key cryptography to sign and verify the file. For example, it can compare the actual file size with the expected size. Regarding the configuration of the splitting method, you can first extract the necessary data from the data file based on the data acquisition requirements, such as the last 50 lines of data mentioned above.

[0139] 5. Plugin Destruction: This is used as a cleanup step and requires configuring the method for clearing intermediate cache data. This is because plugin execution generates intermediate data that is irrelevant to the execution result. Therefore, to reduce resource consumption after plugin execution, this data needs to be cleared. Cleanup can be achieved through manual deletion, using the operating system's temporary file management mechanism, adding automatic cleanup logic to the plugin code, or setting up a scheduled task to periodically clear intermediate cache data in a specific directory.

[0140] 6. Update the original plugin information based on the above configuration to obtain the plugin corresponding to the configuration information.

[0141] Through the above steps, the plugin design can flexibly handle data files of different formats while ensuring data integrity and security.

[0142] Besides the steps mentioned above, plugin implementation doesn't require attention to details such as how the data source is connected, where the split data is stored, how it's recycled, or how to handle failures. The framework handles all the work, a process completely transparent to the plugin. The plugin's workflow is as follows: Figure 6 As shown:

[0143] 1. Common steps: Build the task context and establish connections with the source database, data warehouse Hive, and storage devices (such as servers).

[0144] 2. File Probe: Based on the data source and data file information in the configuration, probe whether the file is ready. If the file is ready, destroy the plugin and trigger the splitting task; otherwise, destroy the plugin and the task fails.

[0145] 3. File Splitting: If the file is ready, download it and verify its integrity. If verification passes, split the file, destroy the plugin, and trigger a synchronization task. If the file is not ready or verification fails, destroy the plugin and end the task.

[0146] The method provided in the above embodiments only needs to determine the parsing method based on the file format whenever a new file format is added, and then implement the plugin corresponding to the parsing method. This method skips the file format customization step, significantly improves the efficiency of plugin development, and makes plugin products easier to standardize.

[0147] Scheduler and Processor are preferably deployed on different servers. Upon startup, they register themselves on specific nodes using a configured coordination component (such as Zookeeper), recording their IP address (Internet Protocol) and port. The coordination component provides an "observation" mechanism, allowing nodes to observe specific paths. Once new data appears on that path, the observer is notified. Scheduler and Processor observe each other's nodes, discovering each other and obtaining their IP addresses and ports, enabling direct communication.

[0148] When discussing elastic scaling mechanisms, adding new nodes is relatively simple. Whether it's the Scheduler or a Processor, a new node automatically registers with the coordination component after startup. Existing nodes in the cluster immediately discover the new node and establish communication with it. The Scheduler can assign tasks to the new Processor, and the Processor can report task status to the new Scheduler.

[0149] Deleting a node is slightly more complex. For a Scheduler, because it maintains an event publisher in memory, these events become unscheduled once it goes offline. To address this, events (i.e., tasks) are persisted to both the database and memory during registration. Once a Scheduler goes offline, other Schedulers in the cluster will detect this through the coordination component and read the events allocated to that Scheduler from the database, loading them into their own memory, thus completing the Scheduler's scaling down.

[0150] For a Processor, since it executes probing or partitioning tasks, these tasks must stop and migrate if the Processor goes offline. The Scheduler detects the Processor's offline status and marks the tasks on that Processor as failover. Because all tasks are assigned to Processors by the Scheduler, the Scheduler knows the specific tasks running on each Processor. The Scheduler then reschedules all tasks from the offline Processor to other working Processors, thus completing the Processor scaling down.

[0151] Through a coordinated component registration and observation mechanism, the Scheduler and Processor can dynamically discover and communicate. When a new node is added, the cluster automatically expands; when a node is deleted, event persistence and task migration ensure the stability and high availability of the cluster. This mechanism enables elastic scaling of the cluster without downtime.

[0152] This invention proposes a novel method for integrating heterogeneous data files into a Hive data warehouse. Based on a scheduling framework and plug-in design, it abstracts the data file integration process into three task steps: file detection, file splitting, and file integration. Specifically, it achieves the following technical innovations:

[0153] 1. Easy to expand: Whenever a new file format is added, only the parsing method needs to be determined according to the file format, and then a plugin corresponding to the parsing method can be implemented. This method skips the file format customization step, significantly improves the efficiency of secondary development, and makes the product easier to standardize.

[0154] 2. Parallel access: By splitting a file into multiple sub-data files, it supports multi-threaded parallel extraction of the same file, thereby significantly improving the access speed of data files and maximizing resource utilization.

[0155] 3. Elastic scaling: The access cluster can be dynamically scaled up or down. Simply add or delete nodes in the cluster without restarting any services, ensuring the system's flexibility and high availability.

[0156] 4. Retry Strategy: In cases where data source files are not supplied in a timely manner, a threshold for the number of retries is set to support a retry mechanism, thereby ensuring the reliability and stability of data access.

[0157] See Figure 7 The diagram shows a schematic representation of the main modules of a data integration processing apparatus 700 provided in an embodiment of the present invention, including:

[0158] Task module 701 is used to generate target tasks based on configuration information related to data integration and load plugins corresponding to the configuration information; wherein, the configuration information includes data source information and data file information;

[0159] The execution module 702 is used to determine the task type of the target task, and perform data integration operations based on one or more of the data source information and the data file information, according to the execution method pre-configured for the task type in the plugin.

[0160] The judgment module 703 is used to determine the stage of the target task in response to the execution result being successful. If the stage is the last stage, it determines that the data integration is complete; if the stage is not the last stage, it generates the next task according to the configuration information and executes it.

[0161] In the apparatus of the present invention, when the target task is a detection task, the task module 701 is used for:

[0162] In response to the detection of newly added configuration information related to data integration in the database, the configuration information is read from the database to generate a probe task based on the configuration information;

[0163] The execution module 702 is used to: detect in the data source whether there is a data file that matches the data file information.

[0164] The apparatus of the present invention further includes: in response to an execution result of failure, counting the number of failures in executing the probe task, and if the number of failures is less than a preset failure retry threshold, regenerating the target task according to the configuration information.

[0165] In the apparatus of the present invention, when the target task is a segmented task, the task module 701 is used for:

[0166] In response to the success of the probe mission, a segmentation mission is generated based on the configuration information of the relevant data integration.

[0167] The execution module 702 is used for:

[0168] Read the data file that matches the data file information from the data source, divide the data file into multiple sub-data files, and store the multiple sub-data files in the storage location of the storage device according to the storage location in the data file information.

[0169] In the apparatus of the present invention, when the target task is a synchronous task, the task module 701 is used for:

[0170] In response to the success of the splitting task, a synchronization task is generated based on the relevant data integration configuration information;

[0171] The execution module 702 is used for:

[0172] The system employs a multi-threaded parallel execution approach to read the multiple sub-data files from the storage location of the storage device and write the multiple sub-data files into the data warehouse.

[0173] In the apparatus of this invention, the process of configuring the plug-in corresponding to the configuration information includes:

[0174] The file format is obtained from the data file information, the parsing method corresponding to the file format is determined, the original plugin pre-configured for the parsing method is loaded, and the initial information of the original plugin is initialized.

[0175] Based on the data source information and data file information, configure the execution methods for detecting the existence of data files, downloading and checking the integrity of data files, and splitting data files; and

[0176] Configure the execution method for clearing intermediate cache data;

[0177] The original plugin is updated based on the configuration execution method to obtain a plugin corresponding to the configuration information.

[0178] In the apparatus of this invention, the task module 701 is used for:

[0179] Identify the target task processing service, transmit the target task to the target task processing service, and load the plugin corresponding to the configuration information through the target task processing service.

[0180] In the apparatus of this invention, the scheduler includes an event publisher and an event looper, and the apparatus further includes:

[0181] The target task is submitted to the event publisher, which then publishes the target task to the event queue according to the preset time range and interval frequency.

[0182] The target task is captured from the event queue through the event looper, and the current load of each task processing service is obtained to determine the target task processing service based on the load.

[0183] The apparatus for implementing this invention further includes an elastic expansion / contraction module, used for:

[0184] The target task is tagged with the scheduler and persisted to the database, and simultaneously written to memory; in response to an exception of the scheduler, other schedulers retrieve the target tasks tagged with the scheduler node that have not been processed from the database and write the target tasks to memory.

[0185] In response to an anomaly in the target task processing service, an anomaly information is transmitted to the scheduler, causing the scheduler to mark the target task as fault-transferred and to re-determine the target task processing service so as to transfer the target task to the new target task processing service.

[0186] Furthermore, the specific implementation details of the device described in the embodiments of the present invention have been described in detail in the above-described method, so the details will not be repeated here.

[0187] Figure 8 An exemplary system architecture 800 to which embodiments of the present invention can be applied is shown, including terminal devices 801, 802, 803, network 804, and server 805 (only an example).

[0188] Terminal devices 801, 802, and 803 can be various electronic devices with displays and web browsing capabilities, and can be equipped with various communication client applications. Users can use terminal devices 801, 802, and 803 to interact with server 805 via network 804 to receive or send messages, etc.

[0189] Network 804 is a medium used to provide a communication link between terminal devices 801, 802, 803 and server 805. Network 804 can include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0190] Server 805 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 801, 802, and 803 (this is just an example). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information and product information—this is just an example) to the terminal devices. It should be noted that the method provided in this embodiment of the invention is generally executed by server 805, and correspondingly, the apparatus is generally set in server 805.

[0191] It should be understood that Figure 8 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0192] The following is for reference. Figure 9 It shows a schematic diagram of the structure of a computer system 900 suitable for implementing a terminal device of the present invention. Figure 9 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0193] like Figure 9As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 902 or programs loaded from storage section 908 into random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the system 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0194] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0195] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this invention.

[0196] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0198] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a task module, an execution module, and a judgment module. The names of these modules do not necessarily limit the module itself; for example, the judgment module can also be described as a "task generation or termination module."

[0199] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform any of the data integration processing methods described above.

[0200] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the data integration processing method in the embodiments of the present invention.

[0201] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data integration and processing method, characterized in that, include: The target task is generated based on the configuration information for data integration, and the plugin corresponding to the configuration information is loaded; wherein, the configuration information includes data source information and data file information; The task type of the target task is determined, and the data integration operation is performed based on one or more of the data source information and data file information, according to the execution method pre-configured for the task type in the plugin. In response to a successful execution result, the current stage of the target task is determined. If it is the last stage, data integration is confirmed to be complete. If it is not the last stage, the next task is generated and executed based on the configuration information.

2. The method according to claim 1, characterized in that, When the target task is a probe task, generating the target task based on the configuration information related to data integration includes: In response to the detection of newly added configuration information related to data integration in the database, the configuration information is read from the database to generate a probe task based on the configuration information; The step of performing data integration operations based on one or more of the data source information and the data file information includes: detecting in the data source whether there is a data file that matches the data file information.

3. The method according to claim 2, characterized in that, The method further includes: In response to an execution failure, the number of failed attempts to perform the probe task is counted. If the number of failed attempts is less than the preset threshold for retrying, the target task is regenerated based on the configuration information.

4. The method according to any one of claims 1-3, characterized in that, When the target task is a segmented task, generating the target task based on the configuration information related to data integration includes: In response to the success of the probe mission, a segmentation mission is generated based on the configuration information of the relevant data integration. The step of performing data integration operations based on one or more of the data source information and the data file information includes: Read the data file that matches the data file information from the data source, divide the data file into multiple sub-data files, and store the multiple sub-data files in the storage location of the storage device according to the storage location in the data file information.

5. The method according to claim 4, characterized in that, When the target task is a synchronous task, generating the target task based on the configuration information related to data integration includes: In response to the success of the split task execution, a synchronization task is generated based on the relevant data integration configuration information; The step of performing data integration operations based on one or more of the data source information and the data file information includes: The system employs a multi-threaded parallel execution approach to read the multiple sub-data files from the storage location of the storage device and write the multiple sub-data files into the data warehouse.

6. The method according to claim 1, characterized in that, The process of configuring the plugin corresponding to the configuration information includes: The file format is obtained from the data file information, the parsing method corresponding to the file format is determined, the original plugin pre-configured for the parsing method is loaded, and the initial information of the original plugin is initialized. Based on the data source information and data file information, configure the execution methods for detecting the existence of data files, downloading and checking the integrity of data files, and splitting data files; and Configure the execution method for clearing intermediate cache data; The original plugin is updated based on the configuration execution method to obtain a plugin corresponding to the configuration information.

7. The method according to claim 1 or 6, characterized in that, The loading of the plugin corresponding to the configuration information includes: Identify the target task processing service, transmit the target task to the target task processing service, and load the plugin corresponding to the configuration information through the target task processing service.

8. The method according to claim 7, characterized in that, The scheduler includes an event publisher and an event looper, and the method further includes: The target task is submitted to the event publisher, which then publishes the target task to the event queue according to the preset time range and interval frequency. The target task is captured from the event queue through the event looper, and the current load of each task processing service is obtained to determine the target task processing service based on the load.

9. The method according to claim 8, characterized in that, The method also includes one or more of the following cases: The target task is tagged with the scheduler and persisted to the database, and simultaneously written to memory; in response to an exception of the scheduler, other schedulers retrieve the target tasks tagged with the scheduler node that have not been processed from the database and write the target tasks to memory. In response to an anomaly in the target task processing service, an anomaly information is transmitted to the scheduler, causing the scheduler to mark the target task as fault-transferred and to re-determine the target task processing service so as to transfer the target task to the new target task processing service.

10. A data integration and processing apparatus, characterized in that, include: The task module is used to generate target tasks based on configuration information related to data integration and load plugins corresponding to the configuration information; wherein, the configuration information includes data source information and data file information; The execution module is used to determine the task type of the target task, and perform data integration operations based on one or more of the data source information and data file information, according to the execution method pre-configured for the task type in the plugin. The judgment module is used to determine the stage of the target task in response to a successful execution result. If the stage is the last stage, it determines that data integration is complete; if the stage is not the last stage, it generates the next task based on the configuration information and executes it.

11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-9.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-9.

Images