Data processing method, apparatus and computer readable storage medium

Abstract

The application discloses a data processing method, device and computer readable storage medium. The data processing method comprises the following steps: obtaining service data of a target service, configuring processing tasks corresponding to processing nodes of the target service according to the service data; configuring an auditing task and a repair task of the target service according to to-be-audited data of the target task; generating a task set of the target service according to a configuration sequence of the processing task, the auditing task and the repair task, so as to sequentially execute each task in the task set when the target service is executed. The application realizes unitization, splitting and combination of service data by grouping multiple tasks into a complete service, thereby improving the accuracy of system service data and processing efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus and computer-readable storage medium. Background Technology

[0002] Currently, to meet the needs of business development and daily operations, telecommunications operators have a large amount of business data across various systems that requires accuracy assurance. For business data worth hundreds of billions or even trillions of records, there has never been a mature and systematic management solution. The traditional management model relies on maintenance personnel understanding the business processes and then manually analyzing and repairing the business data. However, existing methods of manual data processing are inefficient and cannot meet production demands. Summary of the Invention

[0003] This application provides a data processing method, apparatus, and computer-readable storage medium, aiming to solve the problem of low data processing efficiency caused by relying on manual data processing.

[0004] To achieve the above objectives, this application provides a data processing method, the method comprising:

[0005] Obtain the business data of the target business, and configure the processing tasks corresponding to each processing node of the target business based on the business data;

[0006] Configure the audit task and repair task of the target business according to the data to be audited for the target task;

[0007] The task set for the target service is generated according to the configuration order of the processing task, the audit task, and the repair task, so that each task in the task set is executed sequentially when the target service is executed.

[0008] Optionally, the step of configuring the processing tasks corresponding to each processing node of the target service based on the service data includes:

[0009] Obtain the business attribute information of the target business based on the business data;

[0010] Configure the processing tasks corresponding to each processing node of the target service according to the business attribute information.

[0011] Optionally, the step of configuring the audit task and remediation task of the target business based on the data to be audited for the target task includes:

[0012] The audit template for the audit task is determined based on the data to be audited for the target task.

[0013] Configure the audit task for the target business according to the audit template;

[0014] Obtain the audit data corresponding to the audit task, and configure the repair task based on the audit data.

[0015] Optionally, after the step of generating the task set of the target service according to the configuration order of the processing task, the audit task, and the repair task, the method further includes:

[0016] The processing tasks in the task set are called sequentially to process the business data of the target business and the processing result of each processing task is obtained.

[0017] The audit results are obtained by calling the audit tasks in the task set to audit the processing results of each processing task.

[0018] When the audit result is inconsistent with the first preset result, the repair task in the task set is invoked to perform the repair operation.

[0019] Optionally, the step of calling the audit tasks in the task set to audit the processing results of each processing task and obtaining the audit result includes:

[0020] The audit task in the task set is invoked to audit each processing result and its corresponding second preset result to obtain the audit result;

[0021] Alternatively, based on the processing result, obtain at least two data reports corresponding to the same business data, and call the audit task in the task set to audit a preset number of the data reports to obtain the audit result.

[0022] Optionally, after the step of generating the task set of the target service according to the configuration order of the processing task, the audit task, and the repair task, the method further includes:

[0023] Obtain the task configuration information of the target service;

[0024] The number of shards for each task in the task set is obtained based on the task configuration information.

[0025] Each task is divided into task fragments according to the number of fragments, and each task fragment is distributed to an executor so that the executor executes each task fragment.

[0026] Optionally, before the step of distributing each of the task fragments to the executor, the following steps are included:

[0027] Obtain the container scheduling interface of the container orchestration platform, and start the container orchestration platform according to the container scheduling interface;

[0028] The container orchestration platform sends the number of containers required to process each of the tasks, wherein the container orchestration platform starts the containers according to the number of containers and executes each task fragment through the executor corresponding to the container.

[0029] Furthermore, to achieve the above objectives, this application also provides a data processing apparatus, which includes a first configuration module, a second configuration module, and a processing module, wherein:

[0030] The first configuration module is used to acquire business data of the target business and configure the processing tasks corresponding to each processing node of the target business according to the business data.

[0031] The second configuration module is used to configure the audit task and repair task of the target business according to the data to be audited of the target task;

[0032] The processing module is configured to generate a task set for the target service according to the configuration order of the processing task, the audit task, and the repair task, so as to execute each task in the task set sequentially when the target service is executed.

[0033] In addition, to achieve the above objectives, this application also provides a data processing apparatus, which includes a memory, a processor, and a data processing program stored in the memory and running on the processor. When the processor executes the data processing program, it implements the steps of the data processing method described above.

[0034] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a data processing program, which, when executed by a processor, implements the steps of the data processing method described above.

[0035] This application proposes a data processing method that involves acquiring business data of a target business, configuring processing tasks corresponding to each processing node of the target business based on the business data, configuring audit tasks and remediation tasks of the target business based on the data to be audited, and generating a task set for the target business according to the configuration order of processing tasks, audit tasks, and remediation tasks, so that each task in the task set is executed sequentially when the target business is executed. This application achieves the unitized splitting and combination of business data by grouping multiple tasks into a complete business process, thereby improving the accuracy and processing efficiency of the system's business data. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of this application;

[0037] Figure 2 This is a flowchart illustrating the first embodiment of the data processing method of this application;

[0038] Figure 3 This is a flowchart illustrating the process after the step of generating the task set of the target business according to the configuration order of the processing task, the audit task, and the repair task in the data processing method of this application.

[0039] Figure 4 This is another flowchart illustrating the process following the step of generating the task set of the target business according to the configuration order of the processing task, the audit task, and the repair task in the data processing method of this application.

[0040] Figure 5 This is a schematic diagram illustrating the working principle of the task orchestrator in this application;

[0041] Figure 6 This is a schematic diagram of the overall architecture of the task scheduling platform in this application;

[0042] Figure 7 This is a schematic diagram of the data processing device of this application.

[0043] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0045] The main solution of this application embodiment is: to obtain business data of the target business, configure processing tasks corresponding to each processing node of the target business according to the business data; configure audit tasks and repair tasks of the target business according to the data to be audited of the target task; generate a task set of the target business according to the configuration order of the processing tasks, the audit tasks and the repair tasks, so as to execute each task in the task set in sequence when the target business is executed.

[0046] Traditional management relies on maintenance personnel who understand the business processes and then manually analyze and repair business data. However, existing methods of manual data processing are inefficient and cannot meet production demands.

[0047] This application acquires the business data of the target business, configures the processing tasks corresponding to each processing node of the target business based on the business data, configures the audit tasks and repair tasks of the target business based on the data to be audited, and generates a task set of the target business according to the configuration order of the processing tasks, audit tasks, and repair tasks, so that each task in the task set is executed sequentially when the target business is executed. This application realizes the unitized splitting and combination of business data by combining multiple tasks into a complete business, thereby improving the accuracy and processing efficiency of the system's business data.

[0048] like Figure 1 As shown, Figure 1 This is a schematic diagram of the terminal device structure of the hardware operating environment involved in the embodiments of this application.

[0049] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0050] Those skilled in the art will understand that Figure 1 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0051] like Figure 1 As shown, the memory 1005, which is a computer-readable storage medium, may include a data processing program.

[0052] exist Figure 1 In the terminal device shown, the network interface 1004 is mainly used for data communication with the backend server; the user interface 1003 is mainly used for data communication with the client (user terminal); the processor 1001 can be used to call the data processing program in the memory 1005 and perform the following operations:

[0053] Obtain the business data of the target business, and configure the processing tasks corresponding to each processing node of the target business based on the business data;

[0054] Configure the audit task and repair task of the target business according to the data to be audited for the target task;

[0055] The task set for the target service is generated according to the configuration order of the processing task, the audit task, and the repair task, so that each task in the task set is executed sequentially when the target service is executed.

[0056] refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data processing method of this application.

[0057] This application provides a data processing method. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0058] The data processing method includes the following steps:

[0059] Step S10: Obtain the business data of the target business, and configure the processing tasks corresponding to each processing node of the target business according to the business data;

[0060] It should be noted that each business has different process nodes (i.e., processing nodes), and each process node can be regarded as a data model (i.e. processing task). Combining the various data models can determine the data model required by the business.

[0061] The data processing device collects business data of the target business through a data acquisition module and stores the collected business data in a database. The data acquisition module can be configured with different business types, acquisition frequency, and data collection periods to achieve multi-combination and multi-scenario data acquisition, thereby meeting actual business self-healing needs. Then, the data processing device configures the processing business corresponding to each processing node of the target task based on the collected business data. In one embodiment, assuming the target business is an account opening service, the account opening service includes three processing nodes: number, package, and user information. Each processing node corresponds to a processing task (i.e., a data model). Each processing node has a data embedding point used to record and collect the business data of the processing node. The data processing device retrieves the business data of each processing node of the account opening service from the database, then obtains the business attribute information corresponding to each business data, and configures the processing business corresponding to each processing node based on this business attribute information; that is, it automatically generates the data model corresponding to each processing node based on the business attribute information. For example, the attributes of a number include: whether the number is a preferred number, its location, etc.; the attributes of a service plan include: plan type (including individual plans, family plans, 4G plans, and 5G plans, etc.), plan level, plan cost, etc.; and the attributes of user information include the user's ID number, user name, and whether they are a new user, etc. Then, based on the above business attribute information, processing tasks for the number, the service plan, and the user information are configured respectively.

[0062] Step S20: Configure the audit task and repair task of the target business according to the data to be audited of the target task;

[0063] It should be noted that generating processing tasks based solely on the collected business data is insufficient for data repair. Therefore, it is necessary to generate audit tasks (i.e., audit data models) and repair tasks (i.e., repair data models) corresponding to the target business. Then, the business data is audited and repaired based on the audit tasks and repair tasks to improve the accuracy of the business data.

[0064] The data processing device in this application interfaces with an auditing platform. The auditing platform has already identified the target business data to be audited (i.e., data to be reviewed), such as monetary data and intermediate status data. Therefore, the data processing device can directly obtain the data to be reviewed from the auditing platform. Then, based on the data to be reviewed, a general template for the audit task is determined (including a general template for table auditing, a general template for document auditing, etc.). The audit task is configured according to the general template. For example, assuming the audit task is a table audit, the audit task is configured based on the table entries set in the general template for table auditing. Simultaneously, since data auditing and data repair are corresponding, repair tasks can be configured based on the audit data in the audit task. For example, assuming the audit data is monetary data, an monetary repair task is configured.

[0065] Step S30: Generate a task set for the target service according to the configuration order of the processing task, the audit task, and the repair task, so that each task in the task set is executed sequentially when the target service is executed.

[0066] In this embodiment, the configuration order of processing tasks, audit tasks, and repair tasks is obtained, and then a task set for the target task is generated according to this configuration order. When the target task is executed, each task in the task set is executed sequentially. For example, assuming the target business is an account opening business, the account opening business includes three processing tasks, whose corresponding configuration order is processing task 1, processing task 2, and processing task 3. Meanwhile, the audit task is configured before the repair task. Therefore, the task set is generated according to the configuration order: processing task 1, processing task 2, processing task 3, audit task, and repair task.

[0067] In this embodiment, business data of the target business is acquired, and processing tasks corresponding to each processing node of the target business are configured according to the business data. Audit tasks and repair tasks of the target business are configured according to the data to be audited in the target task. A task set of the target business is generated according to the configuration order of processing tasks, audit tasks, and repair tasks, so that each task in the task set is executed sequentially when the target business is executed. This application achieves the unitized splitting and combination of business data by combining multiple tasks into a complete business, thereby improving the accuracy and processing efficiency of the system's business data.

[0068] Further, refer to Figure 3 The second embodiment of the data processing method of this application is proposed.

[0069] The difference between the second embodiment of the data processing method and the first embodiment of the data processing method is that, after the step of generating the task set of the target business according to the configuration order of the processing task, the audit task, and the repair task, the method includes:

[0070] Step S31: In the order of the task set, each processing task is called sequentially to process the business data of the target business and obtain the processing result of each processing task.

[0071] Step S32: Invoke the audit tasks in the task set to audit the processing results of each processing task and obtain the audit results.

[0072] Step S33: When the audit result is inconsistent with the first preset result, the repair task in the task set is invoked to perform the repair operation.

[0073] It's important to note that large CRM (Customer Relationship Management) systems typically handle complex data. Therefore, the traditional approach is to separate auditing and remediation. An auditing system performs data checks, identifies abnormal data, and then performs specific data remediation. This results in a long lifespan for abnormal data. This application, however, connects the two complex processes of auditing and remediation through configuration, enabling real-time auditing and remediation of abnormal data. This achieves complete decoupling between stable system operation and code logic, ensuring business continuity and external transparency.

[0074] In this embodiment, each processing task in the task set is sequentially invoked to process the business data of the target business, and the processing result of each processing task is obtained. Then, the audit task in the task set is invoked to audit the processing results of each processing task to obtain the audit result. Here, data audit refers to realizing the integrity and consistency check of data to improve data quality. At the same time, data audit is a complete data quality control chain from data collection, preprocessing, comparison, analysis, early warning, notification, and problem repair. If an audit anomaly occurs, that is, when the audit result is inconsistent with the first preset result, the business data to be repaired is obtained, and the repair task is invoked to repair the business data to be repaired.

[0075] In one embodiment, this application includes two audit methods. One method is to compare the original collected data. For example, the collected business data is presented in data report 1, data report 2, and data report 3. Since the three data reports are all generated based on the same business data, theoretically, the data in a certain field of the three data reports should be consistent. If they are inconsistent, it indicates that there is a problem with the business data. For example, two data reports (such as data report 1 and data report 3) are selected for comparison. If the data corresponding to field A in data report 1 and data report 3 is consistent, the audit is successful. If the data corresponding to field A in data report 1 and data report 3 is inconsistent, the audit is abnormal. At this time, data report 1 or data report 3 needs to be compared with other data reports to determine the correct data corresponding to field A. Then, based on the repair task, the different data in data report 1 or data report 3 is repaired to the correct data.

[0076] Another approach is to compare the collected data with preset data. For example, the account opening process includes process node 1, process node 2, and process node 3. In process node 1, the phone number is in a pre-occupancy state, while in process node 3, the phone number is in a fully occupied state. Suppose the current audit result shows that the phone number is in a pre-occupancy state in both process node 1 and process node 3. This indicates an audit anomaly, requiring a repair task to be invoked to change the phone number's occupancy state in process node 3 to a fully occupied state.

[0077] This embodiment connects the two complex processes of auditing and repair through configuration, enabling real-time auditing and repair of abnormal data. This achieves stable system operation and complete decoupling of code logic, ensuring business continuity and external non-perception.

[0078] Further, refer to Figure 4 The third embodiment of the data processing method of this application is proposed.

[0079] The difference between the third embodiment of the data processing method and the second embodiment of the data processing method is that, after the step of generating the task set of the target business according to the configuration order of the processing task, the audit task, and the repair task, the method includes:

[0080] Step S34: Obtain the task configuration information of the target service;

[0081] Step S35: Obtain the number of shards for each task in the task set based on the task configuration information;

[0082] Step S36: Divide each task into fragments according to the number of fragments to obtain each task fragment, and distribute each task fragment to the executor so that the executor executes each task fragment.

[0083] It should be noted that repairing business data takes time. If the repair time is too long, users will also have to wait longer, thus degrading the user experience. To address this issue, task sharding and distribution are necessary to improve the efficiency of utilizing executor cluster resources, thereby shortening task processing time and ultimately improving the user experience.

[0084] In this embodiment, the scheduling controller reads task configuration information from the Redis cache. This task configuration information includes the task's data model combination information and system configuration information. The system configuration information includes the sharding information for each task, such as the number of shards and the sharding duration. Then, the scheduling controller obtains the pre-configured number of shards for the task based on the system configuration information and shards the task according to this number of shards to obtain multiple task shards (i.e., multiple independent subtasks). Task sharding improves task execution efficiency and overall resource utilization by dividing a single task into multiple independent subtasks according to specific logic and distributing these subtasks to different nodes for execution. For example, task A needs to process 200,000 data entries, and the business logic processing time for each data entry is 0.1 seconds. For a normal task, processing it with only one thread might take 5 to 6 hours. However, if the 200,000 data entries are evenly distributed among different executors in the cluster for simultaneous processing, the processing time will be greatly reduced, and cluster resources can be fully utilized.

[0085] After task sharding is completed, the scheduling controller writes the task shards to the Redis cache, then calls the container scheduling interface provided by Marathon. Through the container startup interface, it notifies Marathon of the number of containers to start, and Marathon starts the corresponding containers. The executor starts automatically along with the containers. The number of containers required for each task is determined by the system, for example, based on container capacity and the amount of data to be processed. After starting, the executor registers its status information in the task queue and reports it to the Redis cache. Simultaneously, it connects to the task queue to preempt tasks. Each executor is only allowed to preempt one task shard. It executes the corresponding business processing based on the shard information parameters. After the executor completes the task shard processing, the scheduling controller schedules Marathon to release the container memory. Furthermore, the scheduling controller periodically checks the status information reported by the task executors. If the reported status information indicates that an executor is abnormal, the corresponding task record needs to be redistributed for reprocessing.

[0086] It's worth noting that Redis is an in-memory, persistent key-value NoSQL database where each key and value is represented as an object, featuring diverse data types, persistence, and master-slave synchronization. Marathon is a container orchestration platform running on a data center operating system (DC / OS) and Mesos (the kernel of a distributed system, an open-source distributed resource management framework under Apache).

[0087] In this embodiment, by fragmenting the task and distributing the task fragments to different executors for processing, the task execution efficiency is improved, and the overall resource utilization efficiency is enhanced.

[0088] To better illustrate the data processing method of this application, refer to Figures 5 to 6 , Figure 5 This is a schematic diagram illustrating the working principle of the task orchestrator in this application. Figure 6 This is a schematic diagram of the overall architecture of the task scheduling platform in this application.

[0089] In this embodiment, the data processing device includes a data model module (DM), a data acquisition module (DAM), a task processing module (TPM), a schedule controller module (SCM), and a task executor module (TEM). Wherein:

[0090] Data Model Module (DM)

[0091] The data model module includes configurations for business types, business attributes, native business data, and resulting business data. This configuration information is primarily stored in a physical database. To improve task generation efficiency, this configuration information is cached via aiCache, providing internal and external query services (if the business system data is not very large, the physical database can also meet the needs). The data model is the foundation of the entire real-time self-healing mechanism for business data. It runs through the entire data self-healing process and also affects the accuracy of the self-healed business data. The data model module allows for the automatic combination of N factors (the lowest-level data model) of the same business data model through business attribute configuration, forming data model units for collection, auditing, and repair, ultimately achieving self-healing of business data.

[0092] Data Acquisition Module (DAM)

[0093] Data acquisition involves collecting data from business systems into a target database. The acquisition objects and conditions are based on audit data model units, and the acquisition prepares for data auditing. The data acquisition module can be configured with the acquisition business type, acquisition time frequency, and data acquisition cycle to achieve data acquisition in multiple combinations and scenarios, meeting actual business self-healing needs.

[0094] Task Processing Module (TPM)

[0095] The main function of the task orchestrator module is to freely combine data models and production data collection according to business dimensions to form a single task. This task also needs to compare the collected data with the data model to identify discrepancies and correct them. Both the data model and production data collection are implemented through configurable scripts on this device. Each data model is a business data factor that can be reused. N business data factors combine to form a complete set of business data, realizing the unitized splitting and combination of business data. This solves the current problem of difficult real-time data repair due to the complexity and high coupling of business data in telecommunications operators.

[0096] Based on business attribute configuration, N tasks are combined to form a task work set required by the business. The task work set is mainly divided into audit tasks and repair tasks. Detailed information of the data model factors under each task is written to the task queue (Redis cache). The task queue supports settings such as task priority and number of threads. Among them, the setting of task set priority can completely control the logical order of data collection, auditing, and self-healing repair, making it fully configurable.

[0097] Schedule Controller Module (SCM)

[0098] The core functions of the scheduling controller include two parts: task fragmentation and distribution, and exception management.

[0099] The main processes for task sharding and distribution include obtaining task details, determining task distribution status, and handling interface calls between the distributed task and Marathon.

[0100] The task distribution thread periodically scans the task shard details list, retrieves undistributed task records, distributes them, and writes them to the Redis cache. Simultaneously, it calls the container scheduling interface provided by Marathon, notifying Marathon of the number of containers to start. Then, Marathon starts the corresponding container instances for subsequent processing.

[0101] Anomaly scheduling management primarily involves managing task execution anomalies. It is responsible for periodically checking the status reported by task executors, and if an executor is found to be in an abnormal state, the corresponding task record needs to be redistributed for reprocessing. The executor status monitoring thread reads all registered executor instances from the Redis cache, monitors their status, and deletes instances with abnormal status.

[0102] Task Executeor Module (TEM)

[0103] The core functions of the task executor module include executor registration / reporting and task execution. Executors are primarily deployed in containers using image creation. Marathon receives requests to allocate container resources and launches a batch of task shard containers for execution. Simultaneously, the executor also starts automatically. After starting, the executor registers with the task queue, reporting its status information to the Redis cache. It also connects to the task queue to preempt tasks; each executor is only allowed to preempt one task shard and executes the corresponding business logic based on the shard information parameters.

[0104] This embodiment uses business data modeling to automatically combine N data model factors into audit data model units and repair data model units. The audit result set automatically triggers data self-healing, enabling business data to self-heal in real time according to a standardized data model, thereby improving the accuracy of system business data and customer perception. Secondly, compared with traditional data processing methods, the data processing method of this application has the following technical advantages:

[0105] This application significantly reduces the lifespan of anomalous data. Large CRM systems typically handle complex data; therefore, the traditional approach is to separate auditing and remediation. An auditing system audits the data, identifies anomalous data, and then performs specialized data remediation. This results in a long lifespan for anomalous data. However, this application integrates the two complex processes of auditing and remediation through configuration, enabling real-time auditing and remediation of anomalous data. This achieves complete decoupling between stable system operations and code logic, ensuring business continuity and external transparency.

[0106] The application offers high flexibility. Traditional self-maintenance processes for data auditing and repair require custom development, especially for large-scale data repair and cutover, which necessitates separate development. This application, however, allows for the unitization of business processes, dividing them into data models, and arbitrarily combining them with collected data to create individual data auditing and repair tasks. Functionally, this achieves configurability, enhances flexibility, and improves the utilization rate of data models.

[0107] It boasts strong scalability. Due to the highly complex business logic, traditional operation and maintenance platforms or tools are deployed and implemented to audit and repair business data specifically for the business logic. This results in strong coupling with the business logic and a high degree of customization. In contrast, this application adopts a configuration + SQL script approach, where audit and repair tasks are assembled through a unitized data model, thereby improving business scalability.

[0108] The self-closing data accuracy self-maintenance model of auditing + self-healing through repair unifies the management of various business scenarios through platform management, realizing centralized management. The platform's logs can trace the source of problems, transforming the traditional black screen operation into a white screen operation, making operation and maintenance work more data-driven and tool-based.

[0109] Furthermore, this application also provides a data processing apparatus, which includes a memory, a processor, and a data processing program stored in the memory and running on the processor. The apparatus acquires business data of a target business, configures processing tasks corresponding to each processing node of the target business based on the business data, configures audit tasks and repair tasks of the target business based on the data to be audited in the target tasks, and generates a task set for the target business according to the configuration order of the processing tasks, audit tasks, and repair tasks, so that each task in the task set is executed sequentially when the target business is executed. This application achieves the unitized splitting and combination of business data by combining multiple tasks into a complete business, thereby improving the accuracy and processing efficiency of the system's business data.

[0110] Further, refer to Figure 7 The data processing device 100 includes a first configuration module 10, a second configuration module 20, and a processing module 30, wherein:

[0111] The first configuration module 10 is used to acquire business data of the target business and configure the processing tasks corresponding to each processing node of the target business according to the business data;

[0112] The second configuration module 20 is used to configure the audit task and repair task of the target business according to the audit data of the target task;

[0113] The processing module 30 is used to generate a task set for the target service according to the configuration order of the processing task, the audit task, and the repair task, so as to execute each task in the task set sequentially when the target service is executed.

[0114] Furthermore, the first configuration module 10 includes an acquisition unit and a first configuration unit;

[0115] The acquisition unit is used to acquire the business attribute information of the target business based on the business data;

[0116] The first configuration unit is used to configure the processing tasks corresponding to each processing node of the target service according to the service attribute information.

[0117] Furthermore, the second configuration module 20 includes a determining unit and a second configuration unit;

[0118] The determining unit is used to determine the audit template of the audit task based on the audit data to be audited for the target task;

[0119] The second configuration unit is used to configure the audit task of the target business according to the audit template;

[0120] The second configuration unit is further configured to acquire audit data corresponding to the audit task and configure the repair task according to the audit data.

[0121] Furthermore, the processing module 30 includes a calling unit, an auditing unit, and a repair unit;

[0122] The calling unit is used to sequentially call each processing task in the task set to process the business data of the target business and obtain the processing result of each processing task.

[0123] The audit unit is used to call the audit tasks in the task set to audit the processing results of each processing task and obtain the audit results.

[0124] The repair unit is used to call the repair task in the task set to perform a repair operation when the audit result is inconsistent with the first preset result.

[0125] Furthermore, the audit unit includes audit sub-units;

[0126] The audit subunit is used to call the audit tasks in the task set to audit each processing result and the corresponding second preset result to obtain the audit result;

[0127] The audit subunit is used to obtain at least two data reports corresponding to the same business data based on the processing result, and to call the audit tasks in the task set to audit a preset number of the data reports to obtain the audit result.

[0128] Furthermore, the processing module 30 also includes an acquisition unit and a fragmentation unit;

[0129] The acquisition unit is used to acquire the task configuration information of the target service;

[0130] The acquisition unit is further configured to acquire the number of fragments of each task in the task set based on the task configuration information;

[0131] The sharding unit is used to shard each task according to the number of shards to obtain each task shard, and distribute each task shard to the executor so that the executor executes each task shard.

[0132] Furthermore, the fragmentation unit includes a startup unit and a sending unit;

[0133] The startup unit is used to obtain the container scheduling interface of the container orchestration platform and start the container orchestration platform according to the container scheduling interface;

[0134] The sending unit is used to send the number of containers required to process each of the tasks to the container orchestration platform, wherein the container orchestration platform starts the containers according to the number of containers and executes each task fragment through the executor corresponding to the container.

[0135] The implementation of the functions of each module of the data processing device described above is similar to the process in the above method embodiment, and will not be described in detail here.

[0136] In addition, this application also provides a computer-readable storage medium storing a data processing method program, which, when executed by a processor, implements the steps of the data processing method described above.

[0137] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0140] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0141] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0142] Although alternative embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the alternative embodiments as well as all changes and modifications falling within the scope of this application.

[0143] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the business data of the target business, and configure the processing tasks corresponding to each processing node of the target business based on the business data; The audit task and the repair task of the target business are configured according to the pending audit data of the processing task, wherein the audit template of the audit task is determined according to the pending audit data of the processing task; the audit task of the target business is configured according to the audit template; the audit data corresponding to the audit task is obtained, and the repair task is configured according to the audit data. The task set of the target business is generated according to the configuration order of the processing task, the audit task, and the repair task, so that each task in the task set is executed sequentially when the target business is executed. The processing task, the audit task, and the repair task are each composed of at least one reusable data model factor, and the data model factor is a minimum underlying data model pre-configured according to the business attributes.

2. The data processing method as described in claim 1, characterized in that, The step of configuring the processing tasks corresponding to each processing node of the target service based on the service data includes: Obtain the business attribute information of the target business based on the business data; Configure the processing tasks corresponding to each processing node of the target service according to the business attribute information.

3. The data processing method as described in claim 1, characterized in that, After the step of generating the task set for the target service according to the configuration order of the processing task, the audit task, and the repair task, the following steps are included: The processing tasks in the task set are called sequentially to process the business data of the target business and the processing result of each processing task is obtained. The audit results are obtained by calling the audit tasks in the task set to audit the processing results of each processing task. When the audit result is inconsistent with the first preset result, the repair task in the task set is invoked to perform the repair operation.

4. The data processing method as described in claim 3, characterized in that, The step of calling the audit tasks in the task set to audit the processing results of each processing task and obtaining the audit results includes: The audit task in the task set is invoked to audit each processing result and its corresponding second preset result to obtain the audit result; Alternatively, based on the processing result, at least two data reports corresponding to the same business data are obtained, and the audit tasks in the task set are invoked to audit a preset number of the data reports to obtain the audit result.

5. The data processing method as described in claim 1, characterized in that, After the step of generating the task set for the target service according to the configuration order of the processing task, the audit task, and the repair task, the following steps are included: Obtain the task configuration information of the target service; The number of shards for each task in the task set is obtained based on the task configuration information. Each task is divided into task fragments according to the number of fragments, and each task fragment is distributed to an executor so that the executor executes each task fragment.

6. The data processing method as described in claim 5, characterized in that, Prior to the step of distributing each of the task fragments to the executor, the following steps are included: Obtain the container scheduling interface of the container orchestration platform, and start the container orchestration platform according to the container scheduling interface; The container orchestration platform sends the number of containers required to process each of the tasks, wherein the container orchestration platform starts the containers according to the number of containers and executes each task fragment through the executor corresponding to the container.

7. A data processing apparatus, characterized in that, The data processing device includes a first configuration module, a second configuration module, and a processing module, wherein: The first configuration module is used to acquire business data of the target business and configure the processing tasks corresponding to each processing node of the target business according to the business data. The second configuration module is used to configure the audit task and the repair task of the target business according to the audit data to be audited of the processing task, wherein: the audit template of the audit task is determined according to the audit data to be audited of the processing task; the audit task of the target business is configured according to the audit template; the audit data corresponding to the audit task is obtained; and the repair task is configured according to the audit data. The processing module is configured to generate a task set for the target business according to the configuration order of the processing task, the audit task, and the repair task, so that each task in the task set is executed sequentially when the target business is executed. The processing task, the audit task, and the repair task are each composed of at least one reusable data model factor, and the data model factor is a minimum underlying data model pre-configured according to the business attributes.

8. A data processing apparatus, characterized in that, The apparatus includes a memory, a processor, and a data processing program stored in the memory and running on the processor, wherein the processor, when executing the data processing program, implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data processing program, which, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

Images