Table operation method, apparatus and device for business data, and readable storage medium
By batch processing and parallel task operation of business data, the inefficiency problem of traditional methods under large-scale data and complex logic is solved, and efficient business data table operations are achieved.
Patent Information
- Application Number
- CN202411152439.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-03
AI Technical Summary
Traditional business data table operation methods are slow to process and inefficient when facing large-scale data and complex business logic.
By processing the business table data to be processed in batches, multiple batches of business sub-data are generated, and task configuration information is generated based on row data, location identification and batch identification, and business tables are generated in parallel based on configuration information and preset number of concurrent tasks. Operate tasks to realize dual parallel operation at task level and data level.
It improves the efficiency of business data table operations, ensures the accuracy of data processing and task management efficiency, and improves processing speed.
Smart Images

Figure CN119066068A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data technology, and in particular to a method, device, equipment and readable storage medium for operating a table of business data. Background Art
[0002] With the rapid development of database management and data processing technology, efficient processing and management of business data has become an important task of enterprise information management systems. Therefore, table operation technology for business data has emerged, which can perform various processing and management operations on business data stored in the database.
[0003] The traditional method of operating tables for business data is usually to use cursors to perform logical processing on each row of data in the database in turn, and submit the data once after processing multiple rows of data. This method has good adaptability in scenarios with small data volume or relatively simple business logic.
[0004] However, when faced with large-scale data and complex business logic requirements, traditional table operation methods for business data need to process business data row by row, resulting in slow processing speed and low efficiency in table operations for business data. Summary of the invention
[0005] Based on this, it is necessary to provide a method, device, equipment and readable storage medium for table operations on business data that can improve the efficiency of table operations on business data in response to the above technical problems.
[0006] In a first aspect, the present application provides a method for operating a table of business data, comprising:
[0007] The row data contained in the business table data to be processed are processed in batches to obtain multiple batches of business sub-data;
[0008] For any batch of business sub-data in the multiple batches of business sub-data, generating task configuration information corresponding to the any batch of business sub-data according to the row data in the any batch of business sub-data, the location identifier of the row data, and the batch identifier of the any batch of business sub-data;
[0009] Generate multiple business table operation tasks according to the task configuration information corresponding to each batch of the business sub-data and the preset number of concurrent tasks; the business table operation tasks are associated with at least one batch of the business sub-data;
[0010] The multiple business table operation tasks are run in parallel; the business table operation tasks are used to perform a preset type of table operation on at least one batch of the business sub-data associated therewith in parallel.
[0011] In one embodiment, the parallel execution of the multiple business table operation tasks includes:
[0012] Running the multiple business table operation tasks in parallel, and for any business table operation task among the multiple business table operation tasks, performing a data update operation on the business sub-data of the current resource collection stage in at least one batch of the business sub-data associated with the any business table operation task;
[0013] When it is detected that the data update operation of the business sub-data of the current resource collection stage is completed, a data writing operation is performed on the business sub-data of the next resource collection stage in the at least one batch of the business sub-data.
[0014] In one embodiment, before batch processing of the row data included in the business table data to be processed to obtain multiple batches of business sub-data, the method further includes:
[0015] Receiving a batch task processing request at a fixed time; the batch task processing request includes a batch task name;
[0016] In response to the task processing request, determining corresponding business table data to be processed according to the batch task name;
[0017] The step of batch processing the row data contained in the business table data to be processed is executed.
[0018] In one embodiment, the method further comprises:
[0019] For any business table operation task of the multiple business table operation tasks, if there is abnormal business sub-data in which the table operation fails in at least one batch of the business sub-data associated with the any business table operation task, recording a batch identifier of the abnormal business sub-data;
[0020] When the abnormal business sub-data is repaired, a preset type of table operation is performed on the repaired abnormal business sub-data according to the batch identifier of the abnormal business sub-data and through the business table operation task associated with the abnormal business sub-data.
[0021] In one embodiment, the method further comprises:
[0022] Acquire the number of abnormal business sub-data corresponding to any business table operation task of the multiple business table operation tasks; the number of abnormal business sub-data is the number of business sub-data whose table operation fails in at least one batch of the business sub-data associated with any business table operation task;
[0023] When the number of abnormal business sub-data exceeds a failure threshold, execution of any business table operation task is stopped; the failure threshold is equal to the product of the total number of business sub-data associated with any business table operation task and a threshold coefficient.
[0024] In one embodiment, after the multiple business table operation tasks are executed in parallel, the method further includes:
[0025] Acquire task execution information of the multiple business table operation tasks; the task execution information includes the number of business sub-data associated with the business table operation tasks, the number of preset concurrent tasks corresponding to the multiple business table operation tasks, the running time information of the business table operation tasks, and the table operation result information of the business table operation tasks;
[0026] The task execution information of the multiple business table operation tasks is recorded in a task execution information table; the task execution information table is used to indicate the adjustment of the number of business sub-data associated with the business table operation tasks and the preset concurrent task number.
[0027] In a second aspect, the present application further provides a table operation device for business data, including:
[0028] The batch module is used to process the row data contained in the business table data to be processed in batches to obtain multiple batches of business sub-data;
[0029] an information generating module, configured to generate, for any batch of business sub-data among the multiple batches of business sub-data, task configuration information corresponding to the any batch of business sub-data according to row data in the any batch of business sub-data, location identifiers of the row data, and batch identifiers of the any batch of business sub-data;
[0030] A task generation module, used to generate a plurality of business table operation tasks according to the task configuration information corresponding to each batch of the business sub-data and the preset number of concurrent tasks; the business table operation task is associated with at least one batch of the business sub-data;
[0031] The operation module is used to run the multiple business table operation tasks in parallel; the business table operation tasks are used to perform a preset type of table operation on at least one batch of the business sub-data associated therewith in parallel.
[0032] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0033] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.
[0034] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.
[0035] The above-mentioned table operation method, device, equipment and readable storage medium for business data obtain multiple batches of business sub-data by batch processing the row data contained in the business table data to be processed; for any batch of business sub-data in the multiple batches of business sub-data, determine the task configuration information corresponding to any batch of business sub-data based on the row data in any batch of business sub-data, the position identifier of the row data, and the batch identifier of any batch of business sub-data; generate multiple business table operation tasks associated with at least one batch of business sub-data based on the task configuration information corresponding to each batch of business sub-data and the preset number of concurrent tasks; run multiple business table operation tasks in parallel, and perform preset types of table operations in parallel on at least one batch of business sub-data associated with any business table operation task. Large-scale business table data to be processed is processed in batches according to row data to obtain multiple business sub-data with smaller data volumes. The batched business sub-data can be used for parallel table operations, thereby improving the efficiency of table operations on business data; each row of business table data can be accurately located according to row data, location identifier and batch identifier to ensure the accuracy of business data processing; the generated task configuration information can be used to generate business table operation tasks with clear processing objects and processing scopes, thereby improving the efficiency of business table operation task management; and the number of business table operation tasks running in parallel can be flexibly determined according to the preset number of concurrent tasks; not only multiple business table operation tasks run in parallel, but also the business sub-data within each business table operation task can be operated in parallel; dual parallel operation at the task level and the data level can improve the processing speed of business data, thereby achieving the beneficial effect of improving the efficiency of table operations on business data. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0037] FIG1 is an application environment diagram of a method for operating a table of business data in one embodiment;
[0038] FIG2 is a flow chart of a method for operating a table of business data according to an embodiment;
[0039] FIG3 is a schematic diagram of a process flow of a business table operation task in one embodiment;
[0040] FIG4 is a logic diagram of a table operation on business data in one embodiment;
[0041] FIG5 is a flow chart of another method for operating a table of business data in one embodiment;
[0042] FIG6 is a structural block diagram of a device for operating a table of business data in one embodiment;
[0043] FIG. 7 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, groups, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.
[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.
[0046] The table operation method for business data provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The terminal 102 processes the row data contained in the business table data to be processed in batches to obtain multiple batches of business sub-data; the terminal 102 generates task configuration information corresponding to any batch of business sub-data according to the row data in any batch of business sub-data, the position identifier of the row data, and the batch identifier of any batch of business sub-data for any batch of business sub-data in the multiple batches of business sub-data; the terminal 102 generates multiple business table operation tasks according to the task configuration information corresponding to each batch of business sub-data and the preset number of concurrent tasks; the business table operation task is associated with at least one batch of business sub-data; the terminal 102 runs multiple business table operation tasks in parallel; the business table operation task is used to perform a preset type of table operation on at least one batch of business sub-data associated with it in parallel. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.
[0047] In an exemplary embodiment, as shown in FIG. 2 , a method for operating a table of service data is provided, which is described by taking the method applied to the terminal 102 in FIG. 1 as an example, and includes:
[0048] Step S202, batch processing is performed on the row data contained in the business table data to be processed to obtain multiple batches of business sub-data.
[0049] The business table data may refer to business-related data tables stored in a database, such as various business records, such as customer information, transaction records, collection data, etc. The row data contained in the business table data may refer to each row record in the business table data, including specific data of multiple fields, and the row data may be the basic unit of table operation.
[0050] The terminal processes the row data contained in the business table data to be processed in batches, which can be to divide the large-scale business table data into multiple smaller data batches according to certain rules, namely business sub-data, and each batch of business sub-data contains part of the row data in the business table data. Business sub-data is the processing object of table operation, which is convenient for parallel operation and management.
[0051] In a specific implementation, the terminal processes the row data contained in the business table data to be processed in batches. The batch processing may be performed evenly according to the row data, or in batches according to the business type and business stage, or in batches according to the generation time of the row data. There is no specific limitation, so that the number of row data included in each batch of business table data can be flexibly determined according to business needs.
[0052] For example, assuming that the business table data to be processed includes 1000 rows of records, that is, 1000 rows of data, if the business table data is evenly divided according to the rows of data, assuming it is divided into 10 batches of business sub-data, then each batch of business sub-data includes 100 rows of records, that is, 100 rows of data.
[0053] Step S204, for any batch of business sub-data in the multiple batches of business sub-data, generate task configuration information corresponding to any batch of business sub-data according to the row data in any batch of business sub-data, the position identifier of the row data, and the batch identifier of any batch of business sub-data.
[0054] The position identifier can be used to identify the position information of the row data in the business table data, such as the row number or primary key value. Through the position identifier, the position of each row data in the business table can be accurately located to ensure the accuracy of data processing. Optionally, the position identifier of the row data can include the position mark of each row data, or include the position identifier of the starting row data and the position identifier of the ending row data in the business sub-data.
[0055] The batch identifier may be used to identify the number or mark of each batch of business sub-data and distinguish different batches. The batch identifier may be helpful for business table operation task management and scheduling of different batches of business sub-data.
[0056] The task configuration information includes the row data included in the business sub-data, the location identifier of the row data in the business table data, and the batch identifier of the business sub-data, which clarifies the processing requirements and processing scope of each batch of business sub-data, and helps to schedule and execute the business sub-data in subsequent business table operation tasks.
[0057] Step S206: Generate multiple business table operation tasks according to the task configuration information corresponding to each batch of business sub-data and the preset number of concurrent tasks.
[0058] Among them, the preset number of concurrent tasks may refer to the preset number of business table operation tasks that can be run in parallel. The number of business table operation tasks running in parallel can be dynamically adjusted according to the preset number of concurrent tasks, flexibly adapting to different system loads and processing capabilities, as well as adapting to different data processing requirements, thereby improving the processing efficiency and flexibility of business data.
[0059] The business table operation task is associated with at least one batch of business sub-data.
[0060] The business table operation task is a specific operation task generated for one or more batches of business sub-data, which executes the preset table operation.
[0061] Step S208: running multiple business table operation tasks in parallel.
[0062] The business table operation task is used to perform a preset type of table operation on at least one batch of business sub-data associated with it in parallel.
[0063] The terminal runs multiple business table operation tasks in parallel. Multiple business table operation tasks can be run at the same time, and the business sub-data within each business table operation can also be operated in parallel, which improves the efficiency of table operations on business data.
[0064] Optionally, the preset type of table operation may include preset type of table operations such as query, update, insert or delete.
[0065] Taking the bank application scenario as an example, the bank's collection system needs to process a large amount of collection data. Collection data refers to various data generated and used in the collection process. These data are used to manage and execute debt collection activities to ensure that the debt can be recovered in time. Collection data may include the debtor's personal information (identity information and address), debt information (debt details and repayment records), and collection records (collection activities and collection status). The terminal can process the business table recording the collection data in batches according to the corresponding customer identity identifier or the generation time of the row data to obtain multiple batches of business sub-data. For each batch of business sub-data, the terminal generates task configuration information corresponding to the batch of business sub-data based on the row data contained in the batch of business sub-data, the position identifier of the row data in the business table recording the collection data and the batch identifier of the batch of business sub-data. According to the task configuration information corresponding to each batch of business sub-data and the preset number of concurrent tasks, multiple business table operation tasks are generated, and preset table operations are performed on each batch of business sub-data in parallel, such as checking the collection data, updating the collection data, inserting new collection data, generating a collection report based on the collection data, correcting abnormal collection data, etc.
[0066] In the above table operation method for business data, multiple batches of business sub-data are obtained by batch processing the row data contained in the business table data to be processed; for any batch of business sub-data in the multiple batches of business sub-data, the task configuration information corresponding to any batch of business sub-data is determined according to the row data in any batch of business sub-data, the position identifier of the row data, and the batch identifier of any batch of business sub-data; multiple business table operation tasks associated with at least one batch of business sub-data are generated according to the task configuration information corresponding to each batch of business sub-data and the preset number of concurrent tasks; multiple business table operation tasks are run in parallel, and a preset type of table operation is performed in parallel on at least one batch of business sub-data associated with any business table operation task. Large-scale business table data to be processed is processed in batches according to row data to obtain multiple business sub-data with smaller data volumes. The batched business sub-data can be used for parallel table operations, thereby improving the efficiency of table operations on business data; each row of business table data can be accurately located according to row data, location identifier and batch identifier to ensure the accuracy of business data processing; the generated task configuration information can be used to generate business table operation tasks with clear processing objects and processing scopes, thereby improving the efficiency of business table operation task management; and the number of business table operation tasks running in parallel can be flexibly determined according to the preset number of concurrent tasks; not only multiple business table operation tasks run in parallel, but also the business sub-data within each business table operation task can be operated in parallel; dual parallel operation at the task level and the data level can improve the processing speed of business data, thereby achieving the beneficial effect of improving the efficiency of table operations on business data.
[0067] In an exemplary embodiment, the terminal runs multiple business table operation tasks in parallel, including: running multiple business table operation tasks in parallel, and for any business table operation task among the multiple business table operation tasks, performing a data update operation on the business sub-data of the current resource collection stage in at least one batch of business sub-data associated with any business table operation task; and when it is detected that the data update operation of the business sub-data of the current resource collection stage is completed, performing a data write operation on the business sub-data of the next resource collection stage in at least one batch of business sub-data.
[0068] Among them, the current resource collection stage can refer to the specific stage of the collection activity currently being executed in the collection process; the next resource collection stage can refer to the next collection activity stage to be entered after the operation of the current resource collection stage is completed.
[0069] Optionally, each business sub-data may be identified by a stage field to identify the corresponding resource collection stage. The terminal may query the stage field of each batch of business sub-data associated with it through the business table operation task, thereby determining the resource collection stage corresponding to each batch of business sub-data.
[0070] For each business table operation task, firstly, the business sub-data of the current resource collection stage in the business sub-data associated with the business table operation task is updated; after detecting that the data update operation of the current resource collection stage has been completed, the data is written to the business sub-data of the next resource collection stage.
[0071] The data update operation can refer to modifying or updating the business sub-data of existing records, such as modifying the values of certain fields in the business sub-data, without involving the insertion of new records. The data update operation can maintain the timeliness and accuracy of the records. The data update operation will not add new records, but only modify the content of existing records and adjust the values of certain fields to reflect the latest status or information. For example, if a field in a row of data in the business sub-data that indicates the telephone collection status is "Not Contacted", the field can be updated to "Contacted".
[0072] The data writing operation may refer to inserting new data into the business sub-data to generate a new record. For example, a new field may be written into the business sub-data or a new row of data may be inserted. Since the business sub-data of the next resource collection stage is usually blank and unwritten, a new field may be written into each row of the business sub-data or a new row of data may be inserted through the data writing operation.
[0073] Since data processing in the resource collection process usually needs to be carried out in stages, after completing the data update operation in the current stage, immediately entering the data writing operation in the next stage can ensure the continuity and efficiency of the resource collection process, and achieve the technical effect of improving the efficiency of table operations on business data.
[0074] In order to facilitate the understanding of those skilled in the art, FIG3 exemplarily provides a flowchart of a business table operation task. It can be seen that for any business table operation task of multiple business table operation tasks, the business table operation task runs three batches of business sub-data in parallel, and each batch of business sub-data first performs a data update operation on the business sub-data of the current resource collection stage, and then performs a data write operation on the business sub-data of the next resource collection stage. For example, for the business sub-data of the current resource collection stage, the field "not contacted" used to indicate the "telephone collection status" can be updated to "contacted"; for the business sub-data of the next resource collection stage, the fields "2024-06-01", "2024-07-01", and "2024-08-01" used to indicate the "letter sending date" can be written.
[0075] In an exemplary embodiment, the terminal may also periodically receive batch task processing requests; the batch task processing requests include a batch task name; in response to the task processing request, the corresponding business table data to be processed is determined according to the batch task name; and the step of batch processing the row data contained in the business table data to be processed is executed.
[0076] In a specific implementation, the terminal may receive a batch task processing request from a scheduled job system, the batch task processing request including a batch task name, and the batch task name is used to indicate the business table data that needs to be processed in batches. The scheduled job system may trigger the sending of the batch task processing request according to a preset schedule. The terminal responds to the batch task processing request and searches and determines the business table data that needs to be processed according to the batch task name included in the batch task processing request. The terminal may call the corresponding Java service according to the batch task name and execute the step of batch processing the row data included in the business table data to be processed.
[0077] By regularly receiving batch task processing requests, batch task processing can be automatically triggered, which improves the processing efficiency and intelligence of business table data and ensures the continuity of business processes. In addition, scheduled calls can flexibly set the execution frequency and time point, and adjust the execution of business table operation tasks at any time according to business needs, thereby achieving the beneficial effect of improving the processing efficiency and flexibility of business table data.
[0078] In an exemplary embodiment, for any business table operation task of multiple business table operation tasks, if there is abnormal business sub-data whose table operation fails in at least one batch of business sub-data associated with any business table operation task, the terminal records the batch identifier of the abnormal business sub-data; when the abnormal business sub-data is repaired, based on the batch identifier of the abnormal business sub-data, a preset type of table operation is performed on the repaired abnormal business sub-data through the business table operation task associated with the abnormal business sub-data.
[0079] Among them, abnormal business sub-data may refer to business sub-data that causes table operation failure due to various reasons (such as data format error, constraint conflict, etc.) during the business table operation process. Each batch of business sub-data has a unique batch identifier, which is used to track and manage different batches of business sub-data. When the terminal detects that there is abnormal business sub-data in a business table operation task, the batch identifier can be recorded for subsequent processing. The terminal can receive information indicating that the abnormal business sub-data has been repaired, and determine that the abnormal business sub-data has been repaired, and find the business sub-data that needs to be reprocessed according to the batch identifier of the abnormal business sub-data previously recorded; for the repaired abnormal business sub-data, re-execute the preset type of table operation to ensure that the data processing is complete and correct. When a batch of business sub-data fails, it can be re-run according to the batch identifier of the batch of business sub-data after repair, without re-running all batches of business sub-data.
[0080] In the case where there is abnormal business sub-data in the business table operation task, the above method can quickly locate and find the repaired abnormal business sub-data through batch identification after the abnormal business sub-data is repaired. It is only necessary to re-execute the table operation on the repaired abnormal sub-data separately, and there is no need to run each batch of business sub-data in the business table operation task in parallel again, thereby improving the efficiency of table operations on business data.
[0081] In an exemplary embodiment, the terminal obtains the number of abnormal business sub-data corresponding to any business table operation task of multiple business table operation tasks; the number of abnormal business sub-data is the number of business sub-data on which table operation fails in at least one batch of business sub-data associated with any business table operation task; when the number of abnormal business sub-data exceeds a failure threshold, execution of any business table operation task is stopped; the failure threshold is equal to the product of the total number of business sub-data associated with any business table operation task and the threshold coefficient.
[0082] For example, if the total number of business sub-data associated with any business table operation task is 1000 batches, and the threshold coefficient is 0.002, then only 2 batches of business sub-data can fail among the 1000 batches of business sub-data. If 2 batches of business sub-data are detected, the execution of the business table operation task is stopped, and the remaining 998 batches of business sub-data do not need to be run. For some very important business table operation tasks, the threshold coefficient can be set to 0, that is, no batch of business sub-data can fail; for some not very important business table operation tasks, the threshold coefficient can be increased to allow a small number of batches to fail. After the business sub-data that failed in the table operation task is repaired, the business table operation task can be re-executed.
[0083] The above method uses the failure threshold to promptly stop the business table operation tasks with a large number of abnormal business sub-data, which can avoid the waste of resources caused by the continuous operation of business table operation tasks with many abnormal situations, and achieves the beneficial effect of improving the efficiency of table operations on business data.
[0084] In an exemplary embodiment, the terminal may obtain task execution information of multiple business table operation tasks; and record the task execution information of the multiple business table operation tasks into a task execution information table.
[0085] The task execution information includes the number of business sub-data associated with the business table operation task, the number of preset concurrent tasks corresponding to multiple business table operation tasks, the running time information of the business table operation task, and the table operation result information of the business table operation task.
[0086] Among them, the number of business sub-data associated with the business table operation task is the number of batches of business sub-data processed by the business table operation task; the preset number of concurrent tasks is the number of multiple business table operation tasks running in parallel; the running time information of the business table operation task may include the start time, end time, total time, average time, etc. of the business table operation task; the table operation result information of the business table operation task may include the number of batches of business sub-data for which the table operation is successful, the number of batches of business sub-data for which the table operation fails, the failure threshold, and a Boolean value used to indicate whether the number of abnormal business sub-data exceeds the failure threshold.
[0087] Optionally, the task execution information may further include the task identifier of the business table operation task, the number of row data included in each batch of business sub-data, and the like.
[0088] Among them, the task execution information table is used to indicate the adjustment of the number of business sub-data associated with the business table operation task and the preset number of concurrent tasks. For example, if the priority of a business table operation task is high, but the total time consumed by the business table operation task is long according to the task execution information table, the preset number of concurrent tasks can be increased; or, if the number of batches of business sub-data that failed the table operation is large according to the task execution information table, the number of business sub-data associated with the business table operation task can be reduced.
[0089] By recording the task execution information of multiple business table operation tasks into the task execution information table, the number of business sub-data associated with the business table operation task and the preset number of concurrent tasks can be adaptively modified according to the running time information of the business table operation task, the table operation result information of the business table operation task, etc. in the task execution information. Therefore, each business table operation task can be flexibly configured according to the business scenario and business priority to obtain the configuration value of the business table operation task with the best performance. It can also be adjusted according to different terminal device configurations and database server configurations, thereby achieving the beneficial effect of improving the performance of business table operation tasks and improving the efficiency of table operations on business data.
[0090] To facilitate the understanding of those skilled in the art, FIG4 exemplarily provides a logic diagram of table operations on business data. It can be seen that the scheduled job system of the terminal regularly initiates batch task requests, and the terminal can regularly receive batch task requests, and call the corresponding Java service according to the batch task name, batch the processed business table data to obtain multiple batches of business sub-data, and then batch process the business sub-data concurrently. When the business sub-data is logically processed, the business sub-data to be updated and the business sub-data to be written are divided into two sets, and each business table operation task (task 1, task 2, task 3 and task 4) can batch update business table 1 in parallel, and batch write task table 2 in parallel, so that business table 1 can be batch updated and business table 2 can be batch updated. After the task is completed, the task execution information is recorded. Regardless of whether the business table operation task is executed normally or abnormally, the corresponding task execution information can be returned, and the task execution information can be called back. The business table operation task can call back the corresponding task execution information after a single batch of business sub-data is run, or the corresponding task execution information can be called back after all batches of business sub-data are run. In summary, the java service can improve the scalability of the table operation method for business data, and use batch concurrent processing instead of cursor serial processing to support accurate batching of very large data volumes and avoid data processing duplication and concurrent conflicts.
[0091] FIG. 5 is another table operation method for service data according to an exemplary embodiment. As shown in FIG. 5 , the method is applied to the terminal 102 in FIG. 1 as an example for description, and includes the following steps:
[0092] Step S502: receiving batch task processing requests regularly.
[0093] The batch task processing request includes a batch task name.
[0094] Step S504: in response to the task processing request, determine the corresponding business table data to be processed according to the batch task name.
[0095] Step S506, batch processing is performed on the row data contained in the business table data to be processed to obtain multiple batches of business sub-data.
[0096] Step S508, for any batch of business sub-data in the multiple batches of business sub-data, generate task configuration information corresponding to any batch of business sub-data according to the row data in any batch of business sub-data, the position identifier of the row data, and the batch identifier of any batch of business sub-data.
[0097] Step S510: Generate multiple business table operation tasks according to the task configuration information corresponding to each batch of business sub-data and the preset number of concurrent tasks.
[0098] The business table operation task is associated with at least one batch of business sub-data.
[0099] Step S512, running multiple business table operation tasks in parallel, and for any business table operation task among the multiple business table operation tasks, performing a data update operation on the business sub-data of the current resource collection stage in at least one batch of business sub-data associated with any business table operation task.
[0100] The business table operation task is used to perform a preset type of table operation on at least one batch of business sub-data associated with it in parallel.
[0101] Step S514, when it is detected that the data update operation of the business sub-data of the current resource collection stage is completed, a data writing operation is performed on the business sub-data of the next resource collection stage in at least one batch of the business sub-data.
[0102] Step S516, for any business table operation task of the multiple business table operation tasks, if there is abnormal business sub-data with table operation failure in at least one batch of business sub-data associated with any business table operation task, record the batch identifier of the abnormal business sub-data.
[0103] Step S518, when the abnormal business sub-data is repaired, according to the batch identifier of the abnormal business sub-data, a preset type of table operation is performed on the repaired abnormal business sub-data through the business table operation task associated with the abnormal business sub-data.
[0104] Step S520: Acquire the number of abnormal business sub-data corresponding to any business table operation task of the plurality of business table operation tasks.
[0105] The number of abnormal business sub-data is the number of business sub-data whose table operation fails in at least one batch of business sub-data associated with any business table operation task.
[0106] Step S522: when the number of abnormal business sub-data exceeds the failure threshold, stop executing any business table operation task.
[0107] The failure threshold is equal to the product of the total number of business sub-data associated with any business table operation task and the threshold coefficient.
[0108] Step S524: Obtain task execution information of multiple business table operation tasks.
[0109] The task execution information includes the number of business sub-data associated with the business table operation task, the preset number of concurrent tasks corresponding to multiple business table operation tasks, the running time information of the business table operation task, and the table operation result information of the business table operation task.
[0110] Step S526: Record the task execution information of the multiple business table operation tasks into the task execution information table.
[0111] The task execution information table is used to indicate the adjustment of the number of business sub-data associated with the business table operation task and the preset concurrent task number.
[0112] It should be noted that the specific limitations of the above steps can be found in the specific limitations of a business data sharing method mentioned above, which will not be repeated here.
[0113] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0114] Based on the same inventive concept, the embodiment of the present application also provides a table operation device for business data for implementing the table operation method for business data involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the table operation device for business data provided below can refer to the limitations of the table operation method for business data above, and will not be repeated here.
[0115] In an exemplary embodiment, as shown in FIG6 , a table operation device for business data is provided, including:
[0116] The batch module 610 is used to process the row data contained in the business table data to be processed in batches to obtain multiple batches of business sub-data.
[0117] The information generation module 620 is used to generate task configuration information corresponding to any batch of business sub-data among the multiple batches of business sub-data based on the row data in the any batch of business sub-data, the location identifier of the row data, and the batch identifier of the any batch of business sub-data.
[0118] The task generation module 630 is used to generate a plurality of business table operation tasks according to the task configuration information corresponding to each batch of the business sub-data and the preset number of concurrent tasks; the business table operation task is associated with at least one batch of the business sub-data.
[0119] The running module 640 is used to run the multiple business table operation tasks in parallel; the business table operation tasks are used to perform a preset type of table operation on at least one batch of the business sub-data associated therewith in parallel.
[0120] In one of the embodiments, the operation module 640 is used to run the multiple business table operation tasks in parallel, and for any business table operation task among the multiple business table operation tasks, perform a data update operation on the business sub-data of the current resource collection stage in at least one batch of the business sub-data associated with the any business table operation task; when it is detected that the data update operation of the business sub-data of the current resource collection stage is completed, perform a data write operation on the business sub-data of the next resource collection stage in the at least one batch of the business sub-data.
[0121] In one of the embodiments, the device also includes a response module, which is used to periodically receive batch task processing requests; the batch task processing requests include a batch task name; in response to the task processing request, the corresponding business table data to be processed is determined according to the batch task name; and the step of batch processing the row data contained in the business table data to be processed is executed.
[0122] In one of the embodiments, the operation module 640 is used to record the batch identifier of the abnormal business sub-data for any business table operation task of the multiple business table operation tasks, when there is abnormal business sub-data with failed table operation in at least one batch of the business sub-data associated with any of the business table operation tasks; and when the abnormal business sub-data is repaired, based on the batch identifier of the abnormal business sub-data, perform a preset type of table operation on the repaired abnormal business sub-data through the business table operation task associated with the abnormal business sub-data.
[0123] In one of the embodiments, the device also includes a stop module, which is used to obtain the number of abnormal business sub-data corresponding to any business table operation task of the multiple business table operation tasks; the number of abnormal business sub-data is the number of business sub-data on which table operation fails in at least one batch of the business sub-data associated with any business table operation task; when the number of abnormal business sub-data exceeds a failure threshold, the execution of any business table operation task is stopped; the failure threshold is equal to the product of the total number of business sub-data associated with any business table operation task and a threshold coefficient.
[0124] In one embodiment, the device also includes a recording module, which is used to obtain task execution information of the multiple business table operation tasks; the task execution information includes the number of business sub-data associated with the business table operation tasks, the preset number of concurrent tasks corresponding to the multiple business table operation tasks, the running time information of the business table operation tasks, and the table operation result information of the business table operation tasks; the task execution information of the multiple business table operation tasks is recorded in a task execution information table; the task execution information table is used to indicate adjustments to the number of business sub-data associated with the business table operation tasks and the preset number of concurrent tasks.
[0125] Each module in the above-mentioned table operation device for business data can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0126] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a table operation method for business data is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.
[0127] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0128] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned method for operating a table of business data. The steps of the above-mentioned method for operating a table of business data may be the steps of the above-mentioned method for operating a table of business data in each of the above-mentioned embodiments.
[0129] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned method for operating a table of business data. The steps of the above-mentioned method for operating a table of business data may be the steps of the above-mentioned method for operating a table of business data in each of the above-mentioned embodiments.
[0130] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the processor executes the steps of the above-mentioned method for operating a table of business data. The steps of the above-mentioned method for operating a table of business data may be the steps of the above-mentioned method for operating a table of business data in each of the above-mentioned embodiments.
[0131] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0132] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0133] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for operating a table of business data, characterized in that: The method comprises: The row data contained in the business table data to be processed are processed in batches to obtain multiple batches of business sub-data; For any batch of business sub-data in the multiple batches of business sub-data, generating task configuration information corresponding to the any batch of business sub-data according to the row data in the any batch of business sub-data, the location identifier of the row data, and the batch identifier of the any batch of business sub-data; Generate multiple business table operation tasks according to the task configuration information corresponding to each batch of the business sub-data and the preset number of concurrent tasks; the business table operation tasks are associated with at least one batch of the business sub-data; The multiple business table operation tasks are run in parallel; the business table operation tasks are used to perform a preset type of table operation on at least one batch of the business sub-data associated therewith in parallel.
2. The method according to claim 1, characterized in that: The parallel execution of the plurality of business table operation tasks includes: Running the multiple business table operation tasks in parallel, and for any business table operation task among the multiple business table operation tasks, performing a data update operation on the business sub-data of the current resource collection stage in at least one batch of the business sub-data associated with the any business table operation task; When it is detected that the data update operation of the business sub-data of the current resource collection stage is completed, a data writing operation is performed on the business sub-data of the next resource collection stage in the at least one batch of the business sub-data.
3. The method according to claim 1, characterized in that Before batch processing of the row data contained in the business table data to be processed to obtain multiple batches of business sub-data, the method further includes: Receiving a batch task processing request at a fixed time; the batch task processing request includes a batch task name; In response to the task processing request, determining corresponding business table data to be processed according to the batch task name; The step of batch processing the row data contained in the business table data to be processed is executed.
4. The method according to claim 1, characterized in that: The method further comprises: For any business table operation task of the multiple business table operation tasks, if there is abnormal business sub-data in which the table operation fails in at least one batch of the business sub-data associated with the any business table operation task, recording a batch identifier of the abnormal business sub-data; When the abnormal business sub-data is repaired, a preset type of table operation is performed on the repaired abnormal business sub-data according to the batch identifier of the abnormal business sub-data and through the business table operation task associated with the abnormal business sub-data.
5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire the number of abnormal business sub-data corresponding to any business table operation task of the multiple business table operation tasks; the number of abnormal business sub-data is the number of business sub-data whose table operation fails in at least one batch of the business sub-data associated with any business table operation task; When the number of abnormal business sub-data exceeds a failure threshold, execution of any business table operation task is stopped; the failure threshold is equal to the product of the total number of business sub-data associated with any business table operation task and a threshold coefficient.
6. The method according to claim 1, characterized in that After the multiple business table operation tasks are executed in parallel, the method further includes: Acquire task execution information of the multiple business table operation tasks; the task execution information includes the number of business sub-data associated with the business table operation tasks, the number of preset concurrent tasks corresponding to the multiple business table operation tasks, the running time information of the business table operation tasks, and the table operation result information of the business table operation tasks; The task execution information of the multiple business table operation tasks is recorded in a task execution information table; the task execution information table is used to indicate the adjustment of the number of business sub-data associated with the business table operation tasks and the preset concurrent task number.
7. A table operation device for business data, characterized in that: The device comprises: The batch module is used to process the row data contained in the business table data to be processed in batches to obtain multiple batches of business sub-data; an information generating module, configured to generate, for any batch of business sub-data among the multiple batches of business sub-data, task configuration information corresponding to the any batch of business sub-data according to row data in the any batch of business sub-data, location identifiers of the row data, and batch identifiers of the any batch of business sub-data; A task generation module, used to generate a plurality of business table operation tasks according to the task configuration information corresponding to each batch of the business sub-data and the preset number of concurrent tasks; the business table operation task is associated with at least one batch of the business sub-data; The operation module is used to run the multiple business table operation tasks in parallel; the business table operation tasks are used to perform a preset type of table operation on at least one batch of the business sub-data associated therewith in parallel.
8. The device according to claim 7, characterized in that The running module is used to run the multiple business table operation tasks in parallel, and for any business table operation task among the multiple business table operation tasks, perform a data update operation on the business sub-data of the current resource collection stage in at least one batch of the business sub-data associated with the any business table operation task; when it is detected that the data update operation of the business sub-data of the current resource collection stage is completed, perform a data write operation on the business sub-data of the next resource collection stage in the at least one batch of the business sub-data.
9. The device according to claim 7, characterized in that The device also includes a response module, which is used to periodically receive batch task processing requests; the batch task processing requests include a batch task name; in response to the task processing request, the corresponding business table data to be processed is determined according to the batch task name; and the step of batch processing the row data contained in the business table data to be processed is executed.
10. The device according to claim 7, characterized in that The operation module is used for recording a batch identifier of the abnormal business sub-data for any business table operation task of the multiple business table operation tasks, if there is abnormal business sub-data in which the table operation fails in at least one batch of the business sub-data associated with the any business table operation task; When the abnormal business sub-data is repaired, a preset type of table operation is performed on the repaired abnormal business sub-data according to the batch identifier of the abnormal business sub-data and through the business table operation task associated with the abnormal business sub-data.
11. The device according to any one of claims 7 to 10, characterized in that The device further includes a stop module, the stop module being used to obtain the number of abnormal business sub-data corresponding to any business table operation task of the multiple business table operation tasks; the number of abnormal business sub-data is the number of business sub-data of at least one batch of the business sub-data associated with any business table operation task whose table operation fails; when the number of abnormal business sub-data exceeds a failure threshold, stop executing any business table operation task; The failure threshold is equal to the product of the total number of business sub-data associated with any business table operation task and the threshold coefficient.
12. The device according to claim 1, characterized in that The device also includes a recording module, which is used to obtain task execution information of the multiple business table operation tasks; the task execution information includes the number of business sub-data associated with the business table operation tasks, the number of preset concurrent tasks corresponding to the multiple business table operation tasks, the running time information of the business table operation tasks, and the table operation result information of the business table operation tasks; The task execution information of the multiple business table operation tasks is recorded in a task execution information table; the task execution information table is used to indicate the adjustment of the number of business sub-data associated with the business table operation tasks and the preset concurrent task number.
13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
