Task batch processing method and apparatus, computer device, and storage medium

Abstract

This application belongs to the fields of big data and fintech, and is applied to the field of batch processing of insurance policy fees. It relates to a task batch processing method, apparatus, computer equipment, and storage medium, including receiving a batch processing request; obtaining a configuration file; identifying unused configuration text segments in the current batch processing; optimizing and adjusting the batch processing task logic; deploying the optimized and adjusted task logic and configuration file to several batch processing systems; processing fee change information in segments; parsing each piece of fee change information, and using the parsing results as method execution parameters to execute the batch processing task. This application optimizes and adjusts the task logic through configuration files and uses a modulo method to process all fee change information in segments, ensuring that each fee change information is distributed to its corresponding batch processing system. This optimizes task batch processing, improves batch processing efficiency, ensures reasonable utilization of server resources, and avoids system crashes caused by batch processing events.

Description

Technical Field

[0001] This application relates to the fields of big data and financial technology, and in particular to a task batch processing method, apparatus, computer equipment and storage medium. Background Technology

[0002] Batch processing, also known as batch scripting, is, as the name suggests, the process of processing a single object in batches. In insurance policy fee processing, batch processing is frequently used. For example, when processing new year's insurance premiums, a full search of all insurance types purchased by a user is needed to obtain the fees for each type of insurance the user needs to pay in the new year, and then send these fees to the user all at once. Another example is when there are multiple policy agreements and policy fees change frequently; batch processing is used to comprehensively calculate the final fees based on the order in which the agreements were signed or fulfilled. Yet another example is when performing regional aggregate calculations for insurance policy fees, calculating the total claims amount and the total amount of new insurance premiums for each province each quarter; all of these require batch processing of insurance policy fees.

[0003] The current version handles tens of thousands of batch processing tasks nationwide using a single thread. Processing starts at 2 AM and doesn't finish until 2 PM, severely impacting expense reimbursement progress. In case of financial closings or other emergencies, it simply cannot meet user needs. Furthermore, the current processing method places a significant strain on server resources, potentially leading to memory leaks and system crashes. Summary of the Invention

[0004] The purpose of this application is to provide a task batch processing method, apparatus, computer device, and storage medium to optimize task batch processing, improve batch processing efficiency, ensure reasonable utilization of server resources, and avoid system crashes caused by batch processing events.

[0005] To address the aforementioned technical problems, this application provides a task batch processing method, employing the following technical solution:

[0006] A task batch processing method includes the following steps:

[0007] Receive batch processing requests corresponding to fee changes, wherein the batch processing request includes the extraction addresses of n fee change information to be batch processed, where n is a positive integer;

[0008] Read the configuration file database and obtain the initial configuration file and current configuration file of the batch processing according to the distinguishing cache identifier of the target configuration file. The configuration file database caches the target configuration file, which includes the initial configuration file and the current configuration file of the batch processing. The initial configuration file and the current configuration file of the batch processing are preset with distinguishing cache identifiers.

[0009] Based on preset filtering rules, configuration text segments not used in this batch processing are identified from the initial configuration file;

[0010] Based on the configuration text segment, the task logic for batch processing is optimized and adjusted, wherein the task logic for batch processing is a batch processing source execution program pre-deployed on the task execution terminal corresponding to the batch processing task.

[0011] Obtain the optimized and adjusted task logic, and deploy the task logic and the current configuration file to m batch processing systems, where m is the total number of batch processing systems and m is a positive integer;

[0012] The n fee change information entries are obtained based on the extraction address, and the n fee change information entries are processed by segmentation using a preset segmentation rule;

[0013] After parsing the cost change information of each piece of data after sharding, the parsing results are used as method execution parameters and input into the m batch processing systems to execute the batch processing task based on the sharding rules.

[0014] Furthermore, the step of identifying unused configuration text segments from the initial configuration file based on preset filtering rules specifically includes:

[0015] Parse the initial configuration file to obtain the initial script content;

[0016] Parse the current configuration file to obtain the current script content;

[0017] Based on the initial script content and the current script content, the script content that is not used in the current configuration file during the execution of this batch processing task is selected as the unused configuration text segment in this batch processing process.

[0018] Furthermore, before the step of optimizing and adjusting the task logic for batch processing based on the configuration text segment, the method further includes:

[0019] The call relationship between the initial script content and the batch processing task logic is set in advance, wherein the batch processing task logic is used to execute the corresponding cost information update task with the corresponding cost change information as a parameter;

[0020] The call relationship is persistently cached in the form of a form to a preset repository.

[0021] Furthermore, the step of optimizing and adjusting the task logic for batch processing based on the configuration text segment specifically includes:

[0022] Read the repository to obtain the call relationship corresponding to the configuration text segment;

[0023] Based on the aforementioned call relationships, the task logic that has not been enabled in the batch processing task logic is filtered out;

[0024] Based on the preset task call status code, the inactive task logic is set to an uncallable state.

[0025] Furthermore, before the step of setting the inactive task logic to an uncallable state based on a preset task invocation status code, the method further includes:

[0026] A distinguishing status code is pre-set based on whether the task logic is invoked, wherein the distinguishing status code includes an invoked status code and a non-invoked status code;

[0027] The step of setting the inactive task logic to an uncallable state based on a preset task invocation status code specifically includes:

[0028] Set the distinguishing status code corresponding to the inactive task logic to the non-call status code.

[0029] Furthermore, the step of obtaining the n fee change information entries based on the extraction address and performing fragmentation processing on the n fee change information entries using a preset fragmentation rule specifically includes:

[0030] A serial number is assigned to each acquired fee change information, wherein the serial number is a positive integer and is less than or equal to n;

[0031] Based on the preset modulo formula: b = a % m, the n cost change information is segmented to obtain the segment number corresponding to each cost change information, where m is the total number of the batch processing system, a is the sequence number corresponding to the current cost change information, and b is the modulo value;

[0032] A set of cost change information is constructed using the segment number as the set identifier;

[0033] Individual cost change information with the same segment number is cached as a set element in the corresponding cost change information set.

[0034] Furthermore, the step of parsing the cost change information after sharding and inputting the parsing results as method execution parameters into the m batch processing systems to execute the batch processing task based on the sharding rules specifically includes:

[0035] The different set elements within the cost change information set are parsed, and after parsing, a corresponding set identifier is assigned to each parsing result;

[0036] Set a distinguishing number for each of the m batch processing systems, wherein the distinguishing number is an integer greater than or equal to 0, and its value range is [0, m);

[0037] Obtain each parsing result corresponding to the set identifier that has the same difference number, and input each parsing result as a method execution parameter into the batch processing system corresponding to the difference number to execute the batch processing task.

[0038] To address the aforementioned technical problems, this application also provides a task batch processing device, which employs the following technical solution:

[0039] A task batch processing apparatus, comprising:

[0040] The request receiving module is used to receive batch processing requests corresponding to fee changes, wherein the batch processing request includes the extraction address of n fee change information to be batch processed, where n is a positive integer;

[0041] The configuration acquisition module is used to read the configuration file database and obtain the initial configuration file and the current configuration file of the batch processing according to the distinguishing cache identifier of the target configuration file. The configuration file database caches the target configuration file, and the target configuration file includes the initial configuration file and the current configuration file of the batch processing. The initial configuration file and the current configuration file of the batch processing are preset with distinguishing cache identifiers.

[0042] The filtering and identification module is used to identify configuration text segments that are not used in this batch processing from the initial configuration file based on preset filtering rules;

[0043] The optimization and adjustment module is used to optimize and adjust the task logic for batch processing based on the configuration text segment, wherein the task logic for batch processing is a batch processing source execution program pre-deployed on the task execution terminal corresponding to the task batch processing.

[0044] The cluster deployment module is used to obtain the optimized and adjusted task logic, and deploy the task logic and the current configuration file to m batch processing systems, where m is the total number of batch processing systems and m is a positive integer;

[0045] The data sharding module is used to obtain the n fee change information entries based on the extraction address, and to shard the n fee change information entries using preset sharding rules;

[0046] The task execution module is used to parse each cost change information after sharding, and input the parsing results as method execution parameters into the m batch processing systems to execute the batch processing task based on the sharding rules.

[0047] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0048] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the task batch processing method described above.

[0049] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0050] A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the task batch processing method described above.

[0051] Compared with the prior art, the embodiments of this application have the following main advantages:

[0052] The task batch processing method described in this application involves: receiving a batch processing request; obtaining a configuration file; identifying unused configuration text segments in the current batch processing; optimizing and adjusting the batch processing task logic; deploying the optimized and adjusted task logic and configuration file to several batch processing systems; processing fee change information in segments; parsing each piece of fee change information; and using the parsing results as method execution parameters to execute the batch processing task. This application optimizes and adjusts the task logic through configuration files, deploys the optimized and adjusted task logic to the cluster system for batch processing, processes the batch processing task in segments, and distributes the segmented fee change information to the corresponding batch processing systems. This optimizes task batch processing, improves batch processing efficiency, ensures reasonable utilization of server resources, and avoids system crashes caused by batch processing events. Attached Figure Description

[0053] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0055] Figure 2 A flowchart of an embodiment of the task batch processing method according to this application;

[0056] Figure 3 yes Figure 2 A flowchart of a specific implementation of step 203 shown;

[0057] Figure 4 yes Figure 2 A flowchart of a specific implementation of step 204 shown;

[0058] Figure 5 yes Figure 4 A flowchart of a specific implementation of step 403 shown;

[0059] Figure 6 yes Figure 2 A flowchart of a specific implementation of step 206 shown;

[0060] Figure 7 yes Figure 2 A flowchart of a specific implementation of step 207 shown;

[0061] Figure 8 A schematic diagram of a structure of an embodiment of the task batch processing apparatus according to this application;

[0062] Figure 9 A schematic diagram of the structure of an embodiment of the computer device according to this application. Detailed Implementation

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0064] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0066] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0067] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0068] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0069] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0070] It should be noted that the task batch processing method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the task batch processing device is generally set in the server / terminal device.

[0071] It should be understood that Figure 1The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0072] Continue to refer to Figure 2 A flowchart of an embodiment of the task batch processing method according to this application is shown. The task batch processing method includes the following steps:

[0073] Step 201: Receive batch processing requests corresponding to fee changes, wherein the batch processing request includes the extraction addresses of n fee change information to be batch processed, where n is a positive integer.

[0074] When calculating insurance policy fees regionally, taking the total amount of insurance claims and the total amount of new insurance contracts signed in each province for each quarter as an example, since the data volume is large and covers data from several provinces, batch processing is required. Before executing batch processing, the extraction address of the n fee change information to be processed must be obtained through the batch processing request in order to extract the fee change information.

[0075] Step 202: Read the configuration file database and obtain the initial configuration file and current configuration file of the batch processing according to the distinguishing cache identifier of the target configuration file. The configuration file database caches the target configuration file, which includes the initial configuration file and the current configuration file of the batch processing. The initial configuration file and the current configuration file of the batch processing are preset with distinguishing cache identifiers.

[0076] In this embodiment, the initial configuration file and the current configuration file of the batch processing only refer to the configuration files related to the execution logic of the batch processing task, and do not include the hardware resource configuration attributes of the execution system.

[0077] When continuing to calculate regional summaries based on insurance policy fees, taking the calculation of the total insurance claims amount and the total amount of new insurance contracts signed in each province for each quarter as an example, the initial configuration file often involves not only provincial audit summaries, but also county-level and city-level audit summaries. It may also involve not only quarterly summaries, but also monthly, semi-annual, and annual summaries. In this case, the initial configuration file will also contain the configuration text segments corresponding to county-level and city-level audit summaries, monthly summaries, semi-annual summaries, and annual summaries. The current configuration file refers to the configuration text segments required for this quarterly summary of each province.

[0078] Step 203: Based on preset filtering rules, identify configuration text segments that were not used in this batch processing from the initial configuration file.

[0079] When continuing to calculate the regional summary based on insurance follow-up fees, taking the calculation of the total insurance claims amount and the total amount of new insurance contracts for each province in each quarter as an example, the current configuration file and the initial configuration file are compared, and the configuration text segments in the initial configuration file that were not used in this batch processing are filtered out.

[0080] In this embodiment, the step of identifying unused configuration text segments from the initial configuration file based on preset filtering rules specifically includes: parsing the initial configuration file to obtain the initial script content; parsing the current configuration file to obtain the current script content; and filtering out unused script content in the current configuration file during the execution of the current batch processing task as unused configuration text segments during the current batch processing based on the initial script content and the current script content.

[0081] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific implementation of step 203 shown includes the following steps:

[0082] Step 301: Parse the initial configuration file to obtain the initial script content;

[0083] Step 302: Parse the current configuration file to obtain the current script content;

[0084] Step 303: Based on the initial script content and the current script content, filter out the script content that is not used in the current configuration file when this batch processing task is executed, and use it as the unused configuration text segment in this batch processing process.

[0085] By parsing, the script content corresponding to the initial configuration file and the current configuration file is obtained, and the configuration text segments not used in this batch processing are identified by comparing the script content.

[0086] In this embodiment, the step of filtering out unused script content in the current configuration file as unused configuration text segments during the execution of this batch processing task based on the initial script content and the current script content specifically includes: using a preset text recognition technology to identify the initial script content and the current script content respectively, wherein the text recognition technology includes OCR text recognition technology; marking each character in the identified current script content according to its position information in the current script content, wherein the position information includes the line number where the character is located and the corresponding inline character number from the left to the right of the line; obtaining the character content of each line in the current script content line by line according to the markings corresponding to each character in the current script content, as a unit comparison line, with the entire initial script content as the reference text; identifying the script content corresponding to each unit comparison line from the reference text, and marking the script content in the reference text; obtaining the unmarked script content in the reference text, i.e., the unused configuration text segments during this batch processing.

[0087] Step 204: Optimize and adjust the task logic for batch processing based on the configuration text segment, wherein the task logic for batch processing is a batch processing source execution program pre-deployed on the task execution terminal corresponding to the task batch processing.

[0088] When continuing to calculate regional aggregated insurance policy fees, taking the calculation of the total insurance claims amount and the total amount of new insurance contracts signed in each province for each quarter as an example, the initial configuration file generally has more comprehensive batch processing task logic and a larger amount of code. During execution, it does not execute too much logic that is not useful for this batch processing. The current configuration file has relatively less batch processing task logic and a smaller amount of code. In this case, the batch processing task logic is optimized and adjusted to avoid executing useless task logic during this batch processing, thereby reducing the batch processing time.

[0089] In this embodiment, before the step of optimizing and adjusting the batch processing task logic based on the configuration text segment, the method further includes: setting a call relationship in advance for the initial script content and the batch processing task logic, wherein the batch processing task logic is used to execute the corresponding cost information update task with the corresponding cost change information as a parameter; and persistently caching the call relationship in the form of a form to a preset repository.

[0090] In this embodiment, the calling relationship refers to the execution correspondence relationship, that is, each line of script content in the initial script content has a corresponding execution method in the batch processing task logic.

[0091] When continuing to calculate regional summaries based on insurance policy fees, taking the calculation of the total insurance claims amount and the total amount of new insurance contracts signed in each province for each quarter as an example, by pre-setting the calling relationship between the initial script content and the batch processing task logic, it is ensured that when optimizing and adjusting the batch processing task logic, the unused task logic can be quickly located through the calling relationship.

[0092] In this embodiment, the step of optimizing and adjusting the batch processing task logic based on the configuration text segment specifically includes: reading the repository and obtaining the call relationship corresponding to the configuration text segment; filtering out the unenabled task logic in the batch processing task logic based on the call relationship; and setting the unenabled task logic to an uncallable state based on a preset task call status code.

[0093] Continue to refer to Figure 4 , Figure 4 yes Figure 2 A flowchart of a specific implementation of step 204 shown includes the following steps:

[0094] Step 401: Read the repository and obtain the call relationship corresponding to the configuration text segment;

[0095] Step 402: Based on the calling relationship, filter out the task logic that has not been enabled in the batch processing task logic;

[0096] Step 403: Based on the preset task call status code, set the inactive task logic to an uncallable state.

[0097] In this embodiment, the task invocation status code includes an invokeable state and an invokeable state, which are distinguished by the code values ​​"0" and "1" respectively. Assuming that the initial status code values ​​of the task invocation status codes corresponding to the batch processing task logic are all "0", which is the invokeable state, after filtering out the inactive task logic in the batch processing task logic based on the invocation relationship, the task invocation status code corresponding to the inactive task logic is set from the initial status code value "0" to the code value "1", that is, the inactive task logic is set to the invokeable state.

[0098] In this embodiment, before the step of setting the inactive task logic to an uncallable state based on a preset task call status code, the method further includes: pre-setting a distinguishing status code according to whether the task logic is called, wherein the distinguishing status code includes a call status code and a non-call status code;

[0099] In this embodiment, the step of setting the inactive task logic to an uncallable state based on a preset task call status code specifically includes: setting the distinguishing status code corresponding to the inactive task logic to a non-call status code.

[0100] Continue to refer to Figure 5 , Figure 5 yes Figure 4 A flowchart of a specific implementation of step 403 shown includes the following steps:

[0101] Step 501: Pre-set a distinguishing status code based on whether the task logic is invoked, wherein the distinguishing status code includes an invocation status code and a non-invocation status code;

[0102] Step 502: Set the distinguishing status code corresponding to the unenabled task logic to the non-call status code.

[0103] By setting distinct status codes, unused task logic is set to an uncallable state, preventing the current batch task from calling the corresponding code logic.

[0104] Step 205: Obtain the optimized and adjusted task logic, and deploy the task logic and the current configuration file to m batch processing systems, where m is the total number of batch processing systems and m is a positive integer.

[0105] When continuing to calculate regional summaries based on insurance policy fees, taking the calculation of the total insurance claims amount and the total amount of new insurance contracts signed in each province for each quarter as an example, by deploying the task logic and the current configuration file to m batch processing systems, a cluster is built. Using multiple systems to execute batch processing tasks simultaneously reduces batch processing time to a certain extent and avoids the problem of excessive service pressure when a single system executes batch processing tasks.

[0106] Step 206: Obtain the n fee change information entries based on the extraction address, and perform fragmentation processing on the n fee change information entries using preset fragmentation rules.

[0107] When continuing to calculate regional summaries of insurance policy fees, taking the calculation of the total amount of insurance claims and the total amount of new insurance contracts signed in each province for each quarter as an example, by segmenting several fee change information, the segmented fee change information is sent to different batch processing systems for task processing, which further reduces batch processing time and avoids the problem of excessive service pressure when a single system executes batch processing tasks.

[0108] In this embodiment, the step of obtaining the n cost change information entries based on the extraction address and performing fragmentation processing on the n cost change information entries using a preset fragmentation rule specifically includes: setting a sequence number for each obtained cost change information entry, wherein the sequence number is a positive integer and the sequence number is less than or equal to n; performing fragmentation processing on the n cost change information entries based on a preset modulo formula: b = a % m, obtaining the fragmentation number corresponding to each cost change information entry, wherein m is the total number of the batch processing system, a is the sequence number corresponding to the current cost change information entry, and b is the modulo value, i.e., the fragmentation number corresponding to the current cost change information entry with sequence number a; constructing a cost change information set using the fragmentation number as a set identifier; and caching single cost change information entries with the same fragmentation number as set elements in the corresponding cost change information set.

[0109] Assuming there are 5 batch processing systems (m = 5), a fixed value, and the number of cost change messages to be batch processed is uncertain (n can be 41, 344, or 999), taking n = 41 as an example, to distribute these 41 cost change messages among the 5 batch processing systems, a modulo formula b = a % 5 is used to segment the 41 cost change messages into 5 parts, each corresponding to one batch processing system. First, the first cost change message is retrieved, with the number 1 (b = 1 % 5 = 1). Then, the second cost change message is retrieved, with the number 2 (b = 2 % 5 = 2), and so on, until the 6th cost change message is retrieved, with the number 6 (b = 6 % 5 = 1), until the 41st cost change message is retrieved, with the number 41 (b = 41 % 5 = 1). Thus, the value of b, i.e., the modulus, can be 0, 1, 2, 3, or 4, and this modulus is used as the segment number. The fee change information numbered 1, 6, 11, 16, 21, 26, 31, 36, and 41, with a modulus of 1, will be placed into the set corresponding to segment number 1. Similarly, the fee change information numbered 2, 7, 12, 17, 22, 27, 32, and 37, with a modulus of 2, will be placed into the set corresponding to segment number 2. The fee change information numbered 3, 8, 13, 18, 23, 28, 33, and 38, with a modulus of 3, will be placed into the set corresponding to segment number 3. The fee change information numbered 4, 9, 14, 19, 24, 29, 34, and 39, with a modulus of 4, will be placed into the set corresponding to segment number 4. The fee change information numbered 5, 10, 15, 20, 25, 30, 35, and 40, with a modulus of 0, will be placed into the set corresponding to segment number 0.

[0110] Continue to refer to Figure 6 , Figure 6 yes Figure 2A flowchart of a specific implementation of step 206 shown includes the following steps:

[0111] Step 601: Set a serial number for each acquired cost change information, wherein the serial number is a positive integer and the serial number is less than or equal to n;

[0112] Step 602: Based on the preset modulo formula: b = a % m, the n cost change information is segmented to obtain the segment number corresponding to each cost change information, where m is the total number of the batch processing system, a is the sequence number corresponding to the current cost change information, and b is the modulo value, that is, the segment number corresponding to the current cost change information with sequence number a.

[0113] Step 603: Construct a set of cost change information using the segment number as the set identifier;

[0114] Step 604: Cache the single cost change information with the same segment number as a set element in the corresponding cost change information set.

[0115] By using the number of batch processing systems as a fixed value and the number of cost change information as an indefinite value, the modulus between the indefinite value and the fixed value is obtained through a preset modulo formula. By setting a sequence number for each cost change information, its corresponding modulus is obtained. Cost change information with the same modulus is placed in the same set, corresponding to the corresponding batch processing system, so that the batch processing system can execute the cost change information after the segmentation according to the segmentation processing result.

[0116] Step 207: parse the cost change information after sharding, and input the parsing results as method execution parameters into the m batch processing systems to execute the batch processing task based on the sharding rules.

[0117] In this embodiment, the step of parsing the cost change information after sharding and inputting the parsing results as method execution parameters into the m batch processing systems to execute the batch processing task based on the sharding rules specifically includes: parsing different set elements within the cost change information set, and assigning a corresponding set identifier to each parsing result after parsing; setting a distinction number for the m batch processing systems, wherein the distinction number is an integer greater than or equal to 0, and its value range is [0, m); obtaining each parsing result corresponding to the set identifier with the same distinction number, and inputting each parsing result as a method execution parameter into the batch processing system corresponding to the distinction number to execute the batch processing task.

[0118] Continue to refer to Figure 7 , Figure 7 yes Figure 2 A flowchart of a specific implementation of step 207 shown includes the following steps:

[0119] Step 701: parse the different set elements in the set of cost change information, and after parsing, assign a corresponding set identifier to each parsing result;

[0120] Step 702: Set a distinguishing number for the m batch processing systems, wherein the distinguishing number is an integer greater than or equal to 0, and the value range is [0, m);

[0121] Step 703: Obtain each parsing result corresponding to the set identifier that is the same as the difference number, and input each parsing result as a method execution parameter into the batch processing system corresponding to the difference number to execute the batch processing task.

[0122] This application optimizes batch processing by receiving batch processing requests; obtaining configuration files; identifying unused configuration text segments in the current batch processing; optimizing the batch processing task logic; deploying the optimized task logic and configuration files to several batch processing systems; segmenting cost change information; parsing each cost change message; and using the parsing results as method execution parameters to execute the batch processing task. This application optimizes the task logic through configuration files and segments all cost change information using a modulo method, ensuring that each cost change message is distributed to its corresponding batch processing system. This optimizes batch processing, improves batch processing efficiency, ensures reasonable utilization of server resources, and avoids system crashes caused by batch processing events.

[0123] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0124] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0125] In this embodiment, all cost change information can be obtained from different data sources through big data processing technology, or the cost change information after fragmentation can be stored through distributed storage, ensuring the convenience and efficiency of obtaining all cost change information and storing each piece of cost change information after fragmentation.

[0126] Further reference Figure 8 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a task batch processing device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0127] like Figure 8 As shown, the task batch processing device 800 described in this embodiment includes: a request receiving module 801, a configuration acquisition module 802, a filtering and identification module 803, an optimization and adjustment module 804, a cluster deployment module 805, a data sharding module 806, and a task execution module 807. Wherein:

[0128] The request receiving module 801 is used to receive batch processing requests corresponding to fee changes, wherein the batch processing request includes the extraction addresses of n fee change information to be batch processed, where n is a positive integer;

[0129] The configuration acquisition module 802 is used to read the target database and acquire the initial configuration file and the current configuration file of the batch processing, wherein the target database includes the initial configuration file and the current configuration file of the batch processing;

[0130] The filtering and identification module 803 is used to identify configuration text segments that are not used in this batch processing from the initial configuration file based on preset filtering rules;

[0131] The optimization and adjustment module 804 is used to optimize and adjust the task logic for batch processing based on the configuration text segment;

[0132] The cluster deployment module 805 is used to obtain the optimized and adjusted task logic, and deploy the task logic and the current configuration file to m batch processing systems, where m is the total number of batch processing systems and m is a positive integer;

[0133] The data sharding module 806 is used to obtain the n cost change information entries according to the extraction address, and to shard the n cost change information entries using a preset sharding rule;

[0134] The task execution module 807 is used to parse each cost change information after the sharding process, and input the parsing result as a parameter into the m batch processing systems to execute the batch processing task based on the sharding rules.

[0135] This application optimizes batch processing by receiving batch processing requests; obtaining configuration files; identifying unused configuration text segments in the current batch processing; optimizing the batch processing task logic; deploying the optimized task logic and configuration files to several batch processing systems; segmenting cost change information; parsing each cost change message; and using the parsing results as method execution parameters to execute the batch processing task. This application optimizes the task logic through configuration files and segments all cost change information using a modulo method, ensuring that each cost change message is distributed to its corresponding batch processing system. This optimizes batch processing, improves batch processing efficiency, ensures reasonable utilization of server resources, and avoids system crashes caused by batch processing events.

[0136] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0137] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0138] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0139] The computer device 9 includes a memory 91, a processor 92, and a network interface 93 that are interconnected via a system bus. It should be noted that only the computer device 9 with components 91-93 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0140] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0141] The memory 91 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 91 may be an internal storage unit of the computer device 9, such as the hard disk or memory of the computer device 9. In other embodiments, the memory 91 may also be an external storage device of the computer device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 9. Of course, the memory 91 may include both the internal storage unit and its external storage device of the computer device 9. In this embodiment, the memory 91 is typically used to store the operating system and various application software installed on the computer device 9, such as computer-readable instructions for task batch processing methods. In addition, the memory 91 can also be used to temporarily store various types of data that have been output or will be output.

[0142] In some embodiments, the processor 92 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 92 is typically used to control the overall operation of the computer device 9. In this embodiment, the processor 92 is used to execute computer-readable instructions stored in the memory 91 or to process data, for example, to execute computer-readable instructions of the task batch processing method.

[0143] The network interface 93 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 9 and other electronic devices.

[0144] The computer device proposed in this embodiment belongs to the field of big data technology. This application receives batch processing requests; obtains configuration files; identifies unused configuration text segments in the current batch processing; optimizes and adjusts the batch processing task logic; deploys the optimized task logic and configuration files to several batch processing systems; segments cost change information; parses each cost change message, and uses the parsing results as method execution parameters to execute the batch processing task. This application optimizes the task logic through configuration files and segments all cost change information using a modulo method, ensuring that each cost change message is allocated to its corresponding batch processing system. This optimizes batch processing, improves batch processing efficiency, ensures reasonable utilization of server resources, and avoids system crashes caused by batch processing events.

[0145] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the task batch processing method described above.

[0146] The computer-readable storage medium proposed in this embodiment belongs to the field of big data technology. This application receives batch processing requests; obtains configuration files; identifies unused configuration text segments in the current batch processing; optimizes and adjusts the batch processing task logic; deploys the optimized task logic and configuration files to several batch processing systems; performs fragmented processing on cost change information; parses each cost change information, and uses the parsing results as method execution parameters to execute the batch processing task. This application optimizes and adjusts the task logic through configuration files and uses a modulo method to fragment all cost change information, ensuring that each cost change information is allocated to its corresponding batch processing system. This optimizes batch processing, improves batch processing efficiency, ensures reasonable utilization of server resources, and avoids system crashes caused by batch processing events.

[0147] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0148] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A task batch processing method, characterized in that, Includes the following steps: Receive batch processing requests corresponding to fee changes, wherein the batch processing requests include requests to be processed in batches. Address for retrieving fee change information. It is a positive integer; Read the configuration file database and obtain the initial configuration file and current configuration file of the batch processing according to the distinguishing cache identifier of the target configuration file. The configuration file database caches the target configuration file, which includes the initial configuration file and the current configuration file of the batch processing. The initial configuration file and the current configuration file of the batch processing are preset with distinguishing cache identifiers. Based on preset filtering rules, configuration text segments not used in this batch processing are identified from the initial configuration file; Based on the configuration text segment, the task logic for batch processing is optimized and adjusted. The batch processing task logic is a batch processing source execution program pre-deployed on the task execution terminal corresponding to the batch processing task, specifically including: Read the preset repository to obtain the call relationship corresponding to the configuration text segment; Based on the aforementioned call relationships, the task logic that has not been enabled in the batch processing task logic is filtered out; Based on the preset task call status code, the inactive task logic is set to an uncallable state; Obtain the optimized and adjusted task logic, and deploy the task logic and the current configuration file to... A batch processing system, among which... The total number of batch processing systems. It is a positive integer; Obtain the [data] based on the extraction address. The fee change information is processed using preset segmentation rules. Fee change information is processed in segments; The system parses each cost change record after sharding and inputs the parsing results as method execution parameters based on the sharding rules. Each batch processing system executes batch processing tasks.

2. The task batch processing method according to claim 1, characterized in that, The step of identifying unused configuration text segments from the initial configuration file based on preset filtering rules specifically includes: Parse the initial configuration file to obtain the initial script content; Parse the current configuration file to obtain the current script content; Based on the initial script content and the current script content, the script content that is not used in the current configuration file during the execution of this batch processing task is selected as the unused configuration text segment in this batch processing process.

3. The task batch processing method according to claim 2, characterized in that, Before the step of optimizing and adjusting the task logic for batch processing based on the configuration text segment, the method further includes: The calling relationship between the initial script content and the batch processing task logic is set in advance, wherein the batch processing task logic is used to execute the corresponding cost information update task with the corresponding cost change information as a parameter; The call relationship is persistently cached in the form of a form to a preset repository.

4. The task batch processing method according to claim 1, characterized in that, Before the step of setting the inactive task logic to an uncallable state based on a preset task invocation status code, the method further includes: A distinguishing status code is pre-set based on whether the task logic is invoked, wherein the distinguishing status code includes an invoked status code and a non-invoked status code; The step of setting the inactive task logic to an uncallable state based on a preset task invocation status code specifically includes: Set the distinguishing status code corresponding to the inactive task logic to the non-call status code.

5. The task batch processing method according to claim 1, characterized in that, The step of obtaining the extraction address The fee change information is processed using preset segmentation rules. The steps for segmenting and processing fee change information include: Each acquired fee change record is assigned a serial number, wherein the serial number is a positive integer and is less than or equal to... ; Based on the preset modulo formula: Regarding the above Each fee change message is segmented for processing, and the segment number corresponding to each fee change message is obtained. The total number of batch processing systems. This is the sequence number corresponding to the current fee change information. The modulus value; A set of cost change information is constructed using the segment number as the set identifier; Individual cost change information with the same segment number is cached as a set element in the corresponding cost change information set.

6. The task batch processing method according to claim 5, characterized in that, The parsing process retrieves each cost change record after fragmentation, and the parsing results are used as method execution parameters based on the fragmentation rules. The steps a batch processing system takes to execute a batch processing task specifically include: The different set elements within the cost change information set are parsed, and after parsing, a corresponding set identifier is assigned to each parsing result; Regarding the Each batch processing system is configured with a distinguishing number, wherein the distinguishing number is an integer greater than or equal to 0, and its value range is [value range missing]. ; Obtain each parsing result corresponding to the set identifier that has the same difference number, and input each parsing result as a method execution parameter into the batch processing system corresponding to the difference number to execute the batch processing task.

7. A task batch processing device, characterized in that, The task batch processing device implements the steps of the task batch processing method as described in any one of claims 1 to 6, and the task batch processing device includes: The request receiving module is used to receive batch processing requests corresponding to fee changes, wherein the batch processing requests include requests to be processed in batches. Address for retrieving fee change information. It is a positive integer; The configuration acquisition module is used to read the configuration file database and obtain the initial configuration file and the current configuration file of the batch processing according to the distinguishing cache identifier of the target configuration file. The configuration file database caches the target configuration file, and the target configuration file includes the initial configuration file and the current configuration file of the batch processing. The initial configuration file and the current configuration file of the batch processing are preset with distinguishing cache identifiers. The filtering and identification module is used to identify configuration text segments that are not used in this batch processing from the initial configuration file based on preset filtering rules; The optimization and adjustment module is used to optimize and adjust the task logic for batch processing based on the configuration text segment, wherein the task logic for batch processing is a batch processing source execution program pre-deployed on the task execution terminal corresponding to the task batch processing. The cluster deployment module is used to obtain the optimized and adjusted task logic and deploy the task logic and the current configuration file to... A batch processing system, among which... The total number of batch processing systems. It is a positive integer; The data sharding module is used to obtain the data based on the extraction address. The fee change information is processed using preset segmentation rules. Fee change information is processed in segments; The task execution module is used to parse each cost change information after sharding, and input the parsing results as method execution parameters based on the sharding rules. Each batch processing system executes batch processing tasks.

8. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the task batch processing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the task batch processing method as described in any one of claims 1 to 6.

Images