Task data configuration method, device, electronic device and storage medium

Abstract

The present invention relates to the field of data request processing and discloses a task data configuration method, comprising determining N types of data that need to be configured based on key fields of a data task to be processed, generating N configuration tasks based on the N types of data; calculating the load value of each service node based on operating indicator data, screening multiple service nodes with load values ​​less than a preset threshold and dividing them into N node groups, and associating each node group with each configuration task in a one-to-one correspondence; constructing a storage document for storing configuration data for each configuration task in a preset database, creating a transmission channel between each storage document and the corresponding node group; using the transmission channel to download the configuration data from the corresponding node group to the corresponding storage document, performing calculation processing on each configuration data, and writing the calculation result into the data task to be processed. The present invention decomposes and assigns tasks to different node groups for processing, thereby improving the processing speed and stability of data configuration.

Description

Technical Field

[0001] The present invention relates to the field of data request processing, and in particular to a task data configuration method, device, electronic device and storage medium. Background Art

[0002] In the field of data configuration processing, data acquisition and processing efficiency are crucial for real-time performance and resource utilization.

[0003] Especially in insurance and finance scenarios, when data tables (data tasks) involving multiple insurance products require data configuration for each insurance product, traditional methods usually rely on sending data configuration requests to each service node in sequence according to the order of the insurance product list. This serial request method is prone to factors such as network delays, resulting in slow data loading speed.

[0004] In addition, existing technical solutions often store the collected configuration data in a cache database, and need to wait for the entire configuration data to be downloaded before calculating and processing the entire configuration data. Due to the limited capacity of the cache database, it may face the risk of overload, thereby affecting the stability of data configuration processing.

[0005] Therefore, how to improve the processing speed and stability of data configuration is a technical problem that needs to be solved urgently. Summary of the Invention

[0006] In view of the above, it is necessary to provide a task data configuration method, the purpose of which is to improve the processing speed and stability of data configuration by converting data configuration requests into configuration tasks that can be processed in parallel and intelligently allocating configuration tasks according to the current load of each service node.

[0007] The task data configuration method provided by the present invention includes:

[0008] receiving a request for configuring data for a data task to be processed, identifying key fields of the data task to be processed, determining N types of data that need to be configured for the data task to be processed based on the identified N key fields, and generating N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0009] Obtaining operational indicator data for each service node in a preset server cluster, calculating a load value for each service node based on the operational indicator data, screening multiple service nodes with load values ​​less than a preset threshold and dividing them into N node groups, and associating each node group with each configuration task in a one-to-one correspondence;

[0010] Build a storage document for storing configuration data for each configuration task in the preset database, and create a transmission channel between each storage document and the corresponding node group;

[0011] Utilize the transmission channel corresponding to each configuration task to download the configuration data from the corresponding node group to the corresponding storage document, perform calculation processing on the configuration data of each storage document, and write the calculation result into the data task to be processed.

[0012] Optionally, the key fields for identifying the data task to be processed include:

[0013] The key fields for identifying the data task to be processed include:

[0014] Dividing the data task to be processed into a plurality of data segments according to the chapter titles of the data task to be processed, wherein each data segment includes at least one chapter title and a text content under the at least one chapter title;

[0015] A text containing a preset key field is read from each data segment, and the preset key field is identified from the read text.

[0016] Optionally, obtaining the operating indicator data of each service node in the preset server cluster includes:

[0017] Obtaining a node monitoring log from a preset monitoring tool in the preset server cluster, wherein the node monitoring log records operating indicator data of each service node;

[0018] According to the identifier of each service node, the operation indicator data of each service node is extracted from the node monitoring log.

[0019] Optionally, the operating indicator data includes CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks, and running speed of executed tasks. Calculating the load value of each service node based on the operating indicator data includes:

[0020] Randomly select the first and second types of operating indicator data from the above operating indicator data and substitute them into the preset load value formula to calculate the load value of each service node;

[0021] Among them, the load value formula is L t =W1×A t +W2×B i , where W1 and W2 are given weight factors, A i is the first type of operating indicator data of the i-th service node, B iIt is the second type of operating indicator data of the i-th service node.

[0022] Optionally, calculating the load value of each service node according to the operation indicator data includes:

[0023] Preprocessing the operating indicator data, wherein the preprocessing includes removing invalid or erroneous data;

[0024] The preprocessed operation index data is converted into eigenvalues, and the eigenvalues ​​corresponding to all operation indicators are collected to obtain the corresponding eigenvalue array. The eigenvalue array is calculated using the trained load prediction model, and a preset weight is assigned to each eigenvalue in the eigenvalue array. The weighted summation of the eigenvalue array after weight assignment is performed to obtain the load value of each service node.

[0025] Optionally, screening multiple service nodes having load values ​​less than a preset threshold and dividing them into N node groups includes:

[0026] Read the load value of each service node and select multiple service nodes whose load value is less than a preset threshold;

[0027] According to the N configuration tasks, multiple service nodes are evenly divided into corresponding N node groups.

[0028] Optionally, creating a transmission channel between each stored document and the corresponding node group includes:

[0029] Based on the GET method of the preset transmission protocol, a separate GET request is configured for each stored document;

[0030] The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

[0031] In order to solve the above problems, the present invention further provides a task data configuration device, the device comprising:

[0032] a receiving module, configured to receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed based on the identified N key fields, and generate N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0033] A calculation module is used to obtain operating indicator data of each service node in a preset server cluster, calculate the load value of each service node based on the operating indicator data, screen multiple service nodes with load values ​​less than a preset threshold and divide them into N node groups, and associate each node group with each configuration task in a one-to-one correspondence;

[0034] A construction module is used to construct a storage document for storing configuration data for each configuration task in a preset database, and to create a transmission channel between each storage document and the corresponding node group;

[0035] The writing module is used to use the transmission channel corresponding to each configuration task to download the configuration data from the corresponding node group to the corresponding storage document, perform calculation processing on the configuration data of each storage document, and write the calculation results into the data task to be processed.

[0036] In order to solve the above problem, the present invention further provides an electronic device, comprising:

[0037] at least one processor; and,

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores a task data configuration program that can be executed by the at least one processor. The task data configuration program is executed by the at least one processor to enable the at least one processor to perform the above-mentioned task data configuration method.

[0040] In order to solve the above problems, the present invention also provides a computer-readable storage medium, on which a task data configuration program is stored. The task data configuration program can be executed by one or more processors to implement the above task data configuration method.

[0041] Compared with the existing technology, the present invention divides the data task to be processed into multiple configuration tasks based on the key fields of the data task to be processed. By decomposing the tasks and processing them in parallel, it avoids the network delay problem caused by the traditional serial request method and greatly improves the speed of data loading and processing.

[0042] By monitoring the operating indicator data of each service node in real time, multiple service nodes with load values ​​less than the preset threshold are screened and divided into N node groups. Each node group is associated with each configuration task one by one. Through the load balancing algorithm, overload of some nodes is avoided, ensuring the stable operation of the download task.

[0043] A separate storage document is created for each configuration task, and a transmission channel is established between each storage document and the corresponding node group. Using the transmission channel corresponding to each configuration task, the configuration data is downloaded from the corresponding node group to the corresponding storage document. The configuration data in each storage document is then calculated and processed, and the results are written to the pending data task. This download-as-you-process approach reduces reliance on the cache database and avoids the risk of overload due to its limited capacity. By breaking down tasks and assigning them to different node groups, the processing speed and stability of data configuration are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a task data configuration method provided by one embodiment of the present invention;

[0045] Figure 2 A schematic diagram of modules of a task data configuration device provided by one embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of an electronic device for implementing a task data configuration method provided by an embodiment of the present invention;

[0047] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] It should be noted that the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0050] Reference Figure 1 FIG. 1 is a flow chart of a task data configuration method according to an embodiment of the present invention. The method is executed by an electronic device.

[0051] In this embodiment, the task data configuration method includes:

[0052] S1. Receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed based on the identified N key fields, and generate N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID.

[0053] In this embodiment, a request is received from a user or an application to configure data for a data task to be processed. After receiving the request, the data task to be processed needs to be analyzed and key fields of the data task to be processed need to be identified.

[0054] A pending data task refers to one or a group of tasks or data tables that require data configuration or updates. For example, in an insurance and finance scenario, a salesperson uploads a pending data task, which is an insurance policy to be associated. The policy only contains different types of insurance products. However, the various association tables that store detailed information about these insurance products (such as premium calculation rules, claim conditions, product terms, etc.) are distributed across various service nodes in a pre-set server cluster. Therefore, it is necessary to obtain the configuration data of the policy to be associated from the various association tables of each service node, and use the policy to be associated to process the various types of insurance products in the policy, such as supplementing detailed information, verifying information, or calculating the price of insurance products.

[0055] The key field of the data task to be processed is used to distinguish different data items or data types in the data task to be processed. For example, the key field can be the ID or name of an insurance product.

[0056] Based on the identified key fields, determine how many different data items or data types the data task to be processed contains. For example, if the identified key fields include the names of auto insurance, personal accident insurance, and property insurance, then the data task to be processed needs to be configured to include data items or data types such as auto insurance, personal accident insurance, and property insurance.

[0057] Based on the identified key fields, the N types of data that need to be configured for the data task to be processed are determined, and a separate configuration task is generated for each type of data. This means that if there are N different key fields, N configuration tasks will be generated.

[0058] Compared to traditional methods that typically rely on sending data configuration requests to each service node sequentially according to the insurance product list, this serial request method is prone to factors such as network latency, resulting in slow data loading. This invention breaks down a large data configuration task into multiple smaller tasks based on data type, allowing them to be processed in parallel on multiple service nodes, thereby increasing data processing speed.

[0059] In one embodiment, identifying the key fields of the data task to be processed includes:

[0060] Dividing the data task to be processed into a plurality of data segments according to the chapter titles of the data task to be processed, wherein each data segment includes at least one chapter title and a text content under the at least one chapter title;

[0061] A text containing a preset key field is read from each data segment, and the preset key field is identified from the read text.

[0062] According to the chapter title name in the data task to be processed, the content of the data task to be processed is divided into multiple data segments, and the text containing preset key fields (such as configuration data name or configuration data ID) is filtered out from each data segment, and the preset key fields are identified from the read text.

[0063] Assume that the predefined key fields include "Insurance Product ID" and "Product Name." You need to filter the parsed segments to find text containing these key fields. For example, the key fields in segment 1 are: Insurance Product ID: 001, Product Name: Auto Insurance. The key fields in segment 2 are: Insurance Product ID: 002, Product Name: Personal Accident Insurance. Both segments 1 and 2 contain these two fields, so they are retained.

[0064] By parsing the chapter title names in the data task and splitting the content into multiple texts, you can more accurately identify which parts need to be configured and more precisely locate the specific fields that need to be updated, avoiding unnecessary data processing.

[0065] S2. Obtain the operating indicator data of each service node in the preset server cluster, calculate the load value of each service node based on the operating indicator data, screen multiple service nodes with load values ​​less than a preset threshold and divide them into N node groups, and associate each node group with each configuration task in a one-to-one correspondence.

[0066] In this embodiment, the node monitoring log is obtained from a preset monitoring tool in the preset server cluster. The preset monitoring tool can be a Prometheus monitoring and alarm system. The node monitoring log records the operating indicator data of each service node at the current moment. By obtaining the operating indicator data of each service node in real time through a monitoring tool (such as Prometheus), the current status of each service node can be accurately understood, and configuration tasks can be intelligently allocated according to the actual load of the service node, avoiding the situation where some nodes are overloaded and other nodes are idle.

[0067] According to the identifier of each service node, the operating indicator data of each service node is extracted from the node monitoring log, and multiple service nodes with load values ​​less than a preset threshold (for example, the preset threshold is 0.75) are screened and divided into N node groups. This can more effectively utilize the resources in the cluster, and the service nodes in each node group can efficiently process the tasks assigned to them, which can achieve better load balancing. It can not only avoid overloading of a single node, but also make full use of the available resources in the cluster and improve the resource utilization of the overall system.

[0068] By mapping configuration tasks to node groups one by one, we can ensure that each configuration task has a dedicated node group to handle it. Each node group only needs to focus on the part of the configuration task it is responsible for, avoiding interference between tasks and improving the accuracy and efficiency of task processing.

[0069] In one embodiment, screening multiple service nodes whose load values ​​are less than a preset threshold and dividing them into N node groups includes:

[0070] Read the load value of each service node and select multiple service nodes whose load value is less than a preset threshold;

[0071] According to the N configuration tasks, multiple service nodes are evenly divided into corresponding N node groups.

[0072] Multiple service nodes with load values ​​less than the preset threshold are evenly divided into N node groups. By evenly distributing them, there is no need for complex calculations or additional resource allocation logic. To a certain extent, it can ensure that the load levels of all service nodes are roughly the same, thus avoiding the problem of single-point overload.

[0073] In other implementations, multiple service nodes with load values ​​less than a preset threshold may be screened and divided into N node groups in the following manner:

[0074] Method 1: Assign a priority to multiple service nodes whose load values ​​are less than a preset threshold. For example, service nodes with a load value of 0.10-0.29 are assigned a high priority, service nodes with a load value of 0.30-0.49 are assigned a medium priority, and service nodes with a load value of 0.50-0.69 are assigned a low priority. First, divide the service nodes with high priority into N node groups. If the number of service nodes with high priority is not enough to be divided, then call from the service nodes in the priority. Nodes with high priority are usually nodes with better performance and can complete tasks faster, thereby improving the overall system performance. The priority can be dynamically adjusted according to changes in load, making the system more flexible and adaptable to different workloads.

[0075] Method 2: Filter M service nodes with load values ​​below a preset threshold, divide these M service nodes into N node groups, where M is greater than N, monitor the load of these M service nodes in real time, and dynamically adjust the task allocation strategy based on actual conditions. For example, if the load on a node suddenly increases, its task load can be immediately reduced and transferred to other nodes with lower loads. By continuously monitoring node status, task allocation can be adjusted in real time to ensure that the system load is always maintained at an optimal state. Overloaded or faulty nodes can be quickly detected and addressed, and tasks can be transferred in a timely manner, improving the system's fault tolerance and availability.

[0076] The present invention does not limit which method to choose. At least the method to choose depends on the specific requirements of the actual application scenario, such as the system's performance requirements, load volatility, and the importance attached to stability.

[0077] In one embodiment, obtaining the operating indicator data of each service node in the preset server cluster includes:

[0078] Obtaining a node monitoring log from a preset monitoring tool in the preset server cluster, wherein the node monitoring log records operating indicator data of each service node;

[0079] According to the identifier of each service node, the operation indicator data of each service node is extracted from the node monitoring log.

[0080] Use preset monitoring tools (such as the Prometheus monitoring and alarm system) to monitor the entire server cluster, and use preset monitoring tools to capture the operating indicator data of each service node. The operating indicator data includes but is not limited to CPU usage, memory usage, disk I / O read and write speed, network bandwidth usage, number of unexecuted tasks, and the running speed of executed tasks.

[0081] The preset monitoring tool regularly collects the above-mentioned operating indicator data from each service node and stores these operating indicator data in its own time series database. These operating indicator data constitute the node monitoring log, thereby realizing the extraction of the operating indicator data of each service node from the node monitoring log according to the identifier of each service node.

[0082] By extracting operational metrics from monitoring logs, we can monitor the operational status of each service node in real time, providing instant access to the latest status information. This allows us to make timely decisions, such as whether to assign a configuration task to a specific node. We can also promptly identify potential issues, such as abnormal node resource usage and increased network latency, and take measures to prevent these issues from affecting data processing. This helps maintain stable system operation.

[0083] In one embodiment, the operating indicator data includes CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks, and running speed of executed tasks. Calculating the load value of each service node based on the operating indicator data includes:

[0084] Randomly select the first and second types of operating indicator data from the above operating indicator data and substitute them into the preset load value formula to calculate the load value of each service node;

[0085] Among them, the load value formula is L i =W1×A i +W2×B i , where W1 and W2 are given weight factors, A i is the first type of operating indicator data of the i-th service node, B o It is the second type of operating indicator data of the i-th service node.

[0086] Obtaining operational metrics from monitoring tools for each service node and substituting either the first or second type of operational metrics into a pre-set load value formula can more accurately reflect the actual load of the service node. This allows for intelligent task allocation based on the current load of the service node. For example, service nodes with lower load values ​​can be prioritized for new configuration tasks.

[0087] For example:

[0088] There are two service nodes Node1 and Node2. CPU usage and memory usage are selected as calculation indicators, and it is assumed that W1 = 0.5 and W2 = 0.5.

[0089] Node1's CPU usage is 50% and memory usage is 40%.

[0090] Node2's CPU usage is 70% and memory usage is 50%.

[0091] According to the load value formula L i =W1×A i +W2×B i , we can calculate:

[0092] Node1's load value L Node1 =0.5×0.5+0.5×0.4=0.45 or 45%.

[0093] Node2's load value L Node2 =0.5×0.7+0.5×0.5=0.6 or 60%.

[0094] In this case, the load on Node1 is lower, so new configuration tasks can be assigned to Node1 first. In this way, the load of service nodes can be evaluated and managed more accurately, thereby achieving efficient resource allocation and task processing.

[0095] In one embodiment, the calculating the load value of each service node according to the operation indicator data includes:

[0096] Preprocessing the operating indicator data, wherein the preprocessing includes removing invalid or erroneous data;

[0097] The preprocessed operation index data is converted into eigenvalues, and the eigenvalues ​​corresponding to all operation indicators are collected to obtain the corresponding eigenvalue array. The eigenvalue array is calculated using the trained load prediction model, and a preset weight is assigned to each eigenvalue in the eigenvalue array. The weighted summation of the eigenvalue array after weight assignment is performed to obtain the load value of each service node.

[0098] Operational metric data includes CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks, and the running speed of executed tasks. This operational metric data is preprocessed, for example, by checking for outliers, missing values, or erroneous values ​​and addressing them. If the running speed of an executed task reported by a service node is negative or exceeds a reasonable range, it should be considered an outlier and corrected or deleted. If missing data exists, interpolation, forward filling, backward filling, or other statistical methods can be used to fill the missing values.

[0099] The pre-processed operating index data are converted into corresponding eigenvalues, as follows:

[0100] If the performance metrics are given in percentages (such as CPU usage or memory usage), you can use these values ​​directly. For example, if the CPU usage is 75%, you can use 0.75 as a feature value.

[0101] If the disk I / O read / write speed is given in bytes per second, you can convert it to a normalized form, such as converting it to read / write operations per second, or compare it with a benchmark speed to get a relative value.

[0102] If the network bandwidth usage is given in Mbps or Gbps, consider converting it to a percentage. For example, if the maximum bandwidth is 1Gbps and the current usage is 600Mbps, the converted percentage is 60%.

[0103] If the number of tasks executed is an absolute number, you can compare it to the maximum processing capacity of the node to get a relative value. For example, if the maximum processing capacity of the node is 100 tasks and 30 tasks are currently being executed, the converted percentage is 30%.

[0104] For each service node, the eigenvalues ​​corresponding to all the above operating indicators are collected to obtain an eigenvalue array. For example, for Node1, if its CPU utilization is 0.75, memory utilization is 0.4, disk I / O read / write speed is 0.6, network bandwidth utilization is 0.5, and the number of tasks executed is 0.3, the constructed eigenvalue array can be expressed as [0.75, 0.4, 0.6, 0.5, 0.3].

[0105] Input the feature value array of each service node into the trained load prediction model and assign a weight to each feature value in the feature value array. For example, if the load prediction model uses a simple linear regression model, the load value L can be calculated using the following formula:

[0106] L=w1·CPU+w2·Memory+w3·DiskIO+w4·Bandwidth+w5·Tasks

[0107] Among them, w1-w5 are feature weights, CPU, Memory, DiskIO, Bandwidth, and Tasks are eigenvalues ​​in the eigenvalue array. If it is determined that CPU usage has a greater impact on the load, it will be assigned a higher weight, while other features may be assigned lower weights.

[0108] After assigning a preset weight to each eigenvalue in the eigenvalue array, the eigenvalue array with assigned weights is linearly combined (such as weighted summation) or nonlinearly combined (such as through an activation function) to obtain the load value of each service node.

[0109] For example, if the load prediction model calculates a load value of 0.58 for Node 1, it can be considered that Node 1 is currently under medium load. If the load value is low, consider assigning new configuration tasks to this node. If the load value is high, avoid assigning new tasks to this node and instead look for a node with a lower load to handle the new configuration task. This approach achieves load balancing, ensures efficient utilization of cluster resources, and improves system stability and reliability.

[0110] The trained load prediction model automatically calculates the load value of each service node, reducing the need for manual intervention. It can process the latest operating indicator data in real time and provide load prediction results immediately, helping the system to respond quickly to load changes and enable the system to manage resources more intelligently.

[0111] The load prediction model training process includes:

[0112] Use monitoring tools to collect operational metrics data from service nodes, including but not limited to CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks, and the speed of executed tasks. Preprocess the operational metrics data and divide it into training and test sets (for example, a ratio of 70% training and 30% testing). This data can also be further divided into training, validation, and test sets.

[0113] Use a regression model as the initial load prediction model, such as linear regression, support vector machine (SVM), decision tree, random forest, gradient boosting tree (such as XGBoost or LightGBM), or neural network. Use the training set data to train the initial load prediction model. During training, adjust hyperparameters based on the performance of the initial load prediction model to optimize the model performance. To prevent overfitting, use cross-validation techniques to evaluate the generalization ability of the initial load prediction model.

[0114] Adjust the parameters of the initial load prediction model based on the verification results, repeat the training and verification process until you are satisfied, and obtain a trained load prediction model. Integrate the trained load prediction model into the production environment so that it can make predictions on real-time data streams.

[0115] In step S2, the operating status of the service nodes is monitored in real time. The node load values ​​are calculated based on the operating indicator data. Appropriate nodes are selected based on the load values ​​and divided into multiple groups, each handling a single configuration task. This prevents some nodes from being overloaded while others remain idle, achieving more efficient resource utilization, improving the system's load balancing capabilities, and enhancing system stability and reliability.

[0116] S3. Build a storage document for storing configuration data for each configuration task in the preset database, and create a transmission channel between each storage document and the corresponding node group.

[0117] In this embodiment, a shell script is created in a preset database. For example, the preset database may be a Hudi storage database.

[0118] In order to improve the reading speed and data storage space, the present invention optimizes the configuration data storage and reading performance through the indexing and compression technology of the Hudi storage database, which is very beneficial for real-time data processing and can speed up the data reading speed.

[0119] The name of each configuration task is used as a parameter of the shell script. The shell script based on the configuration parameters calls the operation command of the preset database, creates a corresponding storage document for each configuration task in the preset database, and configures a separate GET request for each storage document based on the GET method of the built-in transmission protocol of the preset database. The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

[0120] In one embodiment, constructing a storage document for storing configuration data for each configuration task includes:

[0121] Create a shell script in the preset database, and pass the name of each configuration task as a parameter to the shell script;

[0122] The shell script based on the configuration parameters calls the operation command of the preset database, and creates a corresponding storage document for each configuration task in the preset database.

[0123] Create a shell script in the pre-set database (for example, the Hudi storage database). The main function of the shell script is to receive the name of each configuration task and execute the operation of creating a storage document in the database based on the configuration task name. A shell script is a program written in the shell language of the Unix / Linux operating system. The shell is an interface between the user and the operating system, allowing users to interact with the system through the command line interface (CLI).

[0124] The shell script calls the preset database operation commands, which are used to create storage documents in the database. For each configuration task name passed to the script, the shell script will execute a series of commands to create a storage document for the task in the preset database.

[0125] Compared to existing solutions, which require waiting for the entire configuration data to be downloaded to the cache database before performing computations on it, if the configuration data volume is large, downloading the entire dataset to the cache database can take a long time. Even if only a small amount of data is missing, the entire processing flow needs to wait, resulting in longer computation startup times. While waiting for the data to be fully downloaded, computing resources may be idle and unable to begin processing the downloaded data, resulting in wasted resources.

[0126] The present invention constructs an independent storage document for each configuration task. It only needs to complete the download of the configuration data of the current storage document before the configuration data of the current storage document can be calculated and processed, without waiting for the entire configuration data to be downloaded. This allows multiple tasks to be processed in parallel without interfering with each other, which can significantly improve the speed and efficiency of data processing.

[0127] For example:

[0128] In existing solutions, the configuration data for all three products must be downloaded to the cache database before processing can begin. If the auto insurance data volume is the largest and the health insurance data volume is the smallest, then even if the health insurance data has already been downloaded, the system must wait for all data to be downloaded before processing can begin.

[0129] The method of the present invention creates a separate storage file for each configuration task (auto insurance, personal accident insurance, and health insurance). Once the health insurance configuration data is downloaded, the system immediately begins processing the health insurance data. Similarly, once the personal accident insurance data is downloaded, processing of the personal accident insurance data also begins. Finally, once the auto insurance data is downloaded, processing of the auto insurance data begins.

[0130] This allows for immediate processing of already downloaded data, even if some data has not yet been downloaded, significantly improving processing speed and efficiency. Furthermore, since each task is processed independently without interfering with others, the system can better utilize computing resources, reduce waiting time, and improve overall performance.

[0131] In one embodiment, creating a transmission channel between each stored document and the corresponding node group includes:

[0132] Based on the GET method of the preset transmission protocol, a separate GET request is configured for each stored document;

[0133] The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

[0134] A unique URI (Uniform Resource Identifier) ​​is constructed for each stored document. The URI usually contains the path, table name, partition information, etc. of the stored document, so as to facilitate the location of the specific document. Based on the GET method of the preset database built-in transmission protocol, a separate GET request is configured for each stored document.

[0135] Construct the necessary HTTP header information for the GET request, such as the header information including authentication information, content type, etc., add the URL to the GET request, and send the GET request to the corresponding node group in the HTTP client library. The node group receives the GET request sent by the Hudi storage database, and actually establishes a temporary transmission channel, which returns the configuration data according to the response format of the GET request (usually JSON or Parquet).

[0136] By distributing requests across multiple service nodes, even if a problem occurs on a service node, the overall data transmission process will not be affected, thus improving the stability and reliability of the system. If the load on a service node suddenly increases, dynamic adjustments can be made by redistributing GET requests to other service nodes with lower loads to ensure smooth data transmission.

[0137] In step S3, a separate storage document is created in the database for each configuration task, a transmission channel is established, and the configuration data is downloaded from the specified node group into the storage document via a GET request. This solves the traditional problem of waiting for the entire dataset to be downloaded before processing can begin. It enables simultaneous processing while downloading, improves processing efficiency, reduces pressure on the cache database, and avoids the risk of overload caused by limited cache capacity.

[0138] S4. Using the transmission channel corresponding to each configuration task, download the configuration data from the corresponding node group to the corresponding storage document, perform calculation processing on the configuration data of each storage document, and write the calculation results into the data task to be processed.

[0139] In this embodiment, after the transmission channel corresponding to each configuration task is determined, each configuration task will have its own independent transmission channel, ensuring the independence and efficiency of data transmission.

[0140] The configuration data is downloaded from the corresponding node group to the corresponding storage document through a GET request, thereby achieving parallel reception of the configuration data.

[0141] The configuration data is parsed using a pre-set parsing tool (this can be a corresponding library or tool, such as Python's json module). Based on the pre-set requirements of each configuration task, the parsed configuration data is processed to obtain calculation results. This processing includes data validation, data conversion, and aggregation operations. Specifically, data validation ensures that the configuration data is formatted correctly, for example, by checking whether the configuration data contains necessary fields, such as the insurance product ID and name. The content of the configuration data is verified to meet expectations, for example, by checking whether the premium is positive and whether the claim conditions are reasonable.

[0142] Data conversion converts configuration data into a format suitable for further processing. For example, a string value can be converted to an integer or floating-point number. If the configuration data contains values ​​in different units, they must be converted to a unified unit for comparison or calculation.

[0143] Aggregation operations aggregate configuration data based on the needs of the configuration task, such as calculating the average premium for a specific insurance product. Alternatively, they combine configuration data from different sources to form a complete configuration. For example, they can combine premium information with claim conditions for comprehensive analysis.

[0144] The calculation results are written into the data task to be processed. For example, the data task to be processed is the insurance policy to be associated. If detailed information of various insurance products in the insurance policy to be associated needs to be supplemented, the calculation results obtained are the detailed information of various insurance products, and the detailed information of various insurance products is supplemented into the insurance policy to be associated.

[0145] If it is necessary to verify whether the information of various insurance products in the insurance policy to be associated is correct, then the calculation result obtained is the verification result of the information of various insurance products, and the verification results of various insurance products are added to the insurance policy to be associated.

[0146] If it is necessary to calculate the product prices of various insurance products in the insurance policy to be associated, then the calculation results are the prices of various insurance products, and the prices of various insurance products are added to the insurance policy to be associated.

[0147] In step S4, the configuration data is downloaded to the storage file using the transmission channel. The configuration data in each storage file is then processed and the results are written back to the data task to be processed. By processing multiple configuration tasks in parallel, data processing speed is increased. Data validation, conversion, and aggregation operations ensure data accuracy and consistency, improving processing quality.

[0148] In summary, steps S1 through S4 work together to improve the speed and stability of data configuration processing, resolving issues such as network latency and slow data loading caused by traditional serial request methods. Dynamic load balancing addresses uneven resource utilization, while parallel processing and optimized data storage and transmission mechanisms further enhance the system's processing power and stability. These improvements are crucial for handling large, complex data configuration tasks, particularly in insurance and finance scenarios that demand high real-time performance and resource utilization.

[0149] like Figure 2 FIG. 1 is a module diagram of a task data configuration device provided by an embodiment of the present invention.

[0150] The task data configuration device 100 described in the present invention can be installed in an electronic device. Depending on the functionality to be implemented, the task data configuration device 100 may include a receiving module 110, a computing module 120, a constructing module 130, and a writing module 140. A module, also referred to as a unit, is a series of computer program segments that can be executed by an electronic device processor and perform a fixed function, and is stored in the electronic device's memory.

[0151] In this embodiment, the functions of each module / unit are as follows:

[0152] The receiving module 110 is configured to receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data to be configured for the data task to be processed based on the identified N key fields, and generate N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0153] The calculation module 120 is configured to obtain operating indicator data of each service node in a preset server cluster, calculate the load value of each service node based on the operating indicator data, select multiple service nodes with load values ​​less than a preset threshold, divide the nodes into N node groups, and associate each node group with each configuration task in a one-to-one correspondence;

[0154] A construction module 130 is configured to construct a storage document for storing configuration data for each configuration task in a preset database, and to create a transmission channel between each storage document and a corresponding node group;

[0155] The writing module 140 is used to use the transmission channel corresponding to each configuration task to download the configuration data from the corresponding node group to the corresponding storage document, perform calculation processing on the configuration data of each storage document, and write the calculation results into the data task to be processed.

[0156] like Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a task data configuration method according to an embodiment of the present invention.

[0157] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected via a system bus. The memory 11 stores a task data configuration program 10, and the task data configuration program 10 can be executed by the processor 12. Figure 3 Only the electronic device 1 having the components 11-13 and the task data configuration program 10 is shown. It can be understood by those skilled in the art that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and the electronic device 1 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0158] Memory 11 includes internal memory and at least one type of readable storage medium. The internal memory provides a cache for the operation of electronic device 1; the readable storage medium may be a non-volatile storage medium such as flash memory, a hard disk, a multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of electronic device 1, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. In this embodiment, the readable storage medium of memory 11 is typically used to store the operating system and various application software installed on electronic device 1, such as the code of the task data configuration program 10 in one embodiment of the present invention. Furthermore, memory 11 may also be used to temporarily store various types of data that has been output or is about to be output.

[0159] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to execute program code stored in the memory 11 or process data, such as executing the task data configuration program 10.

[0160] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a terminal (not shown in the figure).

[0161] Optionally, the electronic device 1 may further include a user interface, which may include a display and an input unit such as a keyboard. The optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0162] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0163] The task data configuration program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When executed in the processor 12, it can achieve the following:

[0164] receiving a request for configuring data for a data task to be processed, identifying key fields of the data task to be processed, determining N types of data that need to be configured for the data task to be processed based on the identified N key fields, and generating N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0165] Obtaining operational indicator data for each service node in a preset server cluster, calculating a load value for each service node based on the operational indicator data, screening multiple service nodes with load values ​​less than a preset threshold and dividing them into N node groups, and associating each node group with each configuration task in a one-to-one correspondence;

[0166] Build a storage document for storing configuration data for each configuration task in the preset database, and create a transmission channel between each storage document and the corresponding node group;

[0167] Utilize the transmission channel corresponding to each configuration task to download the configuration data from the corresponding node group to the corresponding storage document, perform calculation processing on the configuration data of each storage document, and write the calculation result into the data task to be processed.

[0168] Specifically, the specific implementation method of the processor 12 for the task data configuration program 10 can refer to Figure 1The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0169] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0170] The computer-readable storage medium stores a task data configuration program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the various embodiments of the above-mentioned task data configuration method, and will not be repeated here.

[0171] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical function division, and other division methods may be used in actual implementation.

[0172] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0173] In addition, the functional modules in various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or hardware plus software functional modules.

[0174] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0175] Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and all changes that come within the meaning and range of equivalents of the claims are intended to be embraced therein. Any reference to a figure in a claim should not be construed as limiting the claim to which it relates.

[0176] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. Second-order terms are used to indicate names and do not imply any particular order.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A task data configuration method, characterized in that: The method comprises: Receiving a request for data configuration for a data task to be processed, identifying key fields of the data task to be processed, determining N types of data that need to be configured for the data task to be processed based on the identified N key fields, and generating N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID; Obtaining operating indicator data of each service node in a preset server cluster, calculating a load value of each service node based on the operating indicator data, reading the load value of each service node, screening multiple service nodes whose load values ​​are less than a preset threshold, and evenly dividing the multiple service nodes into corresponding N node groups according to the N configuration tasks, and associating each node group with each configuration task in a one-to-one correspondence; Build a storage document for storing configuration data for each configuration task in the preset database, and create a transmission channel between each storage document and the corresponding node group; Using the transmission channel corresponding to each configuration task, download the configuration data from the corresponding node group to the corresponding storage file, perform calculation processing on the configuration data of each storage file, and write the calculation results into the data task to be processed; The key fields for identifying the data task to be processed include: Dividing the data task to be processed into a plurality of data segments according to the chapter titles of the data task to be processed, wherein each data segment includes at least one chapter title and a text content under the at least one chapter title; A text containing a preset key field is read from each data segment, and the preset key field is identified from the read text.

2. The task data configuration method according to claim 1, wherein: The step of obtaining the operating indicator data of each service node in the preset server cluster includes: Obtaining a node monitoring log from a preset monitoring tool in the preset server cluster, wherein the node monitoring log records operating indicator data of each service node; According to the identifier of each service node, the operation indicator data of each service node is extracted from the node monitoring log.

3. The task data configuration method according to claim 1, wherein: The operation indicator data includes CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks, and the running speed of executed tasks. The load value of each service node is calculated based on the operation indicator data, including: Randomly select the first and second types of operating indicator data from the above operating indicator data and substitute them into the preset load value formula to calculate the load value of each service node; Among them, the load value formula is: ,in, and is a given weight factor, It is i The first type of operating indicator data of a service node, It is i The second type of operating indicator data for each service node.

4. The task data configuration method according to claim 1, wherein: Calculating the load value of each service node according to the operation indicator data includes: Preprocessing the operating indicator data, wherein the preprocessing includes removing invalid or erroneous data; The preprocessed operation index data is converted into eigenvalues, and the eigenvalues ​​corresponding to all operation indicators are collected to obtain the corresponding eigenvalue array. The eigenvalue array is calculated using the trained load prediction model, and a preset weight is assigned to each eigenvalue in the eigenvalue array. The weighted summation of the eigenvalue array after weight assignment is performed to obtain the load value of each service node.

5. The task data configuration method according to claim 1, wherein: The step of creating a transmission channel between each stored document and the corresponding node group includes: Based on the GET method of the preset transmission protocol, a separate GET request is configured for each stored document; The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

6. A task data configuration device, used to execute the task data configuration method according to any one of claims 1 to 5, characterized in that: The device comprises: a receiving module, configured to receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed based on the identified N key fields, and generate N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID; A calculation module is used to obtain operating indicator data of each service node in a preset server cluster, calculate the load value of each service node based on the operating indicator data, screen multiple service nodes with load values ​​less than a preset threshold and divide them into N node groups, and associate each node group with each configuration task in a one-to-one correspondence; A construction module is used to construct a storage document for storing configuration data for each configuration task in a preset database, and to create a transmission channel between each storage document and the corresponding node group; The writing module is used to use the transmission channel corresponding to each configuration task to download the configuration data from the corresponding node group to the corresponding storage document, perform calculation processing on the configuration data of each storage document, and write the calculation results into the data task to be processed.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a task data configuration program executable by the at least one processor, and the task data configuration program is executed by the at least one processor to enable the at least one processor to perform the task data configuration method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a task data configuration program, which can be executed by one or more processors to implement the task data configuration method according to any one of claims 1 to 5.

Images