Task data configuration method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of data request processing, and discloses a task data configuration method, which comprises the following steps of: determining N types of data needing to be configured according to key fields of a data task to be processed, and generating N configuration tasks according to the N types of data; calculating a load value of each service node according to the operation index data, screening a plurality of service nodes of which the load values are smaller than a preset threshold value, dividing the service nodes into N node groups, and associating each node group with each configuration task in a one-to-one correspondence manner; constructing a storage document for storing configuration data for each configuration task in a preset database, and creating a transmission channel between each storage document and the corresponding node group; and downloading the configuration data from the corresponding node group to the corresponding storage document by using the transmission channel, performing calculation processing on each piece of configuration data, and writing a calculation result into the to-be-processed data task. According to the method, the task is decomposed and allocated to different node groups for processing, so that the processing speed and stability of data configuration are improved.

Description

Technical Field

[0001] The present invention relates to the field of data request processing, and in particular to a task data configuration method, device, electronic device and storage medium. Background Art

[0002] In the field of data configuration processing, data acquisition and processing efficiency are crucial for real-time performance and resource utilization.

[0003] Especially in insurance and financial scenarios, when data tables (data tasks) involving multiple insurance products require data configuration for each insurance product, traditional methods usually rely on sending data configuration requests to each service node in the order of the insurance product list. This serial request method is prone to factors such as network delays, resulting in slow data loading.

[0004] In addition, existing technical solutions often store the collected configuration data in a cache database, and need to wait for the entire configuration data to be downloaded before calculating and processing the entire configuration data. Due to the limited capacity of the cache database, it may face the risk of overload, thereby affecting the stability of data configuration processing.

[0005] Therefore, how to improve the processing speed and stability of data configuration is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of the above, it is necessary to provide a task data configuration method, the purpose of which is to improve the processing speed and stability of data configuration by converting data configuration requests into configuration tasks that can be processed in parallel and intelligently allocating configuration tasks according to the current load of each service node.

[0007] The task data configuration method provided by the present invention comprises:

[0008] Receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed according to the identified N key fields, and generate N configuration tasks for the data task to be processed according to the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0009] Obtaining the operating indicator data of each service node in the preset server cluster, calculating the load value of each service node according to the operating indicator data, screening multiple service nodes whose load values ​​are less than a preset threshold and dividing them into N node groups, and associating each node group with each configuration task one by one;

[0010] A storage document for storing configuration data is constructed for each configuration task in a preset database, and a transmission channel is created between each storage document and a corresponding node group;

[0011] The configuration data is downloaded from the corresponding node group to the corresponding storage document by using the transmission channel corresponding to each configuration task, the configuration data of each storage document is calculated and processed, and the calculation result is written into the data task to be processed.

[0012] Optionally, the key field for identifying the data task to be processed includes:

[0013] The key fields for identifying the data task to be processed include:

[0014] Dividing the data task to be processed into a plurality of data segments according to the chapter title names of the data task to be processed, wherein each data segment includes at least one chapter title name and a text content under the at least one chapter title name;

[0015] The text containing the preset key field is read out from each data segment, and the preset key field is identified from the read text.

[0016] Optionally, obtaining the operation indicator data of each service node in the preset server cluster includes:

[0017] Obtaining a node monitoring log from a preset monitoring tool in the preset server cluster, wherein the node monitoring log records the operation indicator data of each service node;

[0018] According to the identifier of each service node, the operation indicator data of each service node is extracted from the node monitoring log.

[0019] Optionally, the operation indicator data includes CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks and running speed of executed tasks, and the load value of each service node is calculated according to the operation indicator data, including:

[0020] Randomly select the first and second operating indicator data from the above operating indicator data and substitute them into the preset load value formula for calculation to obtain the load value of each service node;

[0021] Among them, the load value formula is L t =W1×A t +W2×B i , where W1 and W2 are given weight factors, A i is the first type of operating indicator data of the i-th service node, B iIt is the second operating indicator data of the i-th service node.

[0022] Optionally, the calculating the load value of each service node according to the operation indicator data includes:

[0023] Preprocessing the operation index data, wherein the preprocessing includes removing invalid or erroneous data;

[0024] The preprocessed operation index data is converted into eigenvalues, and the eigenvalues ​​corresponding to all the operation indicators are collected to obtain the corresponding eigenvalue array. The eigenvalue array is calculated using the trained load prediction model, and a preset weight is assigned to each eigenvalue in the eigenvalue array. The weighted summation of the eigenvalue array after the weight assignment is performed to obtain the load value of each service node.

[0025] Optionally, screening a plurality of service nodes whose load values ​​are less than a preset threshold and dividing them into N node groups includes:

[0026] Read the load value of each service node, and select multiple service nodes whose load values ​​are less than a preset threshold;

[0027] According to the N configuration tasks, multiple service nodes are evenly divided into corresponding N node groups.

[0028] Optionally, the step of creating a transmission channel between each storage document and a corresponding node group includes:

[0029] Based on the GET method of the preset transmission protocol, a separate GET request is configured for each stored document;

[0030] The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

[0031] In order to solve the above problem, the present invention also provides a task data configuration device, the device comprising:

[0032] A receiving module, used to receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed according to the identified N key fields, and generate N configuration tasks for the data task to be processed according to the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0033] A calculation module, used to obtain the operation index data of each service node in the preset server cluster, calculate the load value of each service node according to the operation index data, screen multiple service nodes with load values ​​less than a preset threshold and divide them into N node groups, and associate each node group with each configuration task in a one-to-one correspondence;

[0034] A construction module, used to construct a storage document for storing configuration data for each configuration task in a preset database, and to create a transmission channel between each storage document and a corresponding node group;

[0035] The writing module is used to download the configuration data from the corresponding node group to the corresponding storage document using the transmission channel corresponding to each configuration task, perform calculation processing on the configuration data of each storage document, and write the calculation result into the data task to be processed.

[0036] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0037] at least one processor; and,

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores a task data configuration program executable by the at least one processor, and the task data configuration program is executed by the at least one processor so that the at least one processor can perform the above-mentioned task data configuration method.

[0040] In order to solve the above problem, the present invention further provides a computer-readable storage medium, on which a task data configuration program is stored. The task data configuration program can be executed by one or more processors to implement the above task data configuration method.

[0041] Compared with the prior art, the present invention divides the data task to be processed into multiple configuration tasks based on identifying the key fields of the data task to be processed. By decomposing the tasks for parallel processing, the network delay problem caused by the traditional serial request method is avoided, and the speed of data loading and processing is greatly improved.

[0042] By real-time monitoring of the operating indicator data of each service node, multiple service nodes with load values ​​less than the preset threshold are screened and divided into N node groups, each node group is associated with each configuration task one by one, and the load balancing algorithm is used to avoid overloading of some nodes and ensure the stable operation of the download task.

[0043] Create an independent storage document for each configuration task, create a transmission channel between each storage document and the corresponding node group, use the transmission channel corresponding to each configuration task to download the configuration data from the corresponding node group to the corresponding storage document, perform calculations on the configuration data of each storage document, and write the calculation results into the data task to be processed. By downloading and processing at the same time, the reliance on the cache database is reduced, and the risk of overload caused by the limited capacity of the cache database is avoided. By decomposing and assigning tasks to different node groups for processing, the processing speed and stability of data configuration are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of a task data configuration method provided by an embodiment of the present invention;

[0045] Figure 2 A module diagram of a task data configuration device provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the structure of an electronic device for implementing a task data configuration method provided by an embodiment of the present invention;

[0047] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0050] Reference Figure 1 FIG. 1 is a flow chart of a task data configuration method provided by an embodiment of the present invention. The method is executed by an electronic device.

[0051] In this embodiment, the task data configuration method includes:

[0052] S1. Receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed based on the identified N key fields, and generate N configuration tasks for the data task to be processed based on the N types of data, wherein the key fields include a configuration data name or a configuration data ID.

[0053] In this embodiment, a request from a user or an application to configure data for a data task to be processed is received. After receiving the request, the data task to be processed needs to be analyzed and key fields of the data task to be processed need to be identified.

[0054] A pending data task refers to one or a group of tasks or data tables that need to be configured or updated. For example, in an insurance and financial scenario, a salesperson uploads a pending data task, which is a pending insurance policy. The pending insurance policy only contains different types of insurance products, but the various association tables that store the detailed information of these insurance products (such as premium calculation rules, claim conditions, product terms, etc.) are distributed and stored in various service nodes in the preset server cluster. Therefore, it is necessary to obtain the configuration data of the pending insurance policy from the various association tables of each service node, and use the pending insurance policy to process various types of insurance products in the pending insurance policy, such as supplementing detailed information, verifying information, or calculating the price of insurance products.

[0055] The key field of the data task to be processed is used to distinguish different data items or data types in the data task to be processed. For example, the key field may be the ID or name of an insurance product.

[0056] Determine how many different data items or data types the data task to be processed contains based on the identified key fields. For example, if the identified key fields include the names of auto insurance, personal accident insurance, and property insurance, then the data task to be processed needs to be configured to include data items or data types such as auto insurance, personal accident insurance, and property insurance.

[0057] According to the identified key fields, the N types of data that need to be configured for the data task to be processed are determined, and an independent configuration task is generated for each type of data. This means that if there are N different key fields, N configuration tasks will be generated.

[0058] Compared with the traditional method that usually relies on sending data configuration requests to each service node in sequence according to the insurance product list, this serial request method is prone to factors such as network delay, resulting in slow data loading speed. The present invention decomposes a large data configuration task into multiple small tasks according to data types, which can be processed in parallel on multiple service nodes, thereby improving the speed of data processing.

[0059] In one embodiment, the identifying key fields of the data task to be processed includes:

[0060] Dividing the data task to be processed into a plurality of data segments according to the chapter title names of the data task to be processed, wherein each data segment includes at least one chapter title name and a text content under the at least one chapter title name;

[0061] The text containing the preset key field is read out from each data segment, and the preset key field is identified from the read text.

[0062] According to the chapter title name in the data task to be processed, the content of the data task to be processed is divided into multiple data segments, and the text containing preset key fields (such as configuration data name or configuration data ID) is filtered out from each data segment, and the preset key fields are identified from the read text.

[0063] Assume that the preset key fields include "insurance product ID" and "product name", and you need to filter out the text containing these data segments from the parsed paragraphs. For example, the key fields in segment 1 are: insurance product ID: 001, product name: auto insurance. The key fields in segment 2 are: insurance product ID: 002, product name: personal accident insurance. Both segment 1 and segment 2 contain these two fields, so they are both retained.

[0064] By parsing the chapter title names in the data task and splitting the content into multiple texts, you can more accurately identify which parts need to be configured and more precisely locate the specific fields that need to be updated, avoiding unnecessary data processing work.

[0065] S2. Obtain the operating indicator data of each service node in the preset server cluster, calculate the load value of each service node according to the operating indicator data, select multiple service nodes whose load values ​​are less than a preset threshold and divide them into N node groups, and associate each node group with each configuration task one by one.

[0066] In this embodiment, the node monitoring log is obtained from a preset monitoring tool in the preset server cluster. The preset monitoring tool may be a Prometheus monitoring and alarm system. The node monitoring log records the operating index data of each service node at the current moment. The operating index data of each service node is obtained in real time through a monitoring tool (such as Prometheus). The current status of each service node can be accurately understood, and configuration tasks can be intelligently allocated according to the actual load of the service node, thereby avoiding the situation where some nodes are overloaded while other nodes are idle.

[0067] According to the identifier of each service node, the operating indicator data of each service node is extracted from the node monitoring log, and multiple service nodes with load values ​​less than a preset threshold (for example, the preset threshold is 0.75) are screened and divided into N node groups. This can more effectively utilize the resources in the cluster, and the service nodes in each node group can efficiently process the tasks assigned to them, which can achieve better load balancing, not only avoiding overload of a single node, but also making full use of the available resources in the cluster, thereby improving the resource utilization of the overall system.

[0068] By matching configuration tasks with node groups one by one, we can ensure that each configuration task has a dedicated node group to handle it. Each node group only needs to focus on the part of the configuration task it is responsible for, avoiding interference between tasks and improving the accuracy and efficiency of task processing.

[0069] In one embodiment, the screening of multiple service nodes whose load values ​​are less than a preset threshold and dividing them into N node groups includes:

[0070] Read the load value of each service node, and select multiple service nodes whose load values ​​are less than a preset threshold;

[0071] According to the N configuration tasks, multiple service nodes are evenly divided into corresponding N node groups.

[0072] All service nodes with load values ​​less than the preset threshold are evenly divided into N node groups. By evenly distributing them, there is no need for complex calculations or additional resource allocation logic. To a certain extent, it can ensure that the load levels of all service nodes are roughly the same, thus avoiding the problem of single-point overload.

[0073] In other implementations, multiple service nodes with load values ​​less than a preset threshold may be screened and divided into N node groups in the following manner:

[0074] Method 1: Assign a priority to multiple service nodes whose load values ​​are less than a preset threshold. For example, service nodes with a load value of 0.10-0.29 are assigned a high priority, service nodes with a load value of 0.30-0.49 are assigned a medium priority, and service nodes with a load value of 0.50-0.69 are assigned a low priority. First, the service nodes with high priority are divided into N node groups. If the number of service nodes with high priority is not enough to be divided, they are then retrieved from the service nodes in the priority group. Nodes with high priority are usually nodes with better performance and can complete tasks faster, thereby improving the overall system performance. The priority can be dynamically adjusted according to changes in load, making the system more flexible to adapt to different workloads.

[0075] Method 2: Filter M service nodes whose load values ​​are less than the preset threshold, divide the M service nodes into N node groups, where M is greater than N, monitor the load of the M service nodes in real time, and dynamically adjust the task allocation strategy according to the actual situation. For example, if the load of a node suddenly increases, its task volume can be reduced immediately, and the task can be transferred to other nodes with lower loads. By continuously monitoring the node status, task allocation can be adjusted in real time to keep the system load at the optimal state at all times, quickly discover and handle overloaded or faulty nodes, and transfer tasks in a timely manner, thereby improving the system's fault tolerance and availability.

[0076] The present invention does not limit which method to choose. At least the method to choose depends on the specific requirements of the actual application scenario, such as the system's requirements on performance, load volatility, and the degree of emphasis on stability.

[0077] In one embodiment, the obtaining of the operation indicator data of each service node in the preset server cluster includes:

[0078] Obtaining a node monitoring log from a preset monitoring tool in the preset server cluster, wherein the node monitoring log records the operation indicator data of each service node;

[0079] According to the identifier of each service node, the operation indicator data of each service node is extracted from the node monitoring log.

[0080] Use preset monitoring tools (such as the Prometheus monitoring and alarm system) to monitor the entire server cluster, and use preset monitoring tools to capture the operating indicator data of each service node. The operating indicator data includes but is not limited to CPU usage, memory usage, disk I / O read and write speed, network bandwidth usage, number of unexecuted tasks, and running speed of executed tasks.

[0081] The preset monitoring tool periodically collects the above-mentioned operating indicator data from each service node and stores these operating indicator data in its own time series database. These operating indicator data constitute a node monitoring log, thereby realizing the extraction of the operating indicator data of each service node from the node monitoring log according to the identifier of each service node.

[0082] By extracting operating indicator data from monitoring logs, the operating status of each service node can be monitored in real time, and the latest status information of the service node can be obtained at the first time, so as to make timely decisions, such as whether to assign configuration tasks to a certain node, and to promptly discover potential problems, such as abnormal node resource usage, increased network latency, etc., and take measures to prevent these problems from affecting the data processing process. This helps to maintain the stable operation of the system.

[0083] In one embodiment, the operation index data includes CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks and running speed of executed tasks, and the load value of each service node is calculated according to the operation index data, including:

[0084] Randomly select the first and second operating indicator data from the above operating indicator data and substitute them into the preset load value formula for calculation to obtain the load value of each service node;

[0085] Among them, the load value formula is L i =W1×A i +W2×B i , where W1 and W2 are given weight factors, A i is the first type of operating indicator data of the i-th service node, B o It is the second operating indicator data of the i-th service node.

[0086] The operation index data of each service node is obtained from the monitoring tool. By arbitrarily selecting the first and second operation index data from the above operation index data and substituting them into the preset load value formula, the actual load of the service node can be more accurately reflected, and tasks can be intelligently allocated according to the current load of the service node. For example, a service node with a lower load value can be given priority to allocate new configuration tasks.

[0087] For example:

[0088] There are two service nodes Node1 and Node2. CPU usage and memory usage are selected as calculation indicators, and it is assumed that W1=0.5 and W2=0.5.

[0089] Node1's CPU usage is 50% and memory usage is 40%.

[0090] Node2's CPU usage is 70% and memory usage is 50%.

[0091] According to the load value formula L i =W1×A i +W2×B i , we can calculate:

[0092] Node1's load value L Node1 =0.5×0.5+0.5×0.4=0.45 or 45%.

[0093] Node2's load value L Node2 =0.5×0.7+0.5×0.5=0.6 or 60%.

[0094] In this case, the load of Node1 is lower, so the new configuration task can be assigned to Node1 first. In this way, the load of the service node can be evaluated and managed more accurately, thereby achieving efficient resource allocation and task processing.

[0095] In one embodiment, the calculating the load value of each service node according to the operation indicator data includes:

[0096] Preprocessing the operation index data, wherein the preprocessing includes removing invalid or erroneous data;

[0097] The preprocessed operation index data is converted into eigenvalues, and the eigenvalues ​​corresponding to all the operation indicators are collected to obtain the corresponding eigenvalue array. The eigenvalue array is calculated using the trained load prediction model, and a preset weight is assigned to each eigenvalue in the eigenvalue array. The weighted summation of the eigenvalue array after the weight assignment is performed to obtain the load value of each service node.

[0098] The operation indicator data includes CPU usage, memory usage, disk I / O read and write speed, network bandwidth usage, number of unexecuted tasks, and the running speed of executed tasks. The operation indicator data is preprocessed, such as checking whether there are abnormal values, missing values, or error values ​​in the operation indicator data, and processing them. If the running speed of the executed tasks reported by a service node is negative or exceeds a reasonable range, it should be regarded as an abnormal value and corrected or deleted. If there is missing data, interpolation, forward filling, backward filling, or other statistical methods can be used to fill the missing values.

[0099] The preprocessed operating index data are converted into corresponding eigenvalues, as follows:

[0100] If the operating indicator data is given in percentage form (such as CPU usage, memory usage), you can use these values ​​directly. For example, if the CPU usage is 75%, you can directly use 0.75 as a feature value.

[0101] If the disk I / O read / write speed is given in bytes per second, it can be converted to a standardized form, such as converting it to the number of read and write operations per second, or comparing it with a benchmark speed to get a relative value.

[0102] If the network bandwidth usage is given in Mbps or Gbps, you can convert it into a percentage. For example, if the maximum bandwidth is 1Gbps and the current usage is 600Mbps, the converted percentage is 60%.

[0103] If the number of tasks executed is an absolute number, it can be compared to the maximum processing capacity of the node to get a relative value. For example, if the maximum processing capacity of the node is 100 tasks and 30 tasks are currently being executed, the converted percentage is 30%.

[0104] For each service node, the eigenvalues ​​corresponding to all the above operating indicators are collected to obtain an eigenvalue array. For example, for Node1, if its CPU usage is 0.75, memory usage is 0.4, disk I / O read / write speed is 0.6, network bandwidth usage is 0.5, and the number of executed tasks is 0.3, the constructed eigenvalue array can be expressed as [0.75, 0.4, 0.6, 0.5, 0.3].

[0105] Input the feature value array of each service node into the trained load prediction model, and assign a weight to each feature value in the feature value array. For example, if the load prediction model uses a simple linear regression model, the load value L can be calculated using the following formula:

[0106] L=w1·CPU+w2·Memory+w3·DiskIO+w4·Bandwidth+w5·Tasks

[0107] Among them, w1-w5 are feature weights, CPU, Memory, DiskIO, Bandwidth, and Tasks are feature values ​​in the feature value array. If it is determined that CPU usage has a greater impact on the load, it will be assigned a higher weight, while other features may be assigned lower weights.

[0108] After assigning a preset weight to each eigenvalue in the eigenvalue array, the eigenvalue array with assigned weights is linearly combined (eg, weighted summation) or nonlinearly combined (eg, through an activation function) to obtain the load value of each service node.

[0109] For example, for Node1, if the load value calculated by the load prediction model is 0.58, then it can be considered that Node1 is currently in a medium load state. If the load value is low, you can consider assigning new configuration tasks to this node; if the load value is high, you should avoid assigning new tasks to this node, and instead look for nodes with lower loads to handle new configuration tasks. In this way, load balancing can be achieved, ensuring that resources in the cluster are effectively utilized, and improving the stability and reliability of the system.

[0110] The trained load prediction model automatically calculates the load value of each service node, reducing the need for manual intervention. It can process the latest operating indicator data in real time and provide load prediction results instantly, helping the system to respond quickly to load changes and enable the system to manage resources more intelligently.

[0111] The training process of the load prediction model includes:

[0112] Use monitoring tools to collect the operating indicator data of the service node, including but not limited to CPU usage, memory usage, disk I / O read and write speed, network bandwidth usage, number of unexecuted tasks, and running speed of executed tasks. Preprocess the operating indicator data and divide the preprocessed operating indicator data into a training set and a test set (for example, the ratio is 70% training and 30% testing), or further divide it into a training set, a validation set, and a test set.

[0113] Use a regression model as the initial load prediction model, such as linear regression, support vector machine (SVM), decision tree, random forest, gradient boosting tree (such as XGBoost or LightGBM), or neural network. Use the training set data to train the initial load prediction model. During the training process, adjust the hyperparameters according to the performance of the initial load prediction model to optimize the model performance of the initial load prediction model. To prevent overfitting, cross-validation techniques can be used to evaluate the generalization ability of the initial load prediction model.

[0114] Adjust the parameters of the initial load prediction model according to the verification results, repeat the training and verification process until you are satisfied, and get a trained load prediction model. Integrate the trained load prediction model into the production environment so that it can make predictions on real-time data streams.

[0115] In step S2, the running status of the service node is monitored in real time, the load value of the node is calculated according to the running index data, the appropriate nodes are selected according to the load value, and the nodes are divided into multiple groups, each group processing a configuration task. This avoids some nodes being overloaded while other node resources are idle, achieves more reasonable resource utilization, improves the system's load balancing ability, and enhances the system's stability and reliability.

[0116] S3. Build a storage document for storing configuration data for each configuration task in the preset database, and create a transmission channel between each storage document and the corresponding node group.

[0117] In this embodiment, a shell script is created in a preset database. For example, the preset database may be a Hudi storage database.

[0118] In order to improve the reading speed and data storage space, the present invention optimizes the configuration data storage and reading performance through the indexing and compression technology of the Hudi storage database, which is very beneficial for real-time data processing and can speed up the data reading speed.

[0119] The name of each configuration task is used as a parameter of the shell script. The shell script based on the configuration parameters calls the operation command of the preset database, creates a corresponding storage document for each configuration task in the preset database, and configures a separate GET request for each storage document based on the GET method of the built-in transmission protocol of the preset database. The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

[0120] In one embodiment, constructing a storage document for storing configuration data for each configuration task includes:

[0121] Create a shell script in the preset database, and use the name of each configuration task as a parameter of the shell script;

[0122] The shell script based on the configuration parameters calls the operation command of the preset database, and creates a corresponding storage document for each configuration task in the preset database.

[0123] Create a shell script in the preset database (such as the Hudi storage database). The main function of the shell script is to receive the name of each configuration task and execute the operation of creating a storage document in the database according to the name of the configuration task. Shell script is a program written in the Shell language in the Unix / Linux operating system. Shell is an interface between the user and the operating system, allowing the user to interact with the system through the command line interface (CLI).

[0124] The Shell script calls the preset database operation commands, which are used to create storage documents in the database. For each configuration task name passed to the script, the Shell script will execute a series of commands to create a storage document for the task in the preset database.

[0125] Compared with the existing technical solutions, it is necessary to wait for the entire configuration data to be downloaded to the cache database before calculating and processing the entire configuration data. If the amount of configuration data is large, it takes a long time to download the entire data set to the cache database. Even if only a small amount of data is missing and has not been downloaded, the entire processing flow needs to wait, resulting in a longer start-up time for the calculation and processing. While waiting for the data to be completely downloaded, the computing resources may be idle and unable to start processing the downloaded data, resulting in a waste of resources.

[0126] The present invention constructs an independent storage document for each configuration task. It only needs to download the configuration data of the current storage document before calculating and processing the configuration data of the current storage document. There is no need to wait for the entire configuration data to be downloaded. This allows multiple tasks to be processed in parallel without interfering with each other, which can significantly improve the speed and efficiency of data processing.

[0127] For example:

[0128] In the existing technical solution, the configuration data of all three products must be downloaded to the cache database before processing can begin. If the amount of data for auto insurance is the largest and the amount of data for health insurance is the smallest, then even if the data for health insurance has been downloaded, the system must wait until all data has been downloaded before processing can begin.

[0129] The method of the present invention: for each configuration task (auto insurance, personal accident insurance, health insurance), a separate storage document is created. When the configuration data of health insurance is downloaded, the system immediately starts to process the health insurance data. Similarly, when the personal accident insurance data is downloaded, the personal accident insurance data is also processed. Finally, when the auto insurance data is downloaded, the auto insurance data is processed.

[0130] In this way, even if some data has not been downloaded yet, the downloaded data can be processed immediately, which significantly improves the processing speed and efficiency. At the same time, since each task is processed independently and will not interfere with each other, the system can better utilize computing resources, reduce waiting time, and improve overall performance.

[0131] In one embodiment, the step of creating a transmission channel between each storage document and the corresponding node group includes:

[0132] Based on the GET method of the preset transmission protocol, a separate GET request is configured for each stored document;

[0133] The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

[0134] Construct a unique URI (Uniform Resource Identifier) ​​for each stored document. The URI usually contains the path, table name, partition information, etc. of the stored document to facilitate locating the specific document. Based on the GET method of the preset database built-in transmission protocol, a separate GET request is configured for each stored document.

[0135] Construct the necessary HTTP header information for the GET request, such as the header information including authentication information, content type, etc., add the URL to the GET request, and send the GET request to the corresponding node group in the HTTP client library. The node group receives the GET request sent by the Hudi storage database, and actually establishes a temporary transmission channel, which returns the configuration data according to the response format of the GET request (usually JSON or Parquet, etc.).

[0136] By distributing requests to multiple service nodes, even if a service node has a problem, it will not affect the overall data transmission process, improving the stability and reliability of the system. If the load of a service node suddenly increases, it can be dynamically adjusted by reallocating GET requests to other service nodes with lower loads to ensure smooth data transmission.

[0137] In step S3, a separate storage document is created in the database for each configuration task, a transmission channel is established, and the configuration data is downloaded from the specified node group to the storage document through a GET request. This solves the problem that the traditional method requires waiting for the entire data set to be downloaded before processing can begin, and achieves processing while downloading, improving processing efficiency, reducing the pressure on the cache database, and avoiding the risk of overload caused by the limited capacity of the cache database.

[0138] S4. Using the transmission channel corresponding to each configuration task, download the configuration data from the corresponding node group to the corresponding storage document, perform calculation processing on the configuration data of each storage document, and write the calculation result into the data task to be processed.

[0139] In this embodiment, after the transmission channel corresponding to each configuration task is determined, each configuration task will have its own independent transmission channel, thereby ensuring the independence and efficiency of data transmission.

[0140] The configuration data is downloaded from the corresponding node group to the corresponding storage document through a GET request, thereby achieving parallel reception of the configuration data.

[0141] Use the preset parsing tool (the parsing tool can be a corresponding library or tool, such as Python's json module) to parse the configuration data, and calculate and process the parsed configuration data according to the preset requirements of each configuration task to obtain the calculation results. The calculation processing includes data verification, data conversion, and aggregation operations. Specifically, data verification is to ensure that the format of the configuration data is correct, such as checking whether the configuration data contains necessary fields, such as the insurance product ID and name. Verify whether the content of the configuration data meets expectations, such as checking whether the premium is a positive number and whether the claim conditions are reasonable.

[0142] Data conversion is to convert the configuration data into a format suitable for further processing. For example, convert a string type number to an integer or floating point number. If there are values ​​in different units in the configuration data, they need to be converted to a unified unit for comparison or calculation.

[0143] Aggregation operations summarize configuration data according to the needs of configuration tasks, such as calculating the average premium of a certain type of insurance product. Or they combine configuration data from different sources to form complete configuration information. For example, premium information and claim conditions can be combined for comprehensive analysis.

[0144] The calculation results are written into the data task to be processed. For example, the data task to be processed is the insurance policy to be associated. If detailed information of various insurance products in the insurance policy to be associated needs to be supplemented, the calculation results obtained are the detailed information of various insurance products, so that the detailed information of various insurance products is supplemented into the insurance policy to be associated.

[0145] If it is necessary to check whether the information of various insurance products in the insurance policy to be associated is correct, then the calculation result obtained is the test result of the information of various insurance products, and the test results of various insurance products are added to the insurance policy to be associated.

[0146] If it is necessary to calculate the product prices of various insurance products in the insurance policy to be associated, then the calculation results are the prices of various insurance products, and the prices of various insurance products are added to the insurance policy to be associated.

[0147] In step S4, the configuration data is downloaded to the storage document using the transmission channel, the configuration data in each storage document is calculated and processed, and the processing results are written back to the data task to be processed. By processing multiple configuration tasks in parallel, the speed of data processing is improved, and the accuracy and consistency of data are ensured through data verification, conversion and aggregation operations, thereby improving the processing quality.

[0148] In summary, steps S1 to S4 work together to improve the speed and stability of data configuration processing, solving the problems of network delay and slow data loading caused by the traditional serial request method; solving the problem of uneven resource utilization through dynamic load balancing mechanism; and further improving the system's processing power and stability through parallel processing and optimized data storage and transmission mechanism. These improvements are of great significance for processing large and complex data configuration tasks, especially in insurance and financial scenarios that require high real-time performance and resource utilization.

[0149] like Figure 2 FIG. 1 is a schematic diagram of a module of a task data configuration device provided by an embodiment of the present invention.

[0150] The task data configuration device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the task data configuration device 100 can include a receiving module 110, a calculating module 120, a building module 130 and a writing module 140. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete a fixed function, which is stored in the memory of the electronic device.

[0151] In this embodiment, the functions of each module / unit are as follows:

[0152] The receiving module 110 is used to receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed according to the identified N key fields, and generate N configuration tasks for the data task to be processed according to the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0153] The calculation module 120 is used to obtain the operation index data of each service node in the preset server cluster, calculate the load value of each service node according to the operation index data, select multiple service nodes with load values ​​less than a preset threshold and divide them into N node groups, and associate each node group with each configuration task in a one-to-one correspondence;

[0154] A construction module 130 is used to construct a storage document for storing configuration data for each configuration task in a preset database, and to create a transmission channel between each storage document and a corresponding node group;

[0155] The writing module 140 is used to download the configuration data from the corresponding node group to the corresponding storage document using the transmission channel corresponding to each configuration task, perform calculation processing on the configuration data of each storage document, and write the calculation result into the data task to be processed.

[0156] like Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device for implementing a method for configuring task data according to an embodiment of the present invention.

[0157] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected and communicated via a system bus. The memory 11 stores a task data configuration program 10 that can be executed by the processor 12. Figure 3 Only the electronic device 1 having the components 11-13 and the task data configuration program 10 is shown, and those skilled in the art can understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0158] Among them, the memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. In this embodiment, the readable storage medium of the memory 11 is generally used to store the operating system and various application software installed on the electronic device 1, such as storing the code of the task data configuration program 10 in one embodiment of the present invention, etc. In addition, the memory 11 may also be used to temporarily store various types of data that have been output or are to be output.

[0159] The processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the task data configuration program 10.

[0160] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a terminal (not shown in the figure).

[0161] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0162] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0163] The task data configuration program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 12, it can achieve:

[0164] Receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed according to the identified N key fields, and generate N configuration tasks for the data task to be processed according to the N types of data, wherein the key fields include a configuration data name or a configuration data ID;

[0165] Obtaining the operating indicator data of each service node in the preset server cluster, calculating the load value of each service node according to the operating indicator data, screening multiple service nodes whose load values ​​are less than a preset threshold and dividing them into N node groups, and associating each node group with each configuration task one by one;

[0166] A storage document for storing configuration data is constructed for each configuration task in a preset database, and a transmission channel is created between each storage document and a corresponding node group;

[0167] The configuration data is downloaded from the corresponding node group to the corresponding storage document by using the transmission channel corresponding to each configuration task, the configuration data of each storage document is calculated and processed, and the calculation result is written into the data task to be processed.

[0168] Specifically, the specific implementation method of the processor 12 for the task data configuration program 10 can refer to Figure 1The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0169] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0170] The computer-readable storage medium stores a task data configuration program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the above-mentioned embodiments of the task data configuration method, and will not be repeated here.

[0171] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0172] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0173] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0174] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0175] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0176] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A task data configuration method, characterized in that: The method comprises: Receive a request for data configuration of a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed according to the identified N key fields, and generate N configuration tasks for the data task to be processed according to the N types of data, wherein the key fields include a configuration data name or a configuration data ID; Obtaining the operating indicator data of each service node in the preset server cluster, calculating the load value of each service node according to the operating indicator data, screening multiple service nodes whose load values ​​are less than a preset threshold and dividing them into N node groups, and associating each node group with each configuration task one by one; A storage document for storing configuration data is constructed for each configuration task in a preset database, and a transmission channel is created between each storage document and a corresponding node group; The configuration data is downloaded from the corresponding node group to the corresponding storage document by using the transmission channel corresponding to each configuration task, the configuration data of each storage document is calculated and processed, and the calculation result is written into the data task to be processed.

2. The task data configuration method according to claim 1, characterized in that: The key fields for identifying the data task to be processed include: Dividing the data task to be processed into a plurality of data segments according to the chapter title names of the data task to be processed, wherein each data segment includes at least one chapter title name and a text content under the at least one chapter title name; The text containing the preset key field is read out from each data segment, and the preset key field is identified from the read text.

3. The task data configuration method according to claim 1, characterized in that: The step of obtaining the operation indicator data of each service node in the preset server cluster includes: Obtaining a node monitoring log from a preset monitoring tool in the preset server cluster, wherein the node monitoring log records the operation indicator data of each service node; According to the identifier of each service node, the operation indicator data of each service node is extracted from the node monitoring log.

4. The task data configuration method according to claim 1, characterized in that: The operation index data includes CPU usage, memory usage, disk I / O read / write speed, network bandwidth usage, number of unexecuted tasks and running speed of executed tasks. The load value of each service node is calculated based on the operation index data, including: Randomly select the first and second operating indicator data from the above operating indicator data and substitute them into the preset load value formula for calculation to obtain the load value of each service node; Among them, the load value formula is L i =W1×A i +W2×B i , where W1 and W2 are given weight factors, A i is the first type of operating indicator data of the i-th service node, B i It is the second operating indicator data of the i-th service node.

5. The task data configuration method according to claim 1, characterized in that: The calculating the load value of each service node according to the operation indicator data includes: Preprocessing the operation index data, wherein the preprocessing includes removing invalid or erroneous data; The preprocessed operation index data is converted into eigenvalues, and the eigenvalues ​​corresponding to all the operation indicators are collected to obtain the corresponding eigenvalue array. The eigenvalue array is calculated using the trained load prediction model, and a preset weight is assigned to each eigenvalue in the eigenvalue array. The weighted summation of the eigenvalue array after the weight assignment is performed to obtain the load value of each service node.

6. The task data configuration method according to claim 1, characterized in that: The screening of multiple service nodes whose load values ​​are less than a preset threshold and dividing them into N node groups includes: Read the load value of each service node, and select multiple service nodes whose load values ​​are less than a preset threshold; According to the N configuration tasks, multiple service nodes are evenly divided into corresponding N node groups.

7. The task data configuration method according to claim 1, characterized in that: The step of creating a transmission channel between each storage document and the corresponding node group includes: Based on the GET method of the preset transmission protocol, a separate GET request is configured for each stored document; The GET request is sent to the corresponding node group for communication connection to obtain a transmission channel.

8. A task data configuration device, characterized in that: The device comprises: A receiving module, used to receive a request for configuring data for a data task to be processed, identify key fields of the data task to be processed, determine N types of data that need to be configured for the data task to be processed according to the identified N key fields, and generate N configuration tasks for the data task to be processed according to the N types of data, wherein the key fields include a configuration data name or a configuration data ID; A calculation module, used to obtain the operation index data of each service node in the preset server cluster, calculate the load value of each service node according to the operation index data, screen multiple service nodes with load values ​​less than a preset threshold and divide them into N node groups, and associate each node group with each configuration task in a one-to-one correspondence; A construction module, used to construct a storage document for storing configuration data for each configuration task in a preset database, and to create a transmission channel between each storage document and a corresponding node group; The writing module is used to download the configuration data from the corresponding node group to the corresponding storage document using the transmission channel corresponding to each configuration task, perform calculation processing on the configuration data of each storage document, and write the calculation result into the data task to be processed.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a task data configuration program executable by the at least one processor, and the task data configuration program is executed by the at least one processor so that the at least one processor can perform the task data configuration method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a task data configuration program, and the task data configuration program can be executed by one or more processors to implement the task data configuration method according to any one of claims 1 to 7.

Images