Optimization method and device for big data processing model and computer equipment

Abstract

The invention relates to the technical field of model optimization, is suitable for the field of human life guarantee services, and provides an optimization method and device for a big data processing model and computer equipment, and the method comprises the steps: obtaining model operation associated information involved by the big data processing model for executing a target task for a designated user; determining whether to enable the big data processing model to enter an optimization mode or not based on the model operation association information; if the big data processing model enters the optimization mode, determining a target optimization strategy matched with the target task in a plurality of preset optimization strategies based on the model operation associated information; and executing the target optimization strategy on the big data processing model. According to the technical scheme, the big data processing model can automatically optimize itself in real time when executing different tasks of different users so as to adapt to the target task being executed, the adaptability of the big data processing model to multiple users and multiple tasks is improved, and therefore the life guarantee service level is improved.

Description

[Technical field]

[0001] The present application relates to the field of model optimization technology, is applicable to the field of life insurance services, and in particular, to an optimization method and device and computer equipment for big data processing models. [Background technology]

[0002] In the process of issuing and providing life insurance services, it is often necessary to use big data processing models to serve the normal operation of life insurance services. For example, in each settlement cycle, the big data processing model is used to profile the customer groups of each life insurance service, or the big data processing model is used to evaluate the protection capacity, project risks and other information of each life insurance service. On this basis, due to the variability of various aspects of life insurance service information, it is often necessary to optimize the big data processing model to enhance the data analysis capabilities of the big data processing model.

[0003] In this regard, relevant technologies provide a variety of optimization methods for big data processing models, such as CBO (Cost-Based Optimization) and HBO (History-Based Optimizer).

[0004] However, HBO is good at optimizing ordinary map tasks in big data processing models, but it is difficult to obtain high-quality optimization results for data aggregation tasks. CBO calculates the cost of each execution method based on the collected statistical information of tables, partitions, indexes, etc., and then selects the best execution method to optimize the big data processing model. It has a high degree of compatibility with ordinary map tasks and data aggregation tasks. However, CBO's collection of table and other information also has a cost, especially in a big data environment, where the table size is huge. CBO will consume a lot of system resources to collect information, and the utilization rate of this information during the optimization process is very low, resulting in a waste of system resources. Furthermore, although the existing optimization methods have open source optimizers, they still require developers to manually improve these optimizers for the current tasks required for life insurance services in order to adapt the optimizers to the current tasks required. This development process is time-consuming, labor-intensive, and inefficient.

[0005] Therefore, how to efficiently and quickly optimize the data analysis capabilities of big data processing models for life insurance service-related work has become a technical problem that needs to be solved urgently. [Summary of the invention]

[0006] The embodiments of the present application provide an optimization method, device and computer equipment for a big data processing model, aiming to solve the technical problem that the optimization methods of big data processing models in related technologies have many defects and cannot meet the data analysis requirements of related work of life insurance services.

[0007] In a first aspect, an embodiment of the present application provides an optimization method for a big data processing model, comprising:

[0008] Obtaining model operation association information involved in the execution of a target task by a big data processing model for a designated user, wherein the designated user is a business-related party of a life insurance service, the target task includes a data analysis operation of any relevant work link of the life insurance service, and the model operation association information is used to reflect the processing capability of the big data processing model for the target task;

[0009] Determining whether to put the big data processing model into an optimization mode based on the model operation associated information;

[0010] If the big data processing model enters the optimization mode, based on the model operation associated information, a target optimization strategy matching the target task is determined from a plurality of preset optimization strategies;

[0011] The target optimization strategy is executed on the big data processing model.

[0012] In one embodiment of the present application, optionally, the model operation associated information includes: the current CPU utilization, the current CPU operating voltage, the number of threads of the target task, the average execution time of the threads of the target task, the memory occupancy of the target task, the rate of change of the memory occupancy within a first specified time period, the disk occupancy of the target task, the rate of change of the disk occupancy within a second specified time period, the network input and network output of the big data processing model per unit time, and the data processing volume of each map in the big data processing model per unit time.

[0013] In one embodiment of the present application, optionally, determining whether to make the big data processing model enter the optimization mode based on the model operation association information includes:

[0014] Determining an operation state of the big data processing model based on the model operation associated information, wherein the operation state is a steady state or a high concurrency state; and

[0015] Determining a processing capability characteristic value of the big data processing model based on the model operation associated information;

[0016] Determine whether the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, wherein if the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, determine to make the big data processing model enter the optimization mode.

[0017] In one embodiment of the present application, optionally, determining the processing capability characteristic value of the big data processing model based on the model operation association information includes:

[0018] Encode each item of the model operation associated information according to a preset encoding method, and normalize the encoding result to obtain a first eigenvalue corresponding to each item of the model operation associated information;

[0019] For each of the model runs associated information,

[0020] Obtaining preset associated information of the model operation associated information in all the model operation associated information;

[0021] Determining a second eigenvalue based on the first eigenvalue corresponding to the model operation associated information and the first eigenvalue corresponding to the preset associated information, wherein the second eigenvalue is used to reflect the characteristic performance level of the model operation associated information under the influence of the preset associated information;

[0022] Based on the first eigenvalue and the second eigenvalue, determining a third eigenvalue corresponding to the model operation associated information, wherein the third eigenvalue is used to reflect the overall characteristic performance level of the model operation associated information;

[0023] The processing capability characteristic value of the big data processing model is determined according to the third characteristic values ​​corresponding to each of the multiple pieces of model operation associated information.

[0024] In one embodiment of the present application, optionally, determining a third eigenvalue corresponding to the model operation association information based on the first eigenvalue and the second eigenvalue includes:

[0025] Obtaining a weight of the first eigenvalue and a weight of the second eigenvalue, wherein the weight of the first eigenvalue reflects the contribution of the model operation associated information in a specified number of historical model optimization operations, and the weight of the second eigenvalue reflects the contribution of the preset associated information in a specified number of historical model optimization operations;

[0026] A weighted sum is performed on the first eigenvalue and the second eigenvalue to obtain a third eigenvalue corresponding to the model operation associated information.

[0027] In one embodiment of the present application, optionally, determining a target optimization strategy matching the target task from a plurality of preset optimization strategies based on the model operation association information includes:

[0028] Obtaining task processing requirement information corresponding to the running status of the big data processing model;

[0029] Determine difference information between the task processing requirement information and the model operation associated information;

[0030] If the difference information is within a preset difference range, determining that the target optimization strategy is to obtain historical operation-related information of the big data processing model, and optimizing the big data processing model based on the historical operation-related information and a historical optimizer;

[0031] If the difference information is not within the preset difference range, the target optimization strategy is determined to be selecting an optimization execution mode with the lowest execution cost from available optimization execution modes.

[0032] In one embodiment of the present application, optionally, the method further includes:

[0033] In the process of executing the target optimization strategy on the big data processing model, real-time optimization related information is obtained at every specified time interval, wherein the real-time optimization related information includes: current CPU utilization, current CPU operating voltage, number of threads of the target task, average execution time of threads of the target task, and data processing volume per unit time of each map in the big data processing model;

[0034] For each item of the real-time optimization association information, determining a ratio of the real-time optimization association information to the historical optimization association information corresponding to the real-time optimization association information in the model operation association information before executing the target optimization strategy;

[0035] Determining the optimization capability confidence of the target optimization strategy based on the ratios corresponding to all the real-time optimization related information;

[0036] If the optimization capability confidence is lower than a preset optimization capability threshold, a supplementary optimization strategy is executed on the big data processing model, wherein the supplementary optimization strategy is data distribution optimization.

[0037] In one embodiment of the present application, optionally, determining the optimization capability confidence of the target optimization strategy based on the ratios corresponding to all the real-time optimization related information includes:

[0038] For each item of the real-time optimization associated information, a weighted characteristic value of the real-time optimization associated information is obtained, wherein the weight of the real-time optimization associated information reflects the contribution of the real-time optimization associated information in a specified number of historical model optimization operations;

[0039] Taking the ratio corresponding to the real-time optimization associated information as the weight of the weighted eigenvalue, weighted processing is performed on the weighted eigenvalues ​​of all the real-time optimization associated information to obtain fourth eigenvalues ​​corresponding to all the real-time optimization associated information;

[0040] The variance of the fourth eigenvalues ​​corresponding to all the real-time optimization associated information is determined as the optimization capability confidence of the target optimization strategy.

[0041] In a second aspect, an embodiment of the present application provides an optimization device for a big data processing model, including:

[0042] A model operation related information acquisition unit, used to acquire model operation related information involved in the execution of a target task by a big data processing model for a designated user, wherein the designated user is a business-related party of a life insurance service, the target task includes a data analysis operation of any relevant work link of the life insurance service, and the model operation related information is used to reflect the processing capability of the big data processing model for the target task;

[0043] An optimization timing determination unit, configured to determine whether to make the big data processing model enter an optimization mode based on the model operation associated information;

[0044] An optimization strategy selection unit, configured to determine a target optimization strategy matching the target task from a plurality of preset optimization strategies based on the model operation associated information if the big data processing model enters the optimization mode;

[0045] An optimization strategy execution unit is used to execute the target optimization strategy on the big data processing model.

[0046] In one embodiment of the present application, optionally, the model operation associated information includes: the current CPU utilization, the current CPU operating voltage, the number of threads of the target task, the average execution time of the threads of the target task, the memory occupancy of the target task, the rate of change of the memory occupancy within a first specified time period, the disk occupancy of the target task, the rate of change of the disk occupancy within a second specified time period, the network input and network output of the big data processing model per unit time, and the data processing volume of each map in the big data processing model per unit time.

[0047] In one embodiment of the present application, optionally, the optimization timing determination unit includes:

[0048] An operation state determining unit, configured to determine the operation state of the big data processing model based on the model operation associated information, wherein the operation state is a steady state or a high concurrency state;

[0049] A processing capability characteristic value determining unit, configured to determine a processing capability characteristic value of the big data processing model based on the model operation associated information;

[0050] A capability level judgment unit is used to determine whether the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, wherein if the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, it is determined that the big data processing model enters the optimization mode.

[0051] In one embodiment of the present application, optionally, the processing capability characteristic value determination unit is used to: encode each item of the model operation associated information according to a preset encoding method, and normalize the encoding result to obtain a first characteristic value corresponding to each item of the model operation associated information; for each item of the model operation associated information, obtain the preset associated information of the model operation associated information in all the model operation associated information; determine a second characteristic value based on the first characteristic value corresponding to the model operation associated information and the first characteristic value corresponding to the preset associated information, wherein the second characteristic value is used to reflect the characteristic performance level of the model operation associated information under the influence of the preset associated information; determine a third characteristic value corresponding to the model operation associated information based on the first characteristic value and the second characteristic value, wherein the third characteristic value is used to reflect the overall characteristic performance level of the model operation associated information; determine the processing capability characteristic value of the big data processing model according to the third characteristic values ​​corresponding to each of the multiple items of the model operation associated information.

[0052] In one embodiment of the present application, optionally, the processing capability characteristic value determination unit is used to: obtain the weight of the first characteristic value and the weight of the second characteristic value, wherein the weight of the first characteristic value reflects the contribution of the model operation associated information in a specified number of historical model optimization operations, and the weight of the second characteristic value reflects the contribution of the preset associated information in a specified number of historical model optimization operations; perform weighted summation of the first characteristic value and the second characteristic value to obtain a third characteristic value corresponding to the model operation associated information.

[0053] In one embodiment of the present application, optionally, the optimization strategy selection unit includes:

[0054] A task processing requirement information acquisition unit, used to acquire the task processing requirement information corresponding to the running state of the big data processing model;

[0055] a difference determination unit, used to determine difference information between the task processing requirement information and the model operation associated information;

[0056] A first execution unit is used to determine that the target optimization strategy is to obtain historical operation association information of the big data processing model if the difference information is within a preset difference range, and optimize the big data processing model based on the historical operation association information and a historical optimizer;

[0057] The second execution unit is used to determine that the target optimization strategy is to select an optimization execution mode with the lowest execution cost from available optimization execution modes if the difference information is not within a preset difference range.

[0058] In one embodiment of the present application, optionally, the device further includes:

[0059] A real-time optimization related information extraction unit is used to obtain real-time optimization related information at specified time intervals during the process of executing the target optimization strategy on the big data processing model, wherein the real-time optimization related information includes: current CPU utilization, current CPU operating voltage, number of threads of the target task, average execution time of threads of the target task, and data processing volume per unit time of each map in the big data processing model;

[0060] a ratio calculation unit, for determining, for each item of the real-time optimization association information, a ratio of the real-time optimization association information to the historical optimization association information corresponding to the real-time optimization association information in the model operation association information before executing the target optimization strategy;

[0061] An optimization capability confidence determination unit, configured to determine the optimization capability confidence of the target optimization strategy based on the ratios corresponding to all of the real-time optimization related information;

[0062] A supplementary optimization unit is used to execute a supplementary optimization strategy on the big data processing model if the optimization capability confidence is lower than a preset optimization capability threshold, wherein the supplementary optimization strategy is data distribution optimization.

[0063] In one embodiment of the present application, optionally, the optimization capability confidence determination unit is used to: for each item of the real-time optimization associated information, obtain a weighted eigenvalue of the real-time optimization associated information, wherein the weight of the real-time optimization associated information reflects the contribution of the real-time optimization associated information in a specified number of historical model optimization operations; use the ratio corresponding to the real-time optimization associated information as the weight of the weighted eigenvalue, perform weighted processing on the weighted eigenvalues ​​of all the real-time optimization associated information, and obtain a fourth eigenvalue corresponding to each of the real-time optimization associated information; determine the variance of the fourth eigenvalue corresponding to each of the real-time optimization associated information as the optimization capability confidence of the target optimization strategy.

[0064] In a third aspect, an embodiment of the present application provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method described in the first aspect above.

[0065] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method described in the first aspect above.

[0066] The above technical scheme aims at the technical problem that the optimization method of the big data processing model in the related technology has many defects and cannot meet the data analysis requirements of the related work of life insurance services. For the data analysis requirements of the related work of life insurance services, in the scenario where multiple users share the big data processing model to process their respective tasks, for the target task of a single user, the optimization time of the big data processing model can be reasonably selected based on the performance of the big data processing model's processing ability for the target task in multiple dimensions. Furthermore, at the appropriate optimization time, a target optimization strategy is selected that can adapt the degree of improvement of the current processing ability of the big data processing model to the requirements of the target task. As a result, the big data processing model can automatically optimize itself in real time when executing different tasks of different users to maximize the adaptation to the target task being executed, improve the adaptability of the big data processing model to multiple users and multiple tasks, increase the accuracy of the big data analysis results, and help to obtain more accurate and reliable analysis data in the related work of life insurance services, thereby improving the level of life insurance services.

Brief Description of the Drawings

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0068] Figure 1 A flowchart of an optimization method for a big data processing model according to an embodiment of the present application is shown;

[0069] Figure 2 A flow chart showing a calculation process of a processing capability characteristic value according to an embodiment of the present application is shown;

[0070] Figure 3 A flow chart of a supplementary optimization method according to an embodiment of the present application is shown;

[0071] Figure 4A block diagram of a computer device according to an embodiment of the present application is shown;

[0072] Figure 5 A block diagram of a computer device according to another embodiment of the present application is shown. [Specific implementation method]

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] Figure 1 A flowchart of an optimization method for a big data processing model according to an embodiment of the present application is shown.

[0075] like Figure 1 As shown, the process of the optimization method for a big data processing model according to an embodiment of the present application includes:

[0076] Step 102, obtaining model operation related information involved in the big data processing model executing the target task for the specified user.

[0077] Among them, the designated users are business stakeholders of life insurance services, including but not limited to sellers, service providers, superior inspection units and any other parties with work content related to life insurance services.

[0078] Among them, the target task includes the data analysis operation of any relevant work link of the life insurance service, and the model operation related information is used to reflect the processing ability of the big data processing model for the target task.

[0079] In this technical solution, the basis for optimizing the big data processing model is the model operation-related information involved in the target task. It can be seen that the model optimization of this technical solution is to improve the ability of the big data processing model to process the target task. Therefore, in the case where multiple users share the big data processing model, the big data processing model can be optimized for the target task of a single user, which helps to improve the processing ability of the big data processing model for the designated task of a single user in the case of multiple users sharing, and improve the task compatibility of the big data processing model.

[0080] The situation where multiple users share big data processing models is involved in all aspects of life insurance services.

[0081] For example, a life insurance service with specified service content can be extended by multiple service extension parties. In this way, in each settlement cycle, each service extension party needs to use the big data processing model for user profiling and classification to profile the customer group of this life insurance service, so as to more accurately screen the target customer group based on the customer group profile, so as to improve its own life insurance service promotion capabilities.

[0082] In this case, there are multiple users of the big data processing model of multiple service extension parties, and each service extension party needs to profile the customer groups of multiple life insurance services. When a single service extension party profiles the customer group of a single life insurance service, it is the target task described in this technical solution.

[0083] For example, at the end of the year, all branches of life insurance services at the municipal and provincial levels across the country need to assess the risk levels of different types of life insurance service projects. Different branches have their own unique life insurance service-related data, and their requirements for data analysis capabilities of the big data processing model used for risk assessment are also different. Therefore, each branch is a user of the big data processing model used for risk assessment, and a single branch conducts a risk level assessment on a single life insurance service project, which is a target task of the big data processing model.

[0084] Therefore, the technical solution in the context of this application is to enable different users to obtain accurate data analysis results suitable for their own target tasks when using the same big data processing model to process the same type of target tasks in any data analysis work related to life insurance services.

[0085] In a possible design, the model operation-related information includes but is not limited to any one or more of the following combinations: current CPU utilization, current CPU operating voltage, number of threads of the target task, average thread execution time of the target task, memory usage of the target task, rate of change of the memory usage within a first specified time period, disk usage of the target task, rate of change of the disk usage within a second specified time period, network input and output of the big data processing model per unit time, and data processing volume of each map in the big data processing model per unit time. Each item of the model operation-related information reflects the processing capability of the big data processing model for the target task in a single dimension.

[0086] Step 104, based on the model operation associated information, determine whether to make the big data processing model enter the optimization mode.

[0087] Since the model operation associated information reflects the processing capability of the big data processing model for the target task, the processing capability of the big data processing model for the target task can be determined based on the model operation associated information.

[0088] Specifically, if the big data processing model has a high processing capability for the target task, there is no need to optimize the model. Conversely, if the big data processing model has a low processing capability for the target task, the big data processing model needs to enter the optimization mode to improve the processing capability of the big data processing model for the target task.

[0089] In one possible design, step 104 is implemented as follows.

[0090] First, based on the model operation-related information, the operation state of the big data processing model is determined, wherein the operation state is a steady state or a high-concurrency state.

[0091] Among them, the running status of the big data processing model reflects the current processing status of the big data processing model for the target task. The steady state means that the current processing capacity of the big data processing model is sufficient to cope with the current number of threads of the target task, and the high concurrency state means that the current processing capacity of the big data processing model is insufficient to cope with the large number of threads currently bursting out of the target task.

[0092] Specifically, if the model operation-related information is all within the stable range corresponding to the steady state, or if the specified items in the model operation-related information are within the stable range corresponding to the steady state, it can be determined that the operation state of the big data processing model is the steady state. On the contrary, if the model operation-related information is not all within the stable range corresponding to the steady state, or if the specified items in the model operation-related information are not within the stable range corresponding to the steady state, it can be determined that the operation state of the big data processing model is the high concurrency state.

[0093] Furthermore, the processing capability characteristic value of the big data processing model can be determined based on the model operation associated information, that is, the abstract value of the processing capability of the big data processing model in the current operating state, that is, the processing capability characteristic value, can be determined by the processing level performance of the big data processing model on the target task.

[0094] Further, it is determined whether the processing capacity characteristic value is greater than or equal to the minimum capacity requirement value corresponding to the operating state. Among them, the degree of demand that the big data processing model can meet for the target task in the steady state and the high concurrency state is different. The minimum capacity requirement value corresponding to the steady state reflects the minimum response capability of the big data processing model to the demand for the target task when it can effectively process the target task in the steady state. Similarly, the minimum capacity requirement value corresponding to the high concurrency state reflects the minimum response capability of the big data processing model to the demand for the target task when it can effectively process the target task in the high concurrency state.

[0095] At this point, if the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, it is determined that the big data processing model enters the optimization mode. Among them, if the big data processing model is in a steady state, its processing capability characteristic value is greater than or equal to the minimum capability requirement value in a steady state, indicating that it can meet the minimum requirement of the target task in a steady state when processing the target task. Similarly, if the big data processing model is in a high concurrency state, its processing capability characteristic value is greater than or equal to the minimum capability requirement value in a high concurrency state, indicating that it can meet the minimum requirement of the target task in a high concurrency state when processing the target task.

[0096] Therefore, the processing capability of the big data processing model for the target task in the current state can be accurately reflected based on the model operation related information, so as to reasonably select the timing for optimizing the big data processing model based on the processing capability of the big data processing model for the target task in the current state, thereby improving the reliability of the selection of the optimization timing, and helping to optimize the big data processing model in time when it is most in need of capacity improvement, thereby improving its big data analysis capabilities.

[0097] Step 106: If the big data processing model enters the optimization mode, based on the model operation-related information, a target optimization strategy matching the target task is determined from a plurality of preset optimization strategies.

[0098] Step 108: Execute the target optimization strategy on the big data processing model.

[0099] Since the model operation associated information reflects the processing capability of the big data processing model for the target task, the difference between the processing capability and the processing capability required for the target task reflects the degree to which the big data processing model needs to be optimized. Furthermore, the degree to which the big data processing model needs to be optimized can determine the method to be used to perform the optimization. In other words, the degree to which the big data processing model needs to be optimized needs to match the optimization capability of the optimization method. Therefore, in the optimization mode, based on the processing capability of the big data processing model reflected by the model operation associated information, a target optimization strategy can be selected from a variety of preset optimization strategies to improve the processing capability to adapt to the needs of the target task. Finally, the big data processing model is optimized and trained using the target optimization strategy.

[0100] The above technical scheme, aimed at the data analysis needs of the work related to life insurance services, in the scenario where multiple users share a big data processing model to process their respective tasks, can reasonably select the optimization time of the big data processing model for the target task of a single user based on the performance of the big data processing model's processing ability for the target task in multiple dimensions. Furthermore, at the appropriate optimization time, a target optimization strategy is selected to improve the current processing ability of the big data processing model to the extent that it can adapt to the needs of the target task. As a result, the big data processing model can automatically optimize itself in real time when executing different tasks of different users, so as to adapt to the target task being executed to the greatest extent, improve the adaptability of the big data processing model to multiple users and multiple tasks, increase the accuracy of the big data analysis results, and help to obtain more accurate and reliable analysis data in the work related to life insurance services, thereby improving the level of life insurance services.

[0101] In a possible design, based on the model operation association information, a specific method of determining a target optimization strategy matching the target task from a plurality of preset optimization strategies is as follows:

[0102] First, the task processing requirement information corresponding to the running state of the big data processing model is obtained. The running state of the big data processing model reflects the current processing status of the big data processing model for the target task. In other words, the current processing status of the big data processing model for the target task can explain the big data analysis requirement of the target task. This big data analysis requirement is the task processing requirement information of the target task.

[0103] Next, the difference information between the task processing requirement information and the model operation association information is determined. This difference information reflects the gap between the big data analysis capability required by the target task and the actual analysis capability of the big data processing model currently running.

[0104] The task processing requirement information corresponds to one or more items of information in the model operation associated information.

[0105] Taking the memory usage of the target task in the model running associated information as an example, the expected memory usage of the target task in the task processing requirement information can be obtained, and the difference between the actual memory usage of the target task in the model running associated information and the expected memory usage can be obtained and added to the difference information.

[0106] Taking the network input amount per unit time of the big data processing model as an example, the reasonable network input amount expected for the target task in the task processing requirement information can be obtained. This reasonable network input amount is the minimum network input amount per unit time required for the big data processing model to complete the target task within a reasonable time. Therefore, the difference between the actual network input amount per unit time of the big data processing model and the minimum network input amount per unit time can be obtained and added to the difference information.

[0107] Accordingly, a corresponding preset difference range can be set for the difference information corresponding to each task processing requirement information. The preset difference range represents the range of the difference between the model operation associated information and the task processing requirement information when the big data processing model can meet the task processing requirement of the target task.

[0108] Therefore, if the difference information is within the preset difference range, it means that the big data processing model is in a relatively balanced and stable processing state. At this time, the operation rules of the big data processing model are consistent with the historical level. The target optimization strategy can be determined as obtaining the historical operation related information of the big data processing model, and optimizing the big data processing model based on the historical operation related information and the historical optimizer, that is, optimizing the big data processing model according to the historical operation status of the big data processing model.

[0109] On the contrary, if the difference information is not within the preset difference range, it means that the big data processing model is in a relatively unstable abnormal state. At this time, the operation rules of the big data processing model are inconsistent with the historical level, and the historical operation of the big data processing model is difficult to serve as an effective reference object. In this regard, the target optimization strategy can be determined to select the optimization execution method with the lowest execution cost among the available optimization execution methods, that is, to use CBO to optimize the big data processing model.

[0110] Therefore, based on the difference between the expected requirements of the target task for the big data processing model and the actual processing capabilities of the big data processing model, an optimization strategy that is more suitable for the target task can be selected, so that the selected optimization strategy can make the big data processing model fit the actual processing requirements of the target task as much as possible, thereby improving the adaptability of the big data processing model to the target task.

[0111] Figure 2A flow chart of a calculation process of a processing capability characteristic value according to an embodiment of the present application is shown.

[0112] like Figure 2 As shown, the calculation process of the processing capability characteristic value according to one embodiment of the present application is as follows:

[0113] Step 202: encode each item of the model operation associated information according to a preset encoding method, and normalize the encoding result to obtain a first eigenvalue corresponding to each item of the model operation associated information.

[0114] The preset encoding method includes, but is not limited to, any encoding method such as ASCII (American Standard Code for Information Interchange), Unicode (Unicode, Unicode), UTF-8 (Unicode Transformation Format-8, Unicode character encoding), base64, onehot (single hot encoding), etc. The encoding method of each item of the model operation associated information can be any one of the preset encoding methods.

[0115] Furthermore, the encoding results are normalized so that the first eigenvalues ​​corresponding to the model operation associated information are at the same level to increase the accuracy of subsequent calculation results. It can be understood that the first eigenvalue is a numerical expression of the processing capability of the big data processing model for the target task reflected by the model operation associated information.

[0116] Step 204: for each item of the model operation associated information, obtain preset associated information of the model operation associated information in all the model operation associated information.

[0117] Each item of the model operation associated information has several other items of model operation associated information associated with it.

[0118] For example, the greater the CPU load, the higher the current CPU operating voltage, and the current CPU utilization, the number of threads of the target task, the average execution time of the threads of the target task, and the network input amount of the big data processing model per unit time are all proportional to the CPU load. Therefore, the preset associated information of the current CPU operating voltage is the current CPU utilization, the number of threads of the target task, the average execution time of the threads of the target task, and the network input amount of the big data processing model per unit time.

[0119] For another example, the more content the target task processes, the larger the memory usage of the target task, while the current CPU utilization, the number of threads of the target task, the average execution time of the threads of the target task, and the network input volume of the big data processing model per unit time are all proportional to the amount of content processed by the target task. At the same time, the rate of change of the memory usage within the first specified time period reflects the level of change of the memory usage of the target task. Furthermore, the higher the disk usage of the target task, the higher the memory data written to the disk, and the smaller the memory usage. The disk usage of the target task is inversely proportional to the memory usage of the target task. Relatedly, the rate of change of the disk usage within the second specified time period reflects the level of change of the disk usage of the target task, which is indirectly related to the memory usage of the target task. Therefore, the preset associated information of the memory usage of the target task is the current CPU utilization, the number of threads of the target task, the average thread execution time of the target task, the network input volume of the big data processing model per unit time, the rate of change of the memory usage within a first specified time period, the disk usage of the target task, and the rate of change of the disk usage within a second specified time period.

[0120] The first specified duration and the second specified duration are any durations that meet the actual operational capability assessment requirements of the model.

[0121] Step 206: Determine a second eigenvalue based on the first eigenvalue corresponding to the model running association information and the first eigenvalue corresponding to the preset association information.

[0122] The second characteristic value is used to reflect the characteristic performance level of the model operation associated information under the influence of the preset associated information.

[0123] Since the single item of model operation associated information reflects the processing capability of the big data processing model for the target task in a single dimension, its preset associated information is used to reflect the performance level of the model processing capability in other dimensions associated with the single dimension. Further, the second eigenvalue reflects the final performance of the processing capability of the big data processing model for the target task in a single dimension after being affected by the model processing capability in other single dimensions associated with the single dimension.

[0124] In a possible design, the average of the first eigenvalue corresponding to the model operation associated information and the first eigenvalue corresponding to the preset associated information is determined as the second eigenvalue.

[0125] In another possible design, a weighted average of the first eigenvalue corresponding to the model operation associated information and the first eigenvalue corresponding to the preset associated information is obtained as the second eigenvalue. The weight of each first eigenvalue is the weight of each corresponding model operation associated information, that is, the contribution of each model operation associated information in the specified multiple historical model optimization operations.

[0126] Step 208: Determine a third eigenvalue corresponding to the model operation association information based on the first eigenvalue and the second eigenvalue.

[0127] The first eigenvalue can be said to reflect the contribution level of the single model operation related information to the overall processing capability of the big data processing model. The second eigenvalue can be said to reflect the contribution level of the single model operation related information to the overall processing capability of the big data processing model after being affected by the preset related information. The third eigenvalue obtained by combining the two is used to reflect the overall characteristic performance level of the single model operation related information in terms of its processing capability performance during the operation of the big data processing model.

[0128] In one possible design, the third eigenvalue is calculated as follows:

[0129] First, a weight of the first eigenvalue and a weight of the second eigenvalue are obtained.

[0130] The weight of the first eigenvalue reflects the contribution of the model operation associated information in a specified number of historical model optimization operations, and the weight of the second eigenvalue reflects the contribution of the preset associated information in a specified number of historical model optimization operations.

[0131] Next, a weighted sum is performed on the first eigenvalue and the second eigenvalue to obtain a third eigenvalue corresponding to the model operation association information.

[0132] Step 210, determining the processing capability characteristic value of the big data processing model according to the third characteristic values ​​corresponding to each of the multiple pieces of model operation associated information.

[0133] In a possible design, the sum of the third eigenvalues ​​corresponding to the multiple pieces of model operation associated information is used as the processing capability eigenvalue of the big data processing model.

[0134] In another possible design, the variance of the third eigenvalues ​​corresponding to the multiple pieces of model operation associated information is used as the processing capability eigenvalue of the big data processing model.

[0135] In another possible design, the square root of the sum of the third eigenvalues ​​corresponding to the multiple pieces of model operation-related information is used as the processing capability eigenvalue of the big data processing model to further reduce the entropy value of the sum and improve the accuracy of the assessment of the processing capability of the big data processing model.

[0136] Through the above technical scheme, the data analysis capability of the big data processing model can be automatically and accurately evaluated, so as to select the appropriate optimization time based on the accurate data analysis capability evaluation results, so that the big data processing model can automatically start the optimization strategy at the appropriate time when executing different tasks of different users, so as to maximize the adaptation to the target tasks being executed, improve the adaptability of the big data processing model to multiple users and multiple tasks, increase the accuracy of big data analysis results, and help to obtain more accurate and reliable analysis data in the related work of life insurance services, thereby improving the level of life insurance services.

[0137] exist Figure 1 and Figure 2 On the basis of the illustrated embodiment, after the aforementioned selected target optimization strategy is executed, or during the execution of the aforementioned selected target optimization strategy, the big data processing model may be further optimized.

[0138] Figure 3 A flow chart of a supplementary optimization method according to an embodiment of the present application is shown.

[0139] like Figure 3 As shown, the supplementary optimization method according to one embodiment of the present application includes:

[0140] Step 302, in the process of executing the target optimization strategy on the big data processing model, obtaining real-time optimization related information at every specified time interval.

[0141] The real-time optimization related information includes: current CPU utilization, current CPU operating voltage, number of threads of the target task, average thread execution time of the target task, and data processing volume per unit time of each map in the big data processing model. It can be seen that the real-time optimization related information is part of the model operation related information, or in other words, this part of the model operation related information can be used as a judgment condition for whether to start the supplementary optimization strategy.

[0142] Step 304: for each item of the real-time optimization association information, determine the ratio of the real-time optimization association information to the historical optimization association information corresponding to the real-time optimization association information in the model operation association information before executing the target optimization strategy.

[0143] Before executing the target optimization strategy, the real-time optimization association information corresponds to the historical optimization association information in the model operation association information, that is, the historical performance of the real-time optimization association information before being optimized by the target optimization strategy. It can be seen that the ratio reflects the growth rate or reduction rate of the real-time optimization association information before and after executing the target optimization strategy.

[0144] Step 306: Determine the optimization capability confidence of the target optimization strategy based on the ratios corresponding to all the real-time optimization related information.

[0145] The ratios corresponding to all the real-time optimization related information reflect the size changes of each real-time optimization related information before and after the execution of the target optimization strategy, and these size changes reflect the changes of the current big data processing model after being affected by the execution of the target optimization strategy, which is the embodiment of the optimization ability of the target optimization strategy. Therefore, the optimization ability confidence of the target optimization strategy can be determined based on the ratios corresponding to all the real-time optimization related information.

[0146] In a possible design, the specific method of calculating the optimization capability confidence is as follows.

[0147] First, for each item of the real-time optimization associated information, a weighted characteristic value of the real-time optimization associated information is obtained, wherein the weight of the real-time optimization associated information reflects the contribution of the real-time optimization associated information in the specified multiple historical model optimization operations.

[0148] Next, the weighted characteristic values ​​of all the real-time optimization associated information are weighted by taking the ratio corresponding to the real-time optimization associated information as the weight of the weighted characteristic value, and obtaining the fourth characteristic value corresponding to all the real-time optimization associated information. It can be understood that the fourth characteristic value reflects the comprehensive level of the size change of the single real-time optimization associated information before and after the execution of the target optimization strategy and the change performance of the real-time optimization associated information in the historical optimization operation.

[0149] Finally, the variance of the fourth eigenvalues ​​corresponding to all the real-time optimization related information is determined as the optimization capability confidence of the target optimization strategy.

[0150] Step 308: If the optimization capability confidence level is lower than a preset optimization capability threshold, a supplementary optimization strategy is executed on the big data processing model.

[0151] The preset optimization capability threshold is the minimum optimization capability confidence of the target optimization strategy when executing the target optimization strategy so that the big data processing model can meet the actual data analysis requirements of the target task. Therefore, if the optimization capability confidence is lower than the preset optimization capability threshold, it means that optimizing the big data processing model using only the target optimization strategy cannot meet the actual data analysis requirements of the target task. Therefore, the supplementary optimization strategy can be started to further optimize the big data processing model.

[0152] In a possible design, the supplementary optimization strategy is data distribution optimization.

[0153] In another possible design, the supplementary optimization strategy is any one or more of data distribution optimization, CBO and HBO.

[0154] Through the above technical scheme, while or after executing the target optimization strategy, the ability of the target optimization strategy to optimize the big data processing model can continue to be evaluated in combination with the actual data analysis needs of the designated tasks of a single user, and when the ability of the target optimization strategy to optimize the big data processing model is insufficient, a supplementary optimization strategy can be adopted to further improve the adaptability of the big data processing model to the designated tasks of a single user. When executing different tasks of different users, the big data processing model can not only automatically optimize itself in real time, but also evaluate the level of this optimization in real time or later, and perform supplementary optimization when the optimization level is insufficient, so as to maximize the adaptation to the target task being executed, thereby improving the adaptability of the big data processing model to multiple users and multiple tasks, and increasing the accuracy of big data analysis results, which is helpful to obtain more accurate and reliable analysis data in related work of life insurance services, thereby improving the level of life insurance services.

[0155] In addition, the present application embodiment provides an optimization device for a big data processing model, including:

[0156] A model operation related information acquisition unit, used to acquire model operation related information involved in the execution of a target task by a big data processing model for a designated user, wherein the designated user is a business-related party of a life insurance service, the target task includes a data analysis operation of any relevant work link of the life insurance service, and the model operation related information is used to reflect the processing capability of the big data processing model for the target task;

[0157] An optimization timing determination unit, configured to determine whether to make the big data processing model enter an optimization mode based on the model operation associated information;

[0158] An optimization strategy selection unit, configured to determine a target optimization strategy matching the target task from a plurality of preset optimization strategies based on the model operation associated information if the big data processing model enters the optimization mode;

[0159] An optimization strategy execution unit is used to execute the target optimization strategy on the big data processing model.

[0160] In one embodiment of the present application, optionally, the model operation associated information includes: the current CPU utilization, the current CPU operating voltage, the number of threads of the target task, the average execution time of the threads of the target task, the memory occupancy of the target task, the rate of change of the memory occupancy within a first specified time period, the disk occupancy of the target task, the rate of change of the disk occupancy within a second specified time period, the network input and network output of the big data processing model per unit time, and the data processing volume of each map in the big data processing model per unit time.

[0161] In one embodiment of the present application, optionally, the optimization timing determination unit includes:

[0162] An operation state determining unit, configured to determine the operation state of the big data processing model based on the model operation associated information, wherein the operation state is a steady state or a high concurrency state;

[0163] A processing capability characteristic value determining unit, configured to determine a processing capability characteristic value of the big data processing model based on the model operation associated information;

[0164] A capability level judgment unit is used to determine whether the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, wherein if the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, it is determined that the big data processing model enters the optimization mode.

[0165] In one embodiment of the present application, optionally, the processing capability characteristic value determination unit is used to: encode each item of the model operation associated information according to a preset encoding method, and normalize the encoding result to obtain a first characteristic value corresponding to each item of the model operation associated information; for each item of the model operation associated information, obtain the preset associated information of the model operation associated information in all the model operation associated information; determine a second characteristic value based on the first characteristic value corresponding to the model operation associated information and the first characteristic value corresponding to the preset associated information, wherein the second characteristic value is used to reflect the characteristic performance level of the model operation associated information under the influence of the preset associated information; determine a third characteristic value corresponding to the model operation associated information based on the first characteristic value and the second characteristic value, wherein the third characteristic value is used to reflect the overall characteristic performance level of the model operation associated information; determine the processing capability characteristic value of the big data processing model according to the third characteristic values ​​corresponding to each of the multiple items of the model operation associated information.

[0166] In one embodiment of the present application, optionally, the processing capability characteristic value determination unit is used to: obtain the weight of the first characteristic value and the weight of the second characteristic value, wherein the weight of the first characteristic value reflects the contribution of the model operation associated information in a specified number of historical model optimization operations, and the weight of the second characteristic value reflects the contribution of the preset associated information in a specified number of historical model optimization operations; perform weighted summation of the first characteristic value and the second characteristic value to obtain a third characteristic value corresponding to the model operation associated information.

[0167] In one embodiment of the present application, optionally, the optimization strategy selection unit includes:

[0168] A task processing requirement information acquisition unit, used to acquire the task processing requirement information corresponding to the running state of the big data processing model;

[0169] a difference determination unit, used to determine difference information between the task processing requirement information and the model operation associated information;

[0170] A first execution unit is used to determine that the target optimization strategy is to obtain historical operation association information of the big data processing model if the difference information is within a preset difference range, and optimize the big data processing model based on the historical operation association information and a historical optimizer;

[0171] The second execution unit is used to determine that the target optimization strategy is to select an optimization execution mode with the lowest execution cost from available optimization execution modes if the difference information is not within a preset difference range.

[0172] In one embodiment of the present application, optionally, the device further includes:

[0173] A real-time optimization related information extraction unit is used to obtain real-time optimization related information at specified time intervals during the process of executing the target optimization strategy on the big data processing model, wherein the real-time optimization related information includes: current CPU utilization, current CPU operating voltage, number of threads of the target task, average execution time of threads of the target task, and data processing volume per unit time of each map in the big data processing model;

[0174] a ratio calculation unit, for determining, for each item of the real-time optimization association information, a ratio of the real-time optimization association information to the historical optimization association information corresponding to the real-time optimization association information in the model operation association information before executing the target optimization strategy;

[0175] An optimization capability confidence determination unit, configured to determine the optimization capability confidence of the target optimization strategy based on the ratios corresponding to all of the real-time optimization related information;

[0176] A supplementary optimization unit is used to execute a supplementary optimization strategy on the big data processing model if the optimization capability confidence is lower than a preset optimization capability threshold, wherein the supplementary optimization strategy is data distribution optimization.

[0177] In one embodiment of the present application, optionally, the optimization capability confidence determination unit is used to: for each item of the real-time optimization associated information, obtain a weighted eigenvalue of the real-time optimization associated information, wherein the weight of the real-time optimization associated information reflects the contribution of the real-time optimization associated information in a specified number of historical model optimization operations; use the ratio corresponding to the real-time optimization associated information as the weight of the weighted eigenvalue, perform weighted processing on the weighted eigenvalues ​​of all the real-time optimization associated information, and obtain a fourth eigenvalue corresponding to each of the real-time optimization associated information; determine the variance of the fourth eigenvalue corresponding to each of the real-time optimization associated information as the optimization capability confidence of the target optimization strategy.

[0178] The device uses any one of the solutions described in the above embodiments, and therefore has all the above technical effects, which will not be repeated here.

[0179] In addition, in one embodiment, the present application provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the method described in any of the above embodiments can be implemented.

[0180] In one embodiment, the present application further provides a computer device, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the method described in any of the above embodiments can be implemented.

[0181] Any of the above-mentioned computer devices in the embodiments of the present application may exist in various forms, including but not limited to:

[0182] (1) Mobile communication devices: These devices are characterized by their mobile communication functions and their main purpose is to provide voice and data communications. These terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones.

[0183] (2) Ultra-mobile personal computer devices: These devices fall into the category of personal computers, have computing and processing capabilities, and generally also have mobile Internet access features. These terminals include: PDA, MID and UMPC devices, such as iPad.

[0184] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players (such as iPods), handheld game consoles, e-books, as well as smart toys, wearable devices, and portable car navigation devices.

[0185] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but because it needs to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, and manageability.

[0186] (5) Other electronic devices with data interaction functions.

[0187] In addition, an embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to perform the following steps:

[0188] Obtaining model operation association information involved in the execution of a target task by a big data processing model for a designated user, wherein the designated user is a business-related party of a life insurance service, the target task includes a data analysis operation of any relevant work link of the life insurance service, and the model operation association information is used to reflect the processing capability of the big data processing model for the target task;

[0189] Determining whether to put the big data processing model into an optimization mode based on the model operation associated information;

[0190] If the big data processing model enters the optimization mode, based on the model operation associated information, a target optimization strategy matching the target task is determined from a plurality of preset optimization strategies;

[0191] The target optimization strategy is executed on the big data processing model.

[0192] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.

[0193] The technical solution of the present application is described in detail above in conjunction with the accompanying drawings. The technical solution of the present application is aimed at the data analysis needs of the related work of life insurance services. In the scenario where multiple users share a big data processing model to process their respective tasks, for the target task of a single user, the optimization time of the big data processing model can be reasonably selected based on the performance of the big data processing model's processing ability for the target task in multiple dimensions. Furthermore, at the appropriate optimization time, a target optimization strategy is selected to improve the current processing ability of the big data processing model to a degree that can adapt to the needs of the target task. As a result, the big data processing model can automatically optimize itself in real time when executing different tasks of different users, so as to adapt to the target task being executed to the greatest extent, improve the adaptability of the big data processing model to multiple users and multiple tasks, increase the accuracy of the big data analysis results, and help to obtain more accurate and reliable analysis data in the related work of life insurance services, thereby improving the level of life insurance services.

[0194] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0195] It should be understood that, although the terms first, second, etc. may be used to describe characteristic values ​​in the embodiments of the present application, these characteristic values ​​should not be limited to these terms. These terms are only used to distinguish characteristic values ​​from each other. For example, without departing from the scope of the embodiments of the present application, the first characteristic value may also be referred to as the second characteristic value, and similarly, the second characteristic value may also be referred to as the first characteristic value.

[0196] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0197] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0198] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0200] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0201] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An optimization method for a big data processing model, characterized in that: include: Obtaining model operation association information involved in the execution of a target task by a big data processing model for a designated user, wherein the designated user is a business-related party of a life insurance service, the target task includes a data analysis operation of any relevant work link of the life insurance service, and the model operation association information is used to reflect the processing capability of the big data processing model for the target task; Determining whether to put the big data processing model into an optimization mode based on the model operation associated information; If the big data processing model enters the optimization mode, based on the model operation associated information, a target optimization strategy matching the target task is determined from a plurality of preset optimization strategies; The target optimization strategy is executed on the big data processing model.

2. The method according to claim 1, characterized in that The model operation associated information includes: the current CPU utilization, the current CPU operating voltage, the number of threads of the target task, the average thread execution time of the target task, the memory occupancy of the target task, the change rate of the memory occupancy within a first specified time period, the disk occupancy of the target task, the change rate of the disk occupancy within a second specified time period, the network input and network output of the big data processing model per unit time, and the data processing volume of each map in the big data processing model per unit time.

3. The method according to claim 2, characterized in that The step of determining whether to make the big data processing model enter an optimization mode based on the model operation associated information includes: Determining an operation state of the big data processing model based on the model operation associated information, wherein the operation state is a steady state or a high concurrency state; and Determining a processing capability characteristic value of the big data processing model based on the model operation associated information; Determine whether the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, wherein if the processing capability characteristic value is greater than or equal to the minimum capability requirement value corresponding to the operating state, determine to make the big data processing model enter the optimization mode.

4. The method according to claim 3, characterized in that The determining the processing capability characteristic value of the big data processing model based on the model operation associated information includes: Encode each item of the model operation associated information according to a preset encoding method, and normalize the encoding result to obtain a first eigenvalue corresponding to each item of the model operation associated information; For each of the model runs associated information, Obtaining preset associated information of the model operation associated information in all the model operation associated information; Determining a second eigenvalue based on the first eigenvalue corresponding to the model operation associated information and the first eigenvalue corresponding to the preset associated information, wherein the second eigenvalue is used to reflect the characteristic performance level of the model operation associated information under the influence of the preset associated information; Based on the first eigenvalue and the second eigenvalue, determining a third eigenvalue corresponding to the model operation associated information, wherein the third eigenvalue is used to reflect the overall characteristic performance level of the model operation associated information; The processing capability characteristic value of the big data processing model is determined according to the third characteristic values ​​corresponding to each of the multiple pieces of model operation associated information.

5. The method according to claim 4, characterized in that The determining, based on the first eigenvalue and the second eigenvalue, a third eigenvalue corresponding to the model operation associated information includes: Obtaining a weight of the first eigenvalue and a weight of the second eigenvalue, wherein the weight of the first eigenvalue reflects the contribution of the model operation associated information in a specified number of historical model optimization operations, and the weight of the second eigenvalue reflects the contribution of the preset associated information in a specified number of historical model optimization operations; A weighted sum is performed on the first eigenvalue and the second eigenvalue to obtain a third eigenvalue corresponding to the model operation associated information.

6. The method according to any one of claims 3 to 5, characterized in that The step of determining a target optimization strategy matching the target task from a plurality of preset optimization strategies based on the model operation association information includes: Obtaining task processing requirement information corresponding to the running status of the big data processing model; Determine difference information between the task processing requirement information and the model operation associated information; If the difference information is within a preset difference range, determining that the target optimization strategy is to obtain historical operation-related information of the big data processing model, and optimizing the big data processing model based on the historical operation-related information and a historical optimizer; If the difference information is not within the preset difference range, the target optimization strategy is determined to be selecting an optimization execution mode with the lowest execution cost from available optimization execution modes.

7. The method according to any one of claims 1 to 5, characterized in that Also includes: In the process of executing the target optimization strategy on the big data processing model, real-time optimization related information is obtained at every specified time interval, wherein the real-time optimization related information includes: current CPU utilization, current CPU operating voltage, number of threads of the target task, average execution time of threads of the target task, and data processing volume per unit time of each map in the big data processing model; For each item of the real-time optimization association information, determining a ratio of the real-time optimization association information to the historical optimization association information corresponding to the real-time optimization association information in the model operation association information before executing the target optimization strategy; Determining the optimization capability confidence of the target optimization strategy based on the ratios corresponding to all the real-time optimization related information; If the optimization capability confidence is lower than a preset optimization capability threshold, a supplementary optimization strategy is executed on the big data processing model, wherein the supplementary optimization strategy is data distribution optimization.

8. The method according to claim 7, characterized in that The step of determining the optimization capability confidence of the target optimization strategy based on the ratios corresponding to all the real-time optimization related information includes: For each item of the real-time optimization associated information, a weighted characteristic value of the real-time optimization associated information is obtained, wherein the weight of the real-time optimization associated information reflects the contribution of the real-time optimization associated information in a specified number of historical model optimization operations; Taking the ratio corresponding to the real-time optimization associated information as the weight of the weighted eigenvalue, weighted processing is performed on the weighted eigenvalues ​​of all the real-time optimization associated information to obtain fourth eigenvalues ​​corresponding to all the real-time optimization associated information; The variance of the fourth eigenvalues ​​corresponding to all the real-time optimization associated information is determined as the optimization capability confidence of the target optimization strategy.

9. A computer device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to enable the processor to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and the computer executable instructions are configured to execute the method according to any one of claims 1 to 8.

Images