Method and device for realizing task computing and adaptive parameter adaptation based on xinchuang big data platform, processor and storage medium thereof

Abstract

The present application relates to a kind of method for realizing task computing and adaptive parameter adaptation based on Xinxing big data platform, comprising the following steps: configuration database address and data synchronization frequency;Calculate index data;Select different decision model, if it is new index, then according to GBT regression model generates the parameter required for calculating new index;If it is stock index, then from knowledge base model obtains the parameter value for calculating stock index, according to knowledge base model generates the parameter required for calculating stock index;GBT regression model and the parameter and running package required for calculating task generated by knowledge base model are received.The method for realizing task computing and adaptive parameter adaptation based on Xinxing big data platform, device, processor and its computer readable storage medium of the present application, based on hybrid decision model, improve the index calculation service efficiency, robustness and security, knowledge base model learns sustainable, continuously optimize model precision, continuously improve coverage and accuracy through closed-loop feedback.

Description

Technical Field

[0001] This invention relates to the field of finance and securities, and more particularly to the field of trading strategy platforms. Specifically, it refers to a method, apparatus, processor, and computer-readable storage medium for task calculation and adaptive parameter adaptation based on a domestic big data platform. Background Technology

[0002] To meet the investment decision-making needs of securities clients across multiple scenarios, the trading strategy platform needs to calculate and process strategy indicators based on massive amounts of market data. With the help of the domestic big data platform, the Java application indicator calculation program is submitted to the big data platform via Spark-submit to complete the calculation of complex indicator tasks.

[0003] As the financial securities industry increases its requirements for the security, stability, and intelligence of customer service systems, the development of existing trading strategy platforms faces certain challenges. Current trading strategy platforms require manual setting of runtime parameters when submitting calculation tasks, and these parameters cannot be dynamically modified based on system performance afterward. Furthermore, the middleware and other components used by existing trading strategy platforms are not entirely domestically produced, posing certain security vulnerabilities. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, processor and computer-readable storage medium for task computing and adaptive parameter adaptation based on a domestically developed big data platform, which is characterized by high security, high computing service efficiency and wide applicability.

[0005] To achieve the above objectives, the present invention provides a method, apparatus, processor, and computer-readable storage medium for task computation and adaptive parameter adaptation based on a domestically developed big data platform, as follows:

[0006] The method for task computation and adaptive parameter adaptation based on the domestic IT innovation big data platform is characterized by the following steps:

[0007] (1) Prepare the information technology innovation big data platform, indicator calculation service, adaptive parameter adjuster, data collector and information technology innovation database;

[0008] (2) Install the information technology data acquisition device, configure the database address and data synchronization frequency, and configure the national cryptographic algorithm SM2 to encrypt and transmit data;

[0009] (3) Obtain initialization parameters and calculate indicator data;

[0010] (4) Receive the submitted task and select different decision models according to the type of indicator task. If it is a new indicator, continue to step (5); if it is an existing indicator, continue to step (6).

[0011] (5) Generate the parameters required to calculate the new indicators based on the GBT regression model;

[0012] (6) Obtain the parameter values ​​for calculating stock indicators from the knowledge base model, and generate the parameters required for calculating stock indicators based on the knowledge base model;

[0013] (7) Receive the parameters and runtime packages required for the computation tasks generated by the GBT regression model and knowledge base model, and submit them to the information technology innovation big data platform;

[0014] (8) Perform resource scheduling and performance monitoring.

[0015] Preferably, step (2) specifically includes the following steps:

[0016] (2.1) Install the domestic IT data acquisition tool and download the corresponding database driver, then import it into the installation directory;

[0017] (2.2) Select the external data source and the database type of the domestic IT innovation database, configure the database address and data synchronization frequency, and verify whether the connection is normal. If it is normal, continue to step (2.3); otherwise, continue to step (2.2).

[0018] (2.3) Load the data acquisition script, configure the national cryptographic algorithm SM2 to encrypt and transmit data, and import external historical market data into the information technology innovation database.

[0019] Preferably, step (3) specifically includes the following steps:

[0020] (3.1) Load historical market data from the domestic IT innovation database;

[0021] (3.2) Obtain the initialization parameters required for submitting the task from the parameter knowledge base;

[0022] (3.3) Package the indicator calculation service into a jar file and submit the jar file to the task decision service.

[0023] Preferably, step (5) specifically includes the following steps:

[0024] (5.1) Construct a feature model;

[0025] (5.2) Optimize the model training;

[0026] (5.3) Evaluate the computation time eva_time of the prediction data based on the regression model, prediction dataset and computation task dynamic parameters, read the information innovation database to obtain the task computation time compute_time of the information innovation big data platform, use the regression task evaluator to calculate the root mean square error, and evaluate the model performance.

[0027] (5.4) Update the knowledge base model based on the input and output features trained by the model;

[0028] (5.5) Adjust the dynamic parameter feature values ​​dynamically by adding or removing them according to the server configuration and model performance.

[0029] Preferably, step (5.1) specifically includes the following steps:

[0030] (5.1.1) Construct time window features and calculate the average time spent on past indicators;

[0031] (5.1.2) Input system resource characteristics, data scale characteristics, and dynamic parameter characteristics of computing tasks, and output time series characteristics;

[0032] (5.1.3) Combine system resources, data scale, dynamic parameter characteristics of computing tasks, and time series characteristics into a single vector column;

[0033] (5.1.4) Standardize the process.

[0034] Preferably, step (5.2) specifically includes the following steps:

[0035] (5.2.1) Divide the dataset into training set data and prediction set data according to the time series;

[0036] (5.2.2) Based on the training set data, the GBT regression model is used to train and obtain the regression model.

[0037] Preferably, step (6) specifically includes the following steps:

[0038] (6.1) Store the data in the parameter knowledge base and calculate the task parameter values;

[0039] (6.2) Set the confidence threshold, determine the confidence level, and trigger adaptive parameter adjustment.

[0040] Preferably, step (6.1) specifically includes the following steps:

[0041] (6.1.1) The input and output feature values ​​generated by the GBT regression model and the running results of performance monitoring are filtered by the rule engine and stored in the parameter knowledge base;

[0042] (6.1.2) Provide the parameter values ​​required for the stock index calculation task, input the system resource characteristics and data scale characteristics, and obtain the closest calculation task parameter values ​​based on the similarity variance.

[0043] Preferably, step (6.2) specifically includes the following steps:

[0044] (6.2.1) Store the performance data from training, tuning, and performance monitoring into the parameter knowledge base through the rule engine;

[0045] (6.2.2) Set confidence thresholds based on system resource characteristics and data scale characteristics;

[0046] (6.2.3) Determine the confidence level based on the performance monitoring index data. If the confidence level is low, trigger adaptive parameter adjustment; if the confidence level is medium, trigger model retraining; if the confidence level is high, store the feature values ​​and performance data in the parameter knowledge base.

[0047] Preferably, step (8) specifically includes the following steps:

[0048] (8.1) Access and read data through the standard interface;

[0049] (8.2) Receive the tasks and parameters submitted by the adaptive parameter regulator, complete resource scheduling and task allocation, and calculate the index tasks;

[0050] (8.3) Store the calculated performance data and results data in a knowledge base;

[0051] (8.4) Monitor the server's operating performance in real time through the performance monitoring service. If the threshold is reached, an alarm will be issued and the performance data will be stored in the knowledge base.

[0052] The device for task computation and adaptive parameter adaptation based on the domestic IT innovation big data platform is characterized by the following:

[0053] A processor is configured to execute computer-executable instructions;

[0054] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the various steps of the above-described method for task computation and adaptive parameter adaptation based on the domestic big data platform.

[0055] The processor that implements task computation and adaptive parameter adaptation based on the domestically developed big data platform is characterized in that the processor is configured to execute computer-executable instructions, and when the computer-executable instructions are executed by the processor, the various steps of the above-mentioned method for implementing task computation and adaptive parameter adaptation based on the domestically developed big data platform are realized.

[0056] The main feature of this computer-readable storage medium is that it stores a computer program thereon, which can be executed by a processor to implement the various steps of the above-described method for task calculation and adaptive parameter adaptation based on the domestic big data platform.

[0057] The present invention employs a method, apparatus, processor, and computer-readable storage medium for task computation and adaptive parameter adaptation based on a domestically developed big data platform. Based on the domestically developed big data platform, the platform is more secure. Based on a hybrid decision model, it improves the efficiency, robustness, and security of indicator computation services. The knowledge base model learning is sustainable, continuously optimizing model accuracy, and continuously improving coverage and accuracy through closed-loop feedback. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method for task calculation and adaptive parameter adaptation based on the domestic IT innovation big data platform of the present invention.

[0059] Figure 2 This is a structural diagram of the execution system of the method for task calculation and adaptive parameter adaptation based on the domestic big data platform of the present invention. Detailed Implementation

[0060] To more clearly describe the technical content of the present invention, the following description is provided in conjunction with specific embodiments.

[0061] The method for task computation and adaptive parameter adaptation based on the domestic IT innovation big data platform of the present invention includes the following steps:

[0062] (1) Prepare the information technology innovation big data platform, indicator calculation service, adaptive parameter adjuster, data collector and information technology innovation database;

[0063] (2) Install the information technology data acquisition device, configure the database address and data synchronization frequency, and configure the national cryptographic algorithm SM2 to encrypt and transmit data;

[0064] (3) Obtain initialization parameters and calculate indicator data;

[0065] (4) Receive the submitted task and select different decision models according to the type of indicator task. If it is a new indicator, continue to step (5); if it is an existing indicator, continue to step (6).

[0066] (5) Generate the parameters required to calculate the new indicators based on the GBT regression model;

[0067] (6) Obtain the parameter values ​​for calculating stock indicators from the knowledge base model, and generate the parameters required for calculating stock indicators based on the knowledge base model;

[0068] (7) Receive the parameters and runtime packages required for the computation tasks generated by the GBT regression model and knowledge base model, and submit them to the information technology innovation big data platform;

[0069] (8) Perform resource scheduling and performance monitoring.

[0070] In a preferred embodiment of the present invention, step (2) specifically includes the following steps:

[0071] (2.1) Install the domestic IT data acquisition tool and download the corresponding database driver, then import it into the installation directory;

[0072] (2.2) Select the external data source and the database type of the domestic IT innovation database, configure the database address and data synchronization frequency, and verify whether the connection is normal. If it is normal, continue to step (2.3); otherwise, continue to step (2.2).

[0073] (2.3) Load the data acquisition script, configure the national cryptographic algorithm SM2 to encrypt and transmit data, and import external historical market data into the information technology innovation database.

[0074] In a preferred embodiment of the present invention, step (3) specifically includes the following steps:

[0075] (3.1) Load historical market data from the domestic IT innovation database;

[0076] (3.2) Obtain the initialization parameters required for submitting the task from the parameter knowledge base;

[0077] (3.3) Package the indicator calculation service into a jar file and submit the jar file to the task decision service.

[0078] In a preferred embodiment of the present invention, step (5) specifically includes the following steps:

[0079] (5.1) Construct a feature model;

[0080] (5.2) Optimize the model training;

[0081] (5.3) Evaluate the computation time eva_time of the prediction data based on the regression model, prediction dataset and computation task dynamic parameters, read the information innovation database to obtain the task computation time compute_time of the information innovation big data platform, use the regression task evaluator to calculate the root mean square error, and evaluate the model performance.

[0082] (5.4) Update the knowledge base model based on the input and output features trained by the model;

[0083] (5.5) Adjust the dynamic parameter feature values ​​dynamically by adding or removing them according to the server configuration and model performance.

[0084] In a preferred embodiment of the present invention, step (5.1) specifically includes the following steps:

[0085] (5.1.1) Construct time window features and calculate the average time spent on past indicators;

[0086] (5.1.2) Input system resource characteristics, data scale characteristics, and dynamic parameter characteristics of computing tasks, and output time series characteristics;

[0087] (5.1.3) Combine system resources, data scale, dynamic parameter characteristics of computing tasks, and time series characteristics into a single vector column;

[0088] (5.1.4) Standardize the process.

[0089] In a preferred embodiment of the present invention, step (5.2) specifically includes the following steps:

[0090] (5.2.1) Divide the dataset into training set data and prediction set data according to the time series;

[0091] (5.2.2) Based on the training set data, the GBT regression model is used to train and obtain the regression model.

[0092] In a preferred embodiment of the present invention, step (6) specifically includes the following steps:

[0093] (6.1) Store the data in the parameter knowledge base and calculate the task parameter values;

[0094] (6.2) Set the confidence threshold, determine the confidence level, and trigger adaptive parameter adjustment.

[0095] In a preferred embodiment of the present invention, step (6.1) specifically includes the following steps:

[0096] (6.1.1) The input and output feature values ​​generated by the GBT regression model and the running results of performance monitoring are filtered by the rule engine and stored in the parameter knowledge base;

[0097] (6.1.2) Provide the parameter values ​​required for the stock index calculation task, input the system resource characteristics and data scale characteristics, and obtain the closest calculation task parameter values ​​based on the similarity variance.

[0098] In a preferred embodiment of the present invention, step (6.2) specifically includes the following steps:

[0099] (6.2.1) Store the performance data from training, tuning, and performance monitoring into the parameter knowledge base through the rule engine;

[0100] (6.2.2) Set confidence thresholds based on system resource characteristics and data scale characteristics;

[0101] (6.2.3) Determine the confidence level based on the performance monitoring index data. If the confidence level is low, trigger adaptive parameter adjustment; if the confidence level is medium, trigger model retraining; if the confidence level is high, store the feature values ​​and performance data in the parameter knowledge base.

[0102] In a preferred embodiment of the present invention, step (8) specifically includes the following steps:

[0103] (8.1) Access and read data through the standard interface;

[0104] (8.2) Receive the tasks and parameters submitted by the adaptive parameter regulator, complete resource scheduling and task allocation, and calculate the index tasks;

[0105] (8.3) Store the calculated performance data and results data in a knowledge base;

[0106] (8.4) Monitor the server's operating performance in real time through the performance monitoring service. If the threshold is reached, an alarm will be issued and the performance data will be stored in the knowledge base.

[0107] The present invention relates to an apparatus for task computation and adaptive parameter adaptation based on a domestically developed big data platform, wherein the apparatus comprises:

[0108] A processor is configured to execute computer-executable instructions;

[0109] The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the various steps of the above-described method for task computation and adaptive parameter adaptation based on the domestic big data platform.

[0110] The present invention discloses a processor for task computation and adaptive parameter adaptation based on a domestically developed big data platform. The processor is configured to execute computer-executable instructions. When the computer-executable instructions are executed by the processor, the various steps of the above-described method for task computation and adaptive parameter adaptation based on a domestically developed big data platform are implemented.

[0111] The computer-readable storage medium of the present invention stores a computer program thereon, which can be executed by a processor to implement the various steps of the above-described method for task calculation and adaptive parameter adaptation based on the information technology innovation big data platform.

[0112] To address the stability issues of single-strategy models, the inability to dynamically adjust task parameters submitted by the system, and the compatibility problems with domestically developed big data platforms, this paper proposes a task computation and adaptive parameter adaptation method based on a domestically developed big data platform. Based on this platform, indicator computation is more secure; a hybrid decision model is added to improve the efficiency, robustness, and security of indicator computation services; and the computational performance results are stored in a knowledge base model for continuous learning and self-evolution, constantly improving coverage and accuracy through closed-loop feedback.

[0113] In a specific embodiment of the present invention, the specific steps are as follows:

[0114] 1. Prepare the domestic IT innovation big data platform, indicator calculation service, adaptive parameter adjuster, data collector, and domestic IT innovation database.

[0115] 2. Information technology application data acquisition device:

[0116] 2.1 Install the domestic database data collector loader on the Kylin domestic database innovation server, and download the corresponding database driver and import it into the loader's installation directory;

[0117] 2.2 Select the external data source and the database type of the domestic IT innovation database, configure the database address and data synchronization frequency, and verify whether the connection is normal;

[0118] 2.3 Load the data acquisition script, configure the national cryptographic algorithm SM2 to encrypt and transmit data, confirm that the script is running normally, and import the external historical market data into the domestic IT database.

[0119] 3. Indicator Calculation Service:

[0120] 3.1. Read and load historical market data from the domestic IT innovation database for indicator processing;

[0121] 3.2. The initialization parameters required for submitting Spark tasks are retrieved from the parameter knowledge base. The knowledge base stores input values: the number of computation tasks, the total amount of data required to calculate the metrics, server performance data, and metric computation time; and output values: the initialization parameter values. The data in the knowledge base originates from model training and data from production big data platform computations.

[0122] 3.3. Package the indicator calculation service into a JAR file using the Maven-assembly mode, and confirm that the JAR file can be successfully submitted to the task decision service.

[0123] The Maven Assembly plugin is a powerful project packaging tool in Maven. The maven-assembly mode refers to a specific packaging method within the plugin that allows you to create a complete distribution package containing all project dependencies and define complex directory structures. In this invention, it is primarily used to package the project source code and all dependencies into a single server-executable and runnable JAR file.

[0124] 4. Hybrid Decision Model - Adaptive Parameter Tuning:

[0125] 4.1 Task Decision-Making:

[0126] 4.1.1 Receive submitted tasks and select different decision models based on the task type. For new indicator tasks, generate the parameters required for calculating the indicator based on the GBT regression model; for the calculation of existing indicators, generate the parameters required for calculating the indicator based on the knowledge base model.

[0127] 4.2 GBT Regression Model:

[0128] For newly added indicators and tasks.

[0129] 4.2.1 Constructing a feature model:

[0130] 4.2.1.1 Constructing time window features: Calculate the average time elapsed for the past n metrics (avg_time_last_n);

[0131] 4.2.1.2, Feature Vectorization:

[0132] Input-System resource feature 1: CPU usage (cpu_usage) and memory usage (memory_usage) during current model training or tuning;

[0133] Input - Data Scale Feature 2: Market data volume (num_records), number of indicators (num_columns);

[0134] Input-computation task dynamic parameter feature 3: the size of the memory used by the driver (dm), the size of the memory used by each executor (em), the number of cores allocated to each executor (ec), and the number of executors (ne).

[0135] Output - Time Series Feature 4: Mean (avg_time_last_n), maximum (max_time_last_n), and standard deviation (std_time_last_n) of the indicator's time consumption over the past n calculations;

[0136] Feature vector merging: The transform method of Spark VectorAssembler is used to merge the above system resources, data scale, computing task dynamic parameter features and time series features into a single vector column;

[0137] Standardization: The fit method of Spark's feature tool StandardScaler is called to standardize the vector columns and eliminate differences in units.

[0138] 4.2.2 Model Training and Optimization:

[0139] 4.2.2.1. Divide the dataset: Divide the dataset in 4.2.1.2 into training set data and prediction set data according to the time series. The first 70% of the time data is the training set data and the last 30% of the time data is the prediction set data.

[0140] 4.2.2.2 Training the GBT regression model: Based on the training set data, the GBTRegressionModel is used to train the regression model;

[0141] 4.2.3. Based on the regression model, prediction dataset, and dynamic parameters of the computation task (4.2.1.2), evaluate the computation time eva_time of the "prediction data", read the domestic innovation database to obtain the task computation time compute_time of the domestic innovation big data platform, and use the regression task evaluator provided by Spark MLlib to calculate the root mean square error (rmse) based on eva_time and compute_time to evaluate the model performance.

[0142] 4.2.4 Update the knowledge base model with the input and output features from the model training;

[0143] 4.2.5 Dynamic Parameter Tuning: Based on server configuration and model performance, dynamically increase or decrease the task dynamic parameter characteristic values ​​calculated in 4.2.1.2. Overall increase / decrease strategy: Total server memory (te), total CPU cores (tc); Calculate task dynamic parameter characteristics: Memory usage by the driver (dm), memory usage by each executor (em), number of cores allocated to each executor (ec), and number of executors (ne). Ensure that server CPU, memory, and I / O utilization do not exceed the threshold of 60%, and that the task dynamic parameter characteristic value settings meet the following conditions:

[0144] n e *e m <tm e c *n e <t c ,d m <t c / 2

[0145] Increase the dynamic parameter feature values ​​until the model performance is optimal. If the server performance exceeds the threshold or the dynamic parameter feature value settings do not meet the above formula conditions, decrease them until the optimal value is reached.

[0146] 4.3 Knowledge Base Model

[0147] For the calculation of stock indicators, the parameter values ​​of the calculation task are obtained from the knowledge base model.

[0148] 4.3.1 Parameter Knowledge Base:

[0149] 4.3.1.1 The input and output feature values ​​generated by the GBT regression model, as well as the running results of the performance monitoring big data platform for information technology innovation, are all filtered by the rule engine and fall into the parameter knowledge base;

[0150] 4.3.1.2 Provide the parameter values ​​required for the stock index calculation task. Input the feature values ​​1 and 2 in 4.2.1.2 for the task decision and obtain the closest calculation task parameter values ​​based on the similarity variance.

[0151] 4.3.2 Rule Engine:

[0152] Performance data from training, tuning, and performance monitoring is stored in the parameter knowledge base via a rule engine. The rule engine sets confidence judgment rules and annotation processes to enhance the knowledge base's active learning mechanism. Based on performance monitoring data, adaptive parameter adjustment rules are triggered when thresholds are reached.

[0153] Confidence determination rule: Set the confidence threshold based on feature values ​​1 and 4.

[0154] (1) If CPU or memory usage exceeds 70% or computation time is >1 hour, manual labeling of low confidence will be triggered, and the model will not be stored in the knowledge base. This will trigger model retraining and adaptive parameter adjustment.

[0155] (2) If CPU and memory usage are less than 70% and computation time is greater than 1 hour, the model is judged to be of medium confidence and will be retrained. The parameters required for the computation task will not be regenerated for the time being, and the feature values ​​and performance data will not be stored in the knowledge base.

[0156] (3) If CPU and memory usage are less than 70% and computation time is less than 1 hour, it is judged as high confidence, and the feature value and performance data are stored in the knowledge base.

[0157] Adaptive parameter adjustment: The rule engine receives performance monitoring metrics and calculates performance data. When it determines that the confidence level is low, it automatically triggers adaptive parameter adjustment. Specifically, it triggers the GBT regression model to be retrained, updates the knowledge base, generates the latest calculation task parameters, and submits them to the big data platform for recalculation.

[0158] 4.4 Task Reception

[0159] 4.4.1 The task receiving service receives the parameters and runtime packages required for the computation tasks generated by the GBT regression model and knowledge base model, assembles them, and submits them to the domestic big data platform via spark-submit;

[0160] 4.5. Domestic IT Innovation Big Data Platform and Performance Monitoring

[0161] 4.5.1 Unlike non-IT-innovation open-source big data Spark platforms, IT-innovation big data platforms adopt a storage-compute separation architecture, which separates and deploys the storage layer and the computing layer. Data is deployed separately and does not need to be localized to the computing nodes. The computing nodes access and read data through the standard interface (HDFS API).

[0162] 4.5.2 Receive tasks and parameters submitted by the adaptive parameter regulator, complete resource scheduling and task allocation through YARN, and execute metric task calculations through Spark;

[0163] 4.5.3 Store the calculated performance data and results data in a knowledge base;

[0164] 4.5.4 The performance monitoring service monitors the server's operating performance in real time, issues an alarm when the threshold is reached, and stores the performance data in the knowledge base.

[0165] This invention applies a hybrid decision-making model, including a GBT regression model and a knowledge base model, to solve the problem of adaptive parameter adjustment in the calculation of new and existing indicators. In addition, this invention employs performance monitoring, a rule engine, and confidence level judgment to improve the accuracy of the hybrid decision-making model. Specific implementation details of this embodiment can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0166] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0167] It should be noted that in the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means at least two.

[0168] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0169] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0170] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The corresponding program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiments.

[0171] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0172] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0173] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0174] The present invention employs a method, apparatus, processor, and computer-readable storage medium for task computation and adaptive parameter adaptation based on a domestically developed big data platform. Based on the domestically developed big data platform, the platform is more secure. Based on a hybrid decision model, it improves the efficiency, robustness, and security of indicator computation services. The knowledge base model learning is sustainable, continuously optimizing model accuracy, and continuously improving coverage and accuracy through closed-loop feedback.

[0175] In this specification, the invention has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and variations can be made without departing from the spirit and scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

Claims

1. A method for task computation and adaptive parameter adaptation based on a domestically developed big data platform, characterized in that, The method includes the following steps: (1) Prepare the information technology innovation big data platform, indicator calculation service, adaptive parameter adjuster, data collector and information technology innovation database; (2) Install the information technology data acquisition device, configure the database address and data synchronization frequency, and configure the national cryptographic algorithm SM2 to encrypt and transmit data; (3) Obtain initialization parameters and calculate indicator data; (4) Receive the submitted task and select different decision models according to the type of indicator task. If it is a new indicator, continue to step (5); if it is an existing indicator, continue to step (6). (5) Generate the parameters required for calculating the new indicators based on the GBT regression model; (6) Obtain the parameter values ​​for calculating stock indicators from the knowledge base model, and generate the parameters required for calculating stock indicators based on the knowledge base model; (7) Receive the parameters and runtime packages required for the computation tasks generated by the GBT regression model and knowledge base model, and submit them to the information technology innovation big data platform; (8) Perform resource scheduling and performance monitoring; Step (5) specifically includes the following steps: (5.1) Construct a feature model; (5.2) Optimize the model training; (5.3) Evaluate the computation time eva_time of the prediction data based on the regression model, prediction dataset and computation task dynamic parameters, read the information innovation database to obtain the task computation time compute_time of the information innovation big data platform, use the regression task evaluator to calculate the root mean square error, and evaluate the model performance. (5.4) Update the knowledge base model based on the input and output features trained by the model; (5.5) Optimize the dynamic parameter feature values ​​by dynamically increasing or decreasing them according to the server configuration and model performance; Step (5.1) specifically includes the following steps: (5.1.1) Construct time window features and calculate the average time spent on past indicators; (5.1.2) Input system resource characteristics, data scale characteristics, and dynamic parameter characteristics of computing tasks, and output time series characteristics; (5.1.3) Combine system resources, data scale, dynamic parameter characteristics of computing tasks, and time series characteristics into a single vector column; (5.1.4) Standardize the process; Step (6) specifically includes the following steps: (6.1) Store the data in the parameter knowledge base and calculate the task parameter values; (6.2) Set the confidence threshold, determine the confidence level, and trigger adaptive parameter adjustment; The aforementioned step (6.1) specifically includes the following steps: (6.1.1) The input and output feature values ​​generated by the GBT regression model and the running results of performance monitoring are filtered by the rule engine and stored in the parameter knowledge base; (6.1.2) Provide the parameter values ​​required for the stock index calculation task, input the system resource characteristics and data scale characteristics, and obtain the closest calculation task parameter values ​​based on the similarity variance; Step (6.2) specifically includes the following steps: (6.2.1) Store the performance data from training, tuning, and performance monitoring into the parameter knowledge base through the rule engine; (6.2.2) Set confidence thresholds based on system resource characteristics and data scale characteristics; (6.2.3) Determine the confidence level based on the performance monitoring index data. If the confidence level is low, trigger adaptive parameter adjustment; if the confidence level is medium, trigger model retraining; if the confidence level is high, store the feature values ​​and performance data in the parameter knowledge base. The rule engine refers to the storage of performance data from training, tuning, and performance monitoring into a parameter knowledge base; the rule engine sets confidence judgment rules and annotation processes to enhance the active learning mechanism of the knowledge base; and based on performance monitoring data, adaptive parameter adjustment rules are triggered when a threshold is reached. Specifically, (6.2.3) refers to: If CPU and memory usage exceed 70% or computation time exceeds 1 hour, manual labeling of the model as low confidence will be triggered, and the model will not be stored in the knowledge base. This will trigger model retraining and adaptive parameter adjustment. If CPU and memory usage are less than 70% or computation time is greater than 1 hour, the system is considered to have medium confidence, triggering model retraining. The parameters required for the computation task will not be regenerated, and feature values ​​and performance data will not be stored in the knowledge base. If CPU and memory usage are less than 70% or computation time is less than 1 hour, the system is considered to have high confidence, and the feature values ​​and performance data are stored in the knowledge base.

2. The method for task computation and adaptive parameter adaptation based on a domestically developed big data platform according to claim 1, characterized in that, Step (2) specifically includes the following steps: (2.1) Install the domestic IT data acquisition tool and download the corresponding database driver, then import it into the installation directory; (2.2) Select the external data source and the database type of the information technology innovation database, configure the database address and data synchronization frequency, and verify whether the connection is normal. If it is normal, continue to step (2.3); otherwise, continue to step (2.2). (2.3) Load the data acquisition script, configure the national cryptographic algorithm SM2 to encrypt and transmit data, and import external historical market data into the information technology innovation database.

3. The method for task computation and adaptive parameter adaptation based on a domestically developed big data platform according to claim 1, characterized in that, Step (3) specifically includes the following steps: (3.1) Load historical market data from the domestic IT innovation database; (3.2) Obtain the initialization parameters required for submitting the task from the parameter knowledge base; (3.3) Package the indicator calculation service into a jar file and submit the jar file to the task decision service.

4. The method for task computation and adaptive parameter adaptation based on a domestically developed big data platform according to claim 1, characterized in that, Step (5.2) specifically includes the following steps: (5.2.1) Divide the dataset into training set data and prediction set data according to the time series; (5.2.2) Based on the training set data, the GBT regression model is used to train and obtain the regression model.

5. The method for task computation and adaptive parameter adaptation based on a domestically developed big data platform according to claim 1, characterized in that, Step (8) specifically includes the following steps: (8.1) Access and read data through the standard interface; (8.2) Receive the tasks and parameters submitted by the adaptive parameter regulator, complete resource scheduling and task allocation, and calculate the index tasks; (8.3) Store the calculated performance data and results data in a knowledge base; (8.4) Monitor the server's operating performance in real time through the performance monitoring service. If the threshold is reached, an alarm will be issued and the performance data will be stored in the knowledge base.

6. A device for task computation and adaptive parameter adaptation based on a domestically developed big data platform, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for task computation and adaptive parameter adaptation based on the domestic big data platform as described in any one of claims 1 to 5.

7. A processor for task computation and adaptive parameter adaptation based on a domestically developed big data platform, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for task computation and adaptive parameter adaptation based on the domestic big data platform as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method for task calculation and adaptive parameter adaptation based on the information technology innovation big data platform as described in any one of claims 1 to 5.

Images