Memory operation driving optimization method

Abstract

The invention discloses an optimization method for memory operation driving. According to the time consumption evaluation model adopted by the invention, a scientific and reasonable performance prediction tool can be provided for a system designer by comprehensively considering the relationship between the total amount of calculation data and the execution time consumption of a single service and combining various influence factors (such as hardware configuration and network conditions) in an actual operation environment. In addition, effective adjustment and improvement of an existing system can be achieved by applying the evaluation result, and therefore the purpose of improving the overall operation efficiency is achieved.

Description

Technical Field

[0001] The present invention relates to an optimization strategy driven by memory operation using a time-consuming evaluation model. Background Art

[0002] With the rapid development of information technology, data processing and computing tasks have become increasingly complex and large-scale. In high-load environments such as financial transactions, big data analysis, and cloud computing, the performance optimization of systems has become a crucial issue. Especially in memory operation-intensive applications, how to accurately evaluate and optimize the time-consuming of business execution has become the key to improving the overall system efficiency.

[0003] Traditional time-consuming evaluation methods often rely on empirical rules or simple statistical models, which are unable to cope when faced with dynamically changing workloads and diverse application scenarios. For example, in a distributed computing environment, factors such as communication latency and resource competition between different nodes will have a significant impact on the final task completion time. Summary of the Invention

[0004] The object of the present invention is to provide a more accurate and flexible method for evaluating and optimizing the time-consuming of business execution driven by memory operation.

[0005] To achieve the above object, the technical solution of the present invention discloses an optimization method driven by memory operation, characterized by including the following steps:

[0006] Step 1, set a control group, including the following steps:

[0007] Step 101, set a variety of different data scales according to actual business requirements;

[0008] Step 102, simulate common data operations under each set data scale;

[0009] Step 103, record the specific execution time of each data operation, thereby forming the baseline data of the control group;

[0010] Step 2, set an observation group, including the following steps:

[0011] Step 201, code parsing and static analysis:

[0012] Use tools to parse the source code of the target application program and identify the hot functions or modules on the critical path;

[0013] Step 202, dynamic monitoring and data analysis:

[0014] Deploy monitoring components in the production environment to collect various running metrics and exception log information of the identified hot functions or modules in real time;

[0015] Step 203, Record various data:

[0016] Summarize all relevant information generated in the above process into a structured report document to obtain the actual performance data of the observation group;

[0017] Step 204, Identify time-consuming operations:

[0018] Compare and analyze the actual performance data of the observation group with the baseline data of the control group under the same data scale, find out the existing gaps and their reasons, and automatically detect the bottleneck links or inefficient parts of the target application program;

[0019] Step 3, Optimization strategies, including:

[0020] Optimization based on the time-consuming evaluation model:

[0021] According to the total time-consuming Ttotal calculated by the time-consuming evaluation model, adjust the execution order of tasks and resource allocation to reduce the total execution time;

[0022] Dynamic adjustment and feedback optimization

[0023] Utilize the dynamic adjustment mechanism to automatically optimize task scheduling according to real-time performance data, reduce the impact of bottleneck links, and improve the overall efficiency of the system.

[0024] Preferably, in Step 3, the total time-consuming Ttotal is calculated by the following formula:

[0025] Ttotal = ∑(Di × ∑tj)

[0026] In the formula, Di is the data volume of the i-th task, and tj is the execution time of the j-th subtask.

[0027] The time-consuming evaluation model adopted by the present invention can provide a scientific and reasonable performance prediction tool for system designers by comprehensively considering the relationship between the total amount of calculated data and the time-consuming of single business execution, and combining various influencing factors (such as hardware configuration, network conditions, etc.) in the actual operating environment. In addition, the present invention can realize the effective adjustment and improvement of the existing system through the application of the evaluation results, so as to achieve the purpose of improving the overall operating efficiency. Brief Description of the Drawings

[0028] Figure 1 It is a flowchart of the present invention. Detailed Embodiment

[0029] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

[0030] As Figure 1 shown, an optimization method driven by in-memory computing according to an embodiment of the present invention includes the following steps:

[0031] Step 1, set up a control group

[0032] The purpose of setting up a control group is to establish a baseline for measuring performance differences in actual applications, and further includes the following steps:

[0033] Step 101, set multiple different data scales according to actual business requirements, and ensure that these data scales can cover most application scenarios. For example, data scales at small, medium, and large levels can be set.

[0034] Step 102, simulate common data operations under each set data scale. The common data operations to be simulated include data reading / writing, algorithm calculations (such as sorting, searching), and memory allocation and release.

[0035] Step 103, record the specific execution time of each data operation to form the baseline data of the control group.

[0036] Step 2, set up an observation group

[0037] The purpose of setting up an observation group is to collect operation data in a real environment and conduct in-depth analysis based on this to discover potential problem points and optimization directions, specifically including the following steps:

[0038] Step 201, code parsing and static analysis:

[0039] Use tools to parse the source code of the target application program and identify hot functions or modules on the critical path. These hot functions or modules usually contain multiple tasks and subtasks.

[0040] Step 202, dynamic monitoring and data analysis:

[0041] Deploy monitoring components in the production environment to collect various operation metrics (such as response time, throughput) and exception log information of the identified hot functions or modules in real time.

[0042] Step 203, record each item of data:

[0043] Summarize all relevant information generated in the above process into a structured report document to obtain the actual performance data of the observation group.

[0044] Step 204, Time-consuming operation identification:

[0045] Compare and analyze the actual performance data of the observation group with the results of the control group under the same data scale, find out the existing gaps and their reasons, and automatically detect the bottleneck links or inefficient parts of the target application program.

[0046] Step 3, Optimization strategies, including

[0047] a) Optimization based on the time-consuming evaluation model

[0048] When there are significant differences between the total time-consuming or the time-consuming of subtasks in the actual application and the results calculated by the evaluation model, optimize the performance by means of data preprocessing, controlling the data volume, and using efficient data structures. The specific optimization measures include the following:

[0049] Data preprocessing

[0050] 1. Data cleaning: Remove or correct incorrect and inconsistent data.

[0051] 2. Data standardization: Convert the data into a unified format for easy processing and analysis.

[0052] 3. Feature engineering: Select or create features that contribute to the model performance and reduce the calculation of invalid features.

[0053] 4. Data dimensionality reduction: Use techniques such as PCA to reduce the data dimension and lower the calculation complexity.

[0054] 5. Data grouping: Group the data to reduce the data volume and improve the processing efficiency.

[0055] Controlling the data volume

[0056] 1. Data sampling: Sample a large amount of data to reduce the data volume to be processed.

[0057] 2. Data batching: Divide the data into small batches for processing to avoid performance degradation caused by processing a large amount of data at one time.

[0058] 3. Early filtering: According to business requirements, filter out unnecessary data in advance to reduce the burden of subsequent processing steps.

[0059] 4. Data compression: Use data compression technology to reduce the storage space and transmission time.

[0060] Efficient data structures

[0061] 1. Use dictionaries and hash tables: For data that needs to be quickly looked up, using a dictionary or hash table can significantly improve efficiency.

[0062] 2. Use tree structures: For data that needs to be sorted or range queried, tree structures (such as binary search trees, B-trees, etc.) can be used to improve efficiency.

[0063] 3. Use caching: Utilize caching to store frequently accessed data and reduce repeated calculations.

[0064] 4. Use queues and stacks: For data processing tasks with first-in-first-out or last-in-first-out characteristics,

[0065] using a queue or stack can improve efficiency.

[0066] Other optimization measures

[0067] 1. Parallel processing: Utilize multi-threading or multi-processing to process data in parallel and improve processing speed.

[0068] 2. Use more efficient algorithms: Optimize algorithms to reduce computational complexity and improve processing efficiency.

[0069] 3. Utilize hardware resources: Increase computing resources such as CPU, GPU, or memory to improve processing speed.

[0070] b) Dynamic adjustment and feedback optimization

[0071] Utilize a dynamic adjustment mechanism to automatically optimize task scheduling based on real-time performance data, reduce the impact of bottleneck links, and improve the overall system efficiency. The specific optimization steps are as follows:

[0072] Real-time monitoring: Continuously monitor the running state of the system, including key performance indicators such as CPU usage, memory occupancy, and network latency.

[0073] Performance analysis: Based on the monitoring data, analyze the efficiency of task execution and identify the key subtasks that cause performance bottlenecks.

[0074] Scheduling strategy adjustment: According to the performance analysis results, adjust the task scheduling strategy. For example, prioritize the execution of time-consuming subtasks or increase the number of parallel tasks when the CPU load is low.

[0075] Dynamic resource allocation: Dynamically allocate computing resources according to the real-time requirements of tasks, such as adjusting thread priorities or allocating more processor cores.

[0076] Feedback loop: Feed the optimization results back into the system, continuously monitor performance changes, and form an optimization feedback loop to ensure that the system can continuously self-adjust according to the actual situation.

Claims

1. A memory operation driven optimization method, characterized in that: The following steps are involved: Step 1: Set up a control group, including the following steps: Step 101: Set a variety of different data scales according to actual business needs; Step 102: Simulate common data operations at each set data scale; Step 103: Record the specific execution time of each data operation to form baseline data for the control group; Step 2: Set up the observation group, including the following steps: Step 201: Code parsing and static analysis: Use tools to parse the source code of the target application and identify hot functions or modules on the critical path; Step 202: Dynamic monitoring and data analysis: Deploy monitoring components in the production environment to collect various operating indicators and abnormal log information of identified hot functions or modules in real time; Step 203: Record various data: Summarize all relevant information generated in the above process into a structured report document to obtain the actual performance data of the observation group; Step 204: time-consuming operation identification: Compare and analyze the actual performance data of the observation group with the baseline data of the control group at the same data scale to find out the gaps and their causes, and automatically detect the bottlenecks or inefficient parts of the target application; Step 3: Optimization strategy, including: Optimization based on time-consuming evaluation model: When the total time or subtask time in the actual application is significantly different from the results calculated by the evaluation model, optimize the performance by preprocessing the data, controlling the data volume, and using efficient data structures; Dynamic adjustment and feedback optimization Utilize the dynamic adjustment mechanism to automatically optimize task scheduling based on real-time performance data, reduce the impact of bottlenecks, and improve the overall efficiency of the system.

2. The memory operation driven optimization method according to claim 1, characterized in that: In step 3, the total time Ttotal is calculated by the following formula: Ttotal=∑(Di×∑tj) Where Di is the data volume of the i-th task, and tj is the execution time of the j-th subtask.

Images