Performance optimization method, device and equipment for big data plug-in and storage medium
By automatically collecting and statistically analyzing operation logs during the operation of the Hadoop big data plugin, the problems of low efficiency and poor real-time performance in existing technologies are solved, achieving efficient and accurate performance optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JINAN INSPUR DATA TECH CO LTD
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-29
AI Technical Summary
In existing technologies, monitoring and statistically analyzing the performance metrics of Hadoop big data plugins relies on manual collection and analysis, which is inefficient and prone to data omissions and errors, and makes it impossible to adjust performance in real time.
By automatically collecting and statistically analyzing operation logs during the operation of the big data plugin, and utilizing log aggregators and event tracking technology, the automatic collection and statistical analysis of operation logs are achieved, and performance indicators are output in real time.
It improved the efficiency of log collection and statistics, avoided data omissions and errors, enabled real-time performance tuning, and reduced human resource consumption.
Smart Images
Figure CN122120124A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of electronic computer technology, and in particular to methods, apparatus, devices and storage media for performance optimization of big data plugins. Background Technology
[0002] With the rapid development of big data technology, Hadoop has become an important big data processing platform. However, as the amount of data and processing tasks increase, its performance bottlenecks become increasingly apparent. To optimize Hadoop performance, it is necessary to monitor and statistically analyze the performance metrics of big data plugins within Hadoop, and based on these metrics, optimize the performance of big data plugins, thereby improving Hadoop's overall performance. For example, if statistics show that a big data plugin takes a long time to execute List operations, the plugin's directory traversal logic can be optimized (e.g., by introducing a caching mechanism or reducing the number of repeated traversals), thus reducing the time spent on List operations and ultimately optimizing the big data plugin's performance.
[0003] Currently, some technologies rely on manual collection and analysis of operation logs from big data plugins in Hadoop to monitor and analyze their performance metrics. This method is labor-intensive and time-consuming, resulting in low efficiency. Therefore, there is an urgent need for a method to automate the collection and analysis of operation logs, thereby improving the efficiency of log data collection and analysis. Summary of the Invention
[0004] This application provides a performance optimization method, a performance optimization device, an electronic device, a computer-readable storage medium, and a computer program product for big data plugins, to at least solve the problem in the related art of not being able to automatically collect and statistically analyze the operation logs of big data plugins.
[0005] This application provides a performance optimization method for big data plugins, including:
[0006] Receive a data operation request sent by the target application, the data operation request including the target data operation to be performed; The data manipulation object is instantiated to obtain an instance object; The instance object is initialized, which includes creating a log aggregator for the instance object and reading log parameters, wherein the log parameters refer to pre-configured constraints related to the printing of operation logs for the target data operation; The instance object is invoked to perform the target data operation. During the execution of the target data operation, the instance object collects operation logs. The log aggregator is used to obtain and print the operation logs collected by the instance object based on the log parameters. The operation logs printed by the log aggregator are used to optimize the performance of the big data plugin.
[0007] This application also provides a performance optimization device for big data plugins, including: A request receiving module is used to receive a data operation request sent by a target application, wherein the data operation request includes the target data operation to be performed. The instantiation module is used to instantiate the data operation object to obtain an instance object; An initialization module is used to initialize the instance object. The initialization includes creating a log aggregator for the instance object and reading log parameters. The log parameters refer to pre-configured constraints related to the printing of operation logs for the target data operation. The data operation module is used to call the instance object to perform the target data operation. During the execution of the target data operation, the instance object collects operation logs. The log aggregator is used to obtain and print the operation logs collected by the instance object based on the log parameters. The operation logs printed by the log aggregator are used to optimize the performance of the big data plugin.
[0008] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described performance optimization methods for big data plugins when executing the computer program.
[0009] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described performance optimization methods for big data plugins.
[0010] In some embodiments of this application, upon receiving a data operation request from the target application, a data operation object of the big data plugin is instantiated and initialized. Since initialization includes creating a log aggregator for the instance object and reading log parameters, and the instance object can collect operation logs during the execution of the target data operation, when the instance object is invoked to perform a data operation, the log aggregator can statistically analyze and print the operation logs output by the instance object based on the log parameters. This application, based on the instance object, enables automatic collection of data operation logs, and based on the log aggregator and log parameters, enables automatic statistical analysis and printing of data operation logs, thus solving the problem in related technologies where automatic collection and statistical analysis of operation logs for big data plugins is impossible. Attached Figure Description
[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart illustrating a big data plugin performance optimization method according to some embodiments of this application; Figure 2 Schematic diagram of a performance optimization apparatus provided for some embodiments of this application; Figure 3 A schematic diagram of the modules of an electronic device provided for some embodiments of this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0016] Hadoop provides a unified FileSystem standard interface. Based on whether the protocol logic is compatible with the FileSystem standard interface, Hadoop's underlying storage can be divided into native underlying storage and scalable external underlying storage. Native underlying storage refers to storage compatible with the FileSystem standard interface protocol logic, such as HDFS (Hadoop Distributed File System). Scalable external underlying storage refers to storage incompatible with the FileSystem standard interface protocol logic, such as S3 storage. Specifically, S3 storage refers to storage compatible with the S3 protocol.
[0017] For native underlying storage, external components (such as Spark and Hive) can directly call the FileSystem standard interface to perform data operations on the native underlying storage. For scalable external underlying storage (such as S3 storage), because the protocol logic of these storage systems is incompatible with the protocol logic of the FileSystem standard interface, external components, after calling the FileSystem standard interface, can use Hadoop's big data plugins to perform data operations on these storage systems. Specifically, big data plugins are used to translate FileSystem instructions according to the protocols supported by Hadoop's external underlying storage, and then perform data operations on the external underlying storage based on the translated instructions. Hadoop can include one or more data plugins. Different external underlying storage systems correspond to different data plugins. When external components need to perform data operations on Hadoop's external underlying storage, they can call Hadoop's FileSystem standard interface. Hadoop, based on the external underlying storage to be operated on, finds the corresponding data plugin and then performs the data operation through the appropriate data plugin.
[0018] Currently, with the increase in data volume and processing tasks, the performance bottleneck of Hadoop is becoming increasingly apparent. To optimize Hadoop performance, it is necessary to monitor and statistically analyze the performance metrics of the big data plugin in Hadoop, and based on these metrics, optimize the performance of the big data plugin, thereby optimizing Hadoop performance. For example, if statistics show that the big data plugin takes a long time to execute List operations, the directory traversal logic of the big data plugin can be optimized (e.g., by introducing a caching mechanism or reducing the number of repeated traversals), thereby reducing the time spent on List operations. This optimizes the performance of the big data plugin, and consequently, the performance of Hadoop.
[0019] In some technologies, monitoring and analyzing the performance metrics of big data plugins in Hadoop relies on manually collecting and analyzing the plugin's operation logs. For example, the time taken by the big data plugin in multiple List operations (i.e., operation logs) is manually collected, and then the average time of these multiple List operations is calculated to obtain the average List operation time (i.e., performance metric). This method has the following problems: 1) It requires a lot of manpower and time, and is inefficient.
[0020] 2) During the statistical process, data omissions and errors are prone to occur, resulting in inaccurate statistical results.
[0021] 3) Analysis can only be performed after the plugin operation is completed and the log file is generated. It is impossible to collect log data and calculate performance indicators synchronously during the operation of the data plugin, resulting in a lag in the collected log data and performance indicators, and the performance of the data plugin cannot be adjusted in a timely manner.
[0022] In view of this, this application provides a performance optimization method for big data plugins, which can automatically collect and statistically analyze operation logs during the operation of big data plugins, thereby solving the aforementioned problems. This big data plugin performance optimization method can be applied to electronic devices. Electronic devices can include, but are not limited to, tablets, laptops, desktop computers, servers, etc. (See also...) Figure 1 This is a flowchart illustrating a big data plugin performance optimization method provided in some embodiments of this application. Figure 1 In China, the performance optimization methods for big data plugins include the following steps: Step S101: Receive a data operation request sent by the target application. The data operation request includes the target data operation to be performed.
[0023] Specifically, the target application refers to external components that need to perform data operations on the underlying storage outside of Hadoop, such as Spark, Hive, and other big data components based on Hadoop.
[0024] A data operation request may specifically include the target data to be operated on, the target data operation to be performed on the target data, the storage path of the target data, the storage type of the target data, and storage configuration information (such as storage key, access address, storage access timeout, etc.). The target data may include, but is not limited to, file data, image data, video data, etc. Target data operations may include, but are not limited to: modifying the entity data or attribute data of the target data, deleting the target data, adding the target data, querying the target data, copying the target data, etc.
[0025] Based on the storage type of the target data, a big data plugin can be determined in Hadoop to perform operations on the target data. A big data plugin can include a data manipulation object. A data manipulation object is an implementation class that performs protocol conversion and data manipulation. Specifically, a data manipulation object can inherit the basic methods and properties of Hadoop's FileSystem abstract class. Developers can override the basic methods and properties of the FileSystem abstract class in the data manipulation object according to the protocol compatible with the external underlying storage, thus making the data manipulation object adaptable to the protocol of the external underlying storage.
[0026] Step S102: Instantiate the data operation object to obtain an instance object.
[0027] In this embodiment, when instantiating a data manipulation object, the following operations can be performed: 1) Create an empty instance.
[0028] 2) Allocate the hardware resources (such as memory space) required for running the empty instance, and load the instance properties defined by the data operation object into the empty instance to obtain the initial instance object.
[0029] 3) Inject the storage configuration information from the data operation request into the initial instance object, and bind the configuration parameters parsed from the storage configuration information to the initial instance object. These configuration parameters may include, but are not limited to, connection parameters of the external storage to be accessed by the target application (such as storage keys, access addresses, etc.), the storage path of the target data, and data plugin performance parameters (such as connection pool size, timeout, caching strategy, etc.).
[0030] 4) Initialize the external storage connection and resources, such as verifying the validity of the external storage connection parameters, creating a storage client instance, initializing the connection pool, loading cache configuration, etc., to obtain the instance object.
[0031] After the above instantiation operation, the instance object can convert FileSystem instructions into instructions that the target application can recognize in the external storage to be accessed, and perform data operations in the corresponding external storage.
[0032] Step S103: Initialize the instance object. Initialization includes creating a log aggregator for the instance object and reading log parameters. Log parameters refer to pre-configured constraints related to the printing of operation logs for the target data operation.
[0033] In this embodiment, by using techniques such as event tracking, the instance object can automatically collect operation logs during data operations. Event tracking refers to integrating log collection code into specified code locations within the instance object, enabling the instance object to automatically collect and output corresponding operation logs when performing specified data operations. For example, after integrating log collection code into the instance object's data reading code, the instance object will output corresponding data reading logs when reading data from external storage. Similarly, after integrating log collection code into the instance object's data deletion code, the instance object will output corresponding data deletion logs when deleting data from external storage. In practical applications, log collection code can be integrated into one or more code locations within the instance object as needed. This application does not limit the integration location of the log collection code.
[0034] The log aggregator is used to collect and print operation logs output by instance objects according to log parameters. Specifically, log parameters can include, but are not limited to, whether to print logs, the log printing interval, the trigger condition for log printing, the log printing content, and whether to save logs locally. For example, the log printing trigger condition can be configured as "data read count reaches 3 times". Based on this log parameter, when the instance object outputs 3 data read logs (i.e., the instance object performed 3 data read operations in external storage), the log aggregator is triggered to print the data read logs. As another example, the log printing interval can be configured as "5 seconds". Based on this log parameter, the log aggregator can retrieve the operation logs output by the instance object every 5 seconds and print the retrieved operation logs.
[0035] It should be noted that in practical applications, log parameters can be set according to actual needs, and this application does not impose any restrictions on this.
[0036] Step S104: Invoke the instance object to perform the target data operation. During the execution of the target data operation, the instance object collects operation logs. The log aggregator is used to obtain and print the operation logs collected by the instance object based on the log parameters. The operation logs printed by the log aggregator are used to optimize the performance of the big data plugin.
[0037] For details regarding the collection of instance object operation logs and the printing of operation logs by the log aggregator, please refer to the relevant description in step S103, which will not be repeated here.
[0038] In this embodiment, by setting reasonable log parameters, the log aggregator can statistically analyze the operation logs output by the instance object and print the log statistics results. This statistical analysis may include, but is not limited to, filtering, merging, and averaging of operation logs. For example, suppose the instance object outputs data write logs, data read logs, and data delete logs. Based on the log parameters, the log aggregator can print only the data read logs, thus achieving log filtering. As another example, if the instance object outputs the time consumed for each data read operation, based on the log parameters, the log aggregator can average the time consumed by multiple data read operations and print the calculated average time, thus achieving averaging. Yet another example, if the instance object outputs multiple data read operations, based on the log parameters, the log aggregator can merge these multiple data read operations and print only one data read operation, thus achieving log merging.
[0039] Of course, it's understandable that the log aggregator may also choose not to process the operation logs output by the instance object. In this case, the log aggregator can print all operation logs output by the instance object. In practical applications, log parameters can be set according to actual needs to control the way the log aggregator prints operation logs; this application does not impose any restrictions on this.
[0040] The logs printed by the log aggregator can be seen as performance metrics for big data plugins. Of course, performance metrics can also be obtained by performing further statistical analysis on the logs printed by the log aggregator.
[0041] Based on the above description, since instance objects can automatically collect and output operation logs, and the log aggregator can perform statistics on the operation logs output by instance objects based on log parameters and print the statistical results, automatic collection and statistics of operation logs can be achieved without manual collection and statistics of operation logs.
[0042] In summary, in some embodiments of this application, upon receiving a data operation request from the target application, the data operation object of the big data plugin is instantiated and initialized. Since initialization includes creating a log aggregator for the instance object and reading log parameters, and the instance object can collect operation logs during the execution of the target data operation, when the instance object is invoked to perform data operations, the log aggregator can statistically analyze and print the operation logs output by the instance object based on the log parameters. This application, based on the instance object, enables automatic collection of data operation logs, and based on the log aggregator and log parameters, enables automatic statistical analysis and printing of data operation logs. This solves the problem in related technologies where automatic collection and statistical analysis of operation logs for big data plugins is impossible, greatly improving the efficiency of log collection and statistical analysis.
[0043] In addition, this application uses techniques such as data tracking to enable instance objects to output operation logs in real time while performing data operations, which can realize real-time log collection and statistics, solving the problem of lag in log data collection and performance indicator statistics in some technologies.
[0044] Meanwhile, compared to manually collecting and analyzing log data, the instance object in this application triggers the output of operation logs when performing log operations, which can avoid data omissions and data errors.
[0045] In some embodiments, the log parameters mentioned above include a log display carrier, and initialization further includes creating a log buffer for the instance object; the log aggregator acquires and prints the operation logs collected by the instance object, including: When the instance object saves the collected operation logs to the log buffer, the log aggregator retrieves the operation logs from the log buffer every preset time interval and prints the retrieved operation logs to the log display carrier.
[0046] Specifically, during the initialization process in step S103, in addition to creating the log aggregator and reading the log parameters, it is also necessary to create a log temporary storage for the instance object. The log temporary storage is used to temporarily store the operation logs collected by the instance object when performing target data operations, such as operation type, operation time, number of operations, operation results, etc.
[0047] When an instance object performs a target data operation, the real-time collected operation logs (such as operation type, operation time, operation result, etc.) are continuously saved to the log buffer. The log aggregator periodically triggers the reading mechanism according to the preset interval, pulls and aggregates the data from the log buffer, and after obtaining the log operation, the log aggregator prints the log to the log display carrier, which includes file logs and console interface.
[0048] In the above embodiments, the log parameters include a log display carrier. A log buffer is created during initialization. The log aggregator periodically retrieves operation logs from the log buffer and prints them to the log display carrier, which avoids frequent log processing. At the same time, the logs are displayed in an orderly manner through the log aggregator, which facilitates efficient viewing and analysis and provides a reliable basis for optimizing the performance of big data plugins.
[0049] In some embodiments, the log parameters also include the log printing interval duration, and the initialization also includes setting a timer based on the log printing interval duration.
[0050] The log parameters read during the initialization process in step S103 also include the log printing interval duration, which is the time interval at which the log aggregator prints operation logs. Based on the log printing interval duration, a timer (e.g., 2 seconds) can be preset. The timer is used to count down according to the log printing interval duration, and after the countdown is complete, it outputs a trigger command to the log aggregator. Based on the timer, the log aggregator prints the acquired operation logs to the log display carrier, including: In response to receiving a timer trigger command, the log aggregator prints the operation logs within the most recent preset time period to the log display carrier. The timer counts according to the log printing interval and sends a trigger command to the log aggregator when the log printing interval is reached.
[0051] Once the target data operation is completed, the log aggregator will print the operation logs that have not yet been printed to the log display carrier.
[0052] Specifically, in these embodiments, using a timer to time the log printing interval improves timing accuracy. After the timer outputs a trigger command to the log aggregator, it can restart the timing from 0 according to the log printing interval. After the timing is complete, it outputs a trigger command to the log aggregator again, triggering the log aggregator to print the operation logs within the most recent preset time interval to the log display carrier. This cycle repeats. After the target data operation is completed, the timer can be stopped.
[0053] In some scenarios, the timer may still be counting down when the target data operation is completed. For example, suppose the target data operation lasts for 14 seconds and the log printing interval is 5 seconds, meaning the timer sends a trigger command to the log aggregator every 5 seconds. In this scenario, the timer would send trigger commands to the log aggregator at the 5th and 10th seconds of the target data operation. At the 14th second after the target data operation ends, since the timer's countdown timer has not yet expired, the timer will not send a trigger command to the log aggregator. This means the log aggregator cannot obtain the operation logs from the 10th to the 14th second.
[0054] Therefore, when the target data operation is completed and the instance object stops collecting operation logs, it can send a completion command indicating the completion of the operation to the log aggregator. After receiving the completion command, the log aggregator checks whether there are any unprinted operation logs in the log buffer, that is, the logs collected from the last printing to the completion of the target data operation. If so, it prints all these remaining operation logs to the log display carrier.
[0055] In the above embodiments, the logging is triggered by a timer at the designated log printing interval, ensuring that the operation log is output on a regular basis and that the remaining log is printed after the target data operation is completed, thus achieving timely feedback and complete recording of the log.
[0056] In some embodiments, the log parameters include timer setting parameters that indicate whether a timer is set. The log parameters read during the initialization process in step S103 also include timer setting parameters, which indicate whether a timer needs to be set for the instance object. Based on the timer setting parameters, the log aggregator prints the acquired operation logs to the log display carrier, including: If the timer setting parameter indicates that no timer is set, the timer setting operation will be interrupted, and the log aggregator will print the operation log of the acquired target data operation to the log display carrier after the target data operation is completed.
[0057] Specifically, when the timer setting parameter indicates that no timer is set, the timer setting operation based on the log printing interval can be stopped, meaning no timer is created. In this case, the instance object will still save the collected operation logs to the log buffer while performing the target data operation. When the target data operation is completed, the log aggregator receives the completion instruction, retrieves all operation logs collected throughout the target data operation from the log buffer, and prints them.
[0058] In the above embodiments, flexible log printing options are provided by setting whether or not to set a timer parameter. When the timer setting parameter is set to "no timer," the operation log can be printed all at once after the target data operation is completed, improving the overall efficiency of log management.
[0059] In some embodiments, the log display carrier includes at least one of the log files of the big data plugin and a log display page. The log aggregator prints the acquired operation logs to the log display carrier, and also includes: When the log display medium includes the log files of the big data plugin, if the log files of the big data plugin do not grant log printing permissions to the log aggregator, the log aggregator will print the operation logs to the log files of the target application.
[0060] Specifically, when the log display medium includes the log files of the big data plugin, the log aggregator checks whether the log file has permission to print before printing the operation logs. If permission is not granted, the printing path is switched, and the operation logs are printed to the target application's log file to record the time consumption and performance of the big data plugin.
[0061] In the above embodiments, the permission detection and path switching mechanism avoids log loss caused by big data plugin log file permission issues, ensuring the reliability of log recording and preserving complete original data for subsequent big data plugin performance optimization analysis.
[0062] In some embodiments, after initialization is complete, the method of this application further includes: The initialization duration is saved to the log buffer. The log aggregator retrieves the initialization duration from the log buffer and prints the initialization duration to the log display medium.
[0063] Specifically, the above steps can be performed after the initialization operation in step S103 is completed.
[0064] The total duration of the initialization operation is calculated, which is the time interval from the start of initializing the instance object to its completion, and this initialization duration is saved to the log register. The log register also saves the initialization duration and the subsequent operation logs collected for the instance object. When the log aggregator performs the log printing operation, it obtains both the initialization duration and the operation logs from the log register and prints them together to the log display medium.
[0065] In the above embodiments, recording the initialization time of instance objects makes the entire data operation process more complete, providing complete data for finding performance bottlenecks throughout the entire operation cycle.
[0066] In some embodiments, the log parameters include a print enable parameter indicating whether to print the operation log and a list of operations that need to be printed.
[0067] The log parameters read in step S103 also include print enable parameters and an operation list. The enable parameter indicates whether the operation log printing function is started, and the operation list represents the set of target data operations for which the operation log needs to be printed. For example, the constant class `Statistic` specifies the types of operations to be statistically analyzed: PUT, DELETE, LIST, GET, etc.
[0068] Upon receiving a data operation request from the target application, determine if one of the following conditions is met. If so, interrupt the operation related to the operation log: The print enable parameter indicates that the operation log will not be printed. The print enable parameter indicates that the operation log is to be printed, but the target data operation in the data operation request is not in the operation list.
[0069] Specifically, upon receiving a data operation request from the target application, the log printing conditions are first checked to see if they meet any of the following conditions: The print enable parameter indicates that if the operation log is not printed, operations related to the operation log will be globally interrupted and the log printing function will be turned off. The print enable parameter indicates that the operation log is to be printed, but the target data operation in the current data operation request does not appear in the operation list, meaning that the operation belongs to the type of operation that does not need to be logged.
[0070] If any of the above conditions are met, all subsequent operations related to the operation log will be interrupted, including log collection of instance objects, storage in the log buffer, printing in the log aggregator, and other processes.
[0071] In the above embodiments, the operation logs that need to be recorded are filtered by the operation list to avoid processing irrelevant operations, reduce unnecessary resource consumption, make the operation log data more targeted, and improve the efficiency of subsequent performance analysis.
[0072] See also Figure 2 This is a schematic diagram of a performance optimization device for big data plugins provided in some embodiments of this application. Figure 2 In this context, performance optimization devices for big data plugins include: The request receiving module 201 is used to receive a data operation request sent by the target application. The data operation request includes the target data operation to be performed. Instantiation module 202 is used to instantiate data operation objects to obtain instance objects; Initialization module 203 is used to initialize the instance object. Initialization includes creating a log aggregator for the instance object and reading log parameters. Log parameters refer to the pre-configured constraints related to the printing of operation logs for the target data operation. The data operation module 204 is used to call the instance object to perform the target data operation. During the execution of the target data operation, the instance object collects operation logs. The log aggregator is used to obtain and print the operation logs collected by the instance object based on the log parameters. The operation logs printed by the log aggregator are used to optimize the performance of the big data plugin.
[0073] In some embodiments, log parameters include a log display carrier, and initialization further includes creating a log register for the instance object; the log aggregator is also used for: When the instance object saves the collected operation logs to the log buffer, the log aggregator retrieves the operation logs from the log buffer every preset time interval and prints the retrieved operation logs to the log display carrier.
[0074] In some embodiments, the log parameters further include the log printing interval duration, and the initialization further includes setting a timer based on the log printing interval duration; the log aggregator is also used for: In response to receiving a timer trigger command, the log aggregator prints the operation logs within the most recent preset time period to the log display carrier. The timer counts according to the log printing interval and sends a trigger command to the log aggregator when the log printing interval is reached. Once the target data operation is completed, the log aggregator will print the operation logs that have not yet been printed to the log display carrier.
[0075] In some embodiments, log parameters include timer setting parameters that characterize whether a timer is set; the log aggregator is also used for: If the timer setting parameter indicates that no timer is set, the timer setting operation will be interrupted, and the log aggregator will print the operation log of the acquired target data operation to the log display carrier after the target data operation is completed.
[0076] In some embodiments, the log display carrier includes at least one of the log files of the big data plugin and the log display page; the log aggregator is also used for: When the log display medium includes the log files of the big data plugin, if the log files of the big data plugin do not grant log printing permissions to the log aggregator, the log aggregator will print the operation logs to the log files of the target application.
[0077] In some embodiments, after initialization is completed, the initialization module 203 is further configured to: The initialization duration is saved to the log buffer. The log aggregator retrieves the initialization duration from the log buffer and prints the initialization duration to the log display medium.
[0078] In some embodiments, the log parameters include a print enable parameter that indicates whether to print the operation log and a list of operations that need to be printed. After receiving a data operation request from the target application, the request receiving module 201 further determines whether one of the following conditions is met. If so, it interrupts the operation log-related operations: The print enable parameter indicates that the operation log will not be printed. The print enable parameter indicates that the operation log is to be printed, but the target data operation in the data operation request is not in the operation list.
[0079] For a description of the features in the embodiment corresponding to the performance optimization device for big data plugins, please refer to the relevant description in the embodiment corresponding to the sample data processing method, which will not be repeated here.
[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0081] See also Figure 3 Embodiments of this application also provide an electronic device, including a memory 10 and a processor 20, wherein the memory 10 stores a computer program and the processor 20 is configured to run the computer program to perform the steps in any of the above embodiments of the performance optimization method for big data plugins.
[0082] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the performance optimization method for big data plugins at runtime.
[0083] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0084] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the performance optimization method for big data plugins.
[0085] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the performance optimization method for big data plugins.
[0086] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0087] The foregoing has provided a detailed description of a performance optimization method, apparatus, device, and storage medium for big data plugins provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A performance optimization method for big data plugins, characterized in that, The big data plugin includes data manipulation objects, and the methods include: Receive a data operation request sent by the target application, the data operation request including the target data operation to be performed; The data manipulation object is instantiated to obtain an instance object; The instance object is initialized, which includes creating a log aggregator for the instance object and reading log parameters, wherein the log parameters refer to pre-configured constraints related to the printing of operation logs for the target data operation; The instance object is invoked to perform the target data operation. During the execution of the target data operation, the instance object collects operation logs. The log aggregator is used to obtain and print the operation logs collected by the instance object based on the log parameters. The operation logs printed by the log aggregator are used to optimize the performance of the big data plugin.
2. The method according to claim 1, characterized in that, The log parameters include the log display carrier, and the initialization also includes creating a log buffer for the instance object; The log aggregator acquires and prints the operation logs collected by the instance object, including: When the instance object saves the collected operation logs to the log buffer, the log aggregator retrieves the operation logs from the log buffer every preset time interval and prints the retrieved operation logs to the log display carrier.
3. The method according to claim 2, characterized in that, The log parameters also include the log printing interval duration, and the initialization also includes setting a timer based on the log printing interval duration; The log aggregator prints the acquired operation logs to the log display carrier, including: In response to receiving the trigger command from the timer, the log aggregator prints the operation logs within the most recent preset duration to the log display carrier, wherein the timer counts according to the log printing interval duration, and sends the trigger command to the log aggregator when the log printing interval duration is reached; Once the target data operation is completed, the log aggregator will print the operation logs that have not yet been printed to the log display carrier.
4. The method according to claim 3, characterized in that, The log parameters include timer setting parameters that indicate whether a timer is set; The log aggregator prints the acquired operation logs to the log display carrier, including: If the timer setting parameter indicates that no timer is set, the timer setting operation is interrupted, and when the target data operation is completed, the log aggregator prints the obtained operation log of the target data operation to the log display carrier.
5. The method according to claim 2, characterized in that, The log display carrier includes at least one of the log files of the big data plugin and the log display page; The log aggregator prints the acquired operation logs to the log display carrier, and also includes: When the log display carrier includes the log file of the big data plugin, if the log file of the big data plugin does not grant log printing permissions to the log aggregator, the log aggregator will print the operation log to the log file of the target application.
6. The method according to claim 2, characterized in that, After completing the initialization, the method further includes: The initialization duration is saved to the log buffer, the log aggregator obtains the initialization duration from the log buffer, and prints the initialization duration to the log display carrier.
7. The method according to any one of claims 1 to 6, characterized in that, The log parameters include a print enable parameter that indicates whether to print the operation log and a list of operations that need to be printed. Upon receiving the data operation request from the target application, determine whether one of the following conditions is met. If so, interrupt the operation log-related operations: The print enable parameter indicates that the operation log is not printed; The print enable parameter represents the print operation log, but the target data operation in the data operation request is not in the operation list.
8. A performance optimization device for big data plugins, characterized in that, The big data plugin includes data manipulation objects; the device includes: A request receiving module is used to receive a data operation request sent by a target application, wherein the data operation request includes the target data operation to be performed. The instantiation module is used to instantiate the data operation object to obtain an instance object; An initialization module is used to initialize the instance object. The initialization includes creating a log aggregator for the instance object and reading log parameters. The log parameters refer to pre-configured constraints related to the printing of operation logs for the target data operation. The data operation module is used to call the instance object to perform the target data operation. During the execution of the target data operation, the instance object collects operation logs. The log aggregator is used to obtain and print the operation logs collected by the instance object based on the log parameters. The operation logs printed by the log aggregator are used to optimize the performance of the big data plugin.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the performance optimization method for big data plugins as described in any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the performance optimization method for big data plugins as described in any one of claims 1 to 7.