Efficient data processing system, method and equipment based on Java and medium
By introducing multi-layer processing modules, algorithm optimization modules and resource management modules into the Java data processing framework, and using technologies such as parallel processing, compression algorithms and distributed computing, the existing Java data processing framework's performance bottlenecks and low resource utilization are solved, and efficient and low-cost data processing is achieved.
Patent Information
- Application Number
- CN202411689658.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-25
- Publication Date
- 2025-03-07
AI Technical Summary
The existing Java data processing framework has performance bottlenecks and low resource utilization, which is difficult to meet the needs of high efficiency and low consumption.
Design an efficient data processing system based on Java, including multi-layer processing module, algorithm optimization module and resource management module. Through parallel processing mechanisms, compression algorithms, distributed computing technology, real-time resource monitoring and dynamic adjustment, data processing flow and resource management are optimized.
It improves the speed and efficiency of data processing, reduces system resource consumption, and realizes more efficient data processing and resource management.
Smart Images
Figure CN119576562A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a Java-based efficient data processing system, method, device and medium. Background Art
[0002] With the advent of the big data era, the complexity and scale of data processing are increasing. Traditional data processing methods can no longer meet the needs of high efficiency and low consumption. Although the existing Java data processing framework has improved the processing efficiency to a certain extent, it still has problems such as performance bottlenecks and low resource utilization. Therefore, it is necessary to propose a new type of efficient data processing framework and its implementation method. Summary of the invention
[0003] The object of the present invention is to provide an efficient data processing system, method, device and medium based on Java, so as to solve all or one of the above problems existing in the prior art.
[0004] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows: On the one hand, the present invention provides an efficient data processing system based on Java, comprising: Multi-layer processing module, algorithm optimization module and resource management module; The multi-layer processing module is provided with a data source layer, a processing layer, a cache layer and an output layer; the data source layer, the processing layer, the cache layer and the output layer communicate and exchange data through a standardized interface; the data source layer, the processing layer, the cache layer and the output layer use a parallel processing mechanism to process data; The algorithm optimization module is used to pre-process the input data using a compression algorithm; the algorithm optimization module is also used to use distributed computing technology to divide a large data set into several small blocks for parallel processing and then merge the results; The resource management module is used to monitor the system resource usage in real time and dynamically adjust the priority of processing tasks and the amount of allocated resources; the algorithm optimization module is also used to release resources that are no longer in use in a timely manner.
[0005] As an improved solution, the data source layer is also used to receive data from different data sources using JDBC, ODBC, file I / O, and API interfaces; The different data sources include databases, file systems and network streams.
[0006] As an improved solution, the processing layer is also used to dynamically increase or decrease processing threads or processing processes according to task processing requirements; The processing layer is also used to perform parallel data processing using a multi-threaded or multi-processor architecture; the processing layer divides the large data set into several small blocks by calling the algorithm optimization module, and allocates the several small blocks to different processing units for parallel processing; after the processing by the different processing units is completed, the processing layer merges the results of the several small blocks into a final data set.
[0007] As an improved solution, the cache layer is also used to store temporary data and intermediate results in the data processing process; the cache layer adopts efficient memory management and data compression technology; the cache layer supports persistent data storage.
[0008] As an improved solution, the output layer is used to output the processed data to a target location, and the target location includes: a database, a file system, and a network stream; The output layer adopts a uniformly defined interface to support multiple output modes, and provides functions such as data format conversion and encryption.
[0009] As an improved solution, the algorithm optimization module is also used to pre-process the data using a compression algorithm before the data enters the processing layer; the compression algorithm includes: LZ4, Snappy and Zstandard; The algorithm optimization module is also used to segment large-scale data sets.
[0010] As an improved solution, the resource management module is also used to monitor the CPU usage, memory occupancy and disk I / O of the system in real time; the resource management module discovers and handles resource bottlenecks and performance problems by monitoring data; The resource management module is also used to dynamically adjust the priority of processing tasks and the amount of resources allocated according to monitoring data; the resource management module reduces the processing priority of tasks with high resource usage or allocates more resources; the resource management module increases the processing priority of tasks with idle resources or releases some resources to other tasks; The resource management module is also used to reclaim the memory and processor resources occupied by threads or processes that have completed processing tasks to the system; the resource management module is also used to clear cached data that has not been used for a long time from the memory and store it on the disk.
[0011] On the other hand, the present invention also provides an efficient data processing method based on Java, comprising the following steps: A data source layer, a processing layer, a cache layer and an output layer are configured; the data source layer, the processing layer, the cache layer and the output layer communicate and exchange data through a standardized interface; a parallel processing mechanism is used to process data among the data source layer, the processing layer, the cache layer and the output layer; Use compression algorithms to pre-process input data; use distributed computing technology to split large data sets into several small blocks for parallel processing and then merge the results; Monitor system resource usage in real time, dynamically adjust the priority of processing tasks and the amount of allocated resources; and promptly release resources that are no longer in use.
[0012] On the other hand, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the Java-based efficient data processing method are implemented.
[0013] On the other hand, the present invention further provides a computer device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; wherein: The memory is used to store computer programs; The processor is used to execute the steps of the Java-based efficient data processing method by running the program stored in the memory.
[0014] The beneficial effects of the technical solution of the present invention are: 1. The Java-based efficient data processing system described in the present invention can optimize the framework architecture, data processing algorithm and resource management mechanism through the mutual cooperation of system modules, thereby improving the speed and efficiency of data processing and reducing system resource consumption.
[0015] 2. The Java-based efficient data processing method described in the present invention can call the system modules in an orderly manner, thereby realizing the system logic of the Java-based efficient data processing system described in the present invention.
[0016] 3. The computer-readable storage medium described in the present invention can realize the cooperation of the guiding system module, thereby realizing the efficient Java-based data processing method described in the present invention, and the computer-readable storage medium described in the present invention also effectively improves the operability of the efficient Java-based data processing method.
[0017] 4. The computer device described in the present invention can store and execute the computer-readable storage medium, thereby realizing the Java-based efficient data processing method described in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0019] Figure 1 Schematic diagram of the architecture of the efficient data processing system based on Java according to Embodiment 1 of the present invention; Figure 2 is a flowchart of an efficient data processing method based on Java according to Embodiment 2 of the present invention; Figure 3 is a schematic diagram of the structure of the computer device described in Example 4 of the present invention; The symbols in the accompanying drawings are explained as follows: 1501, processor; 1502, communication interface; 1503, memory; 1504, communication bus. DETAILED DESCRIPTION
[0020] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.
[0021] In the description of the present invention, it should be noted that the embodiments described in the present invention are only part of the embodiments of the present invention, rather than all of the embodiments; based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present invention.
[0022] The terms "first", "second", etc. in the specification and claims of this article and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of this article described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. Example 1
[0023] This embodiment provides an efficient data processing system based on Java, such as Figure 1 As shown, including: Multi-layer processing module, algorithm optimization module and resource management module; The multi-layer processing module is provided with a data source layer, a processing layer, a cache layer and an output layer; the data source layer, the processing layer, the cache layer and the output layer communicate and exchange data through a standardized interface; the data source layer, the processing layer, the cache layer and the output layer use a parallel processing mechanism to process data; The algorithm optimization module is used to pre-process the input data using a compression algorithm; the algorithm optimization module is also used to use distributed computing technology to divide a large data set into several small blocks for parallel processing and then merge the results; The resource management module is used to monitor the system resource usage in real time and dynamically adjust the priority of processing tasks and the amount of allocated resources; the algorithm optimization module is also used to release resources that are no longer in use in a timely manner.
[0024] As an improved solution, the data source layer is also used to receive data from different data sources using JDBC, ODBC, file I / O, and API interfaces; The different data sources include databases, file systems and network streams.
[0025] As an improved solution, the processing layer is also used to dynamically increase or decrease processing threads or processing processes according to task processing requirements; The processing layer is also used to perform parallel data processing using a multi-threaded or multi-processor architecture; the processing layer divides the large data set into several small blocks by calling the algorithm optimization module, and allocates the several small blocks to different processing units for parallel processing; after the processing by the different processing units is completed, the processing layer merges the results of the several small blocks into a final data set.
[0026] As an improved solution, the cache layer is also used to store temporary data and intermediate results in the data processing process; the cache layer adopts efficient memory management and data compression technology; the cache layer supports persistent data storage.
[0027] As an improved solution, the output layer is used to output the processed data to a target location, and the target location includes: a database, a file system, and a network stream; The output layer adopts a uniformly defined interface to support multiple output modes, and provides functions such as data format conversion and encryption.
[0028] As an improved solution, the algorithm optimization module is also used to pre-process the data using a compression algorithm before the data enters the processing layer; the compression algorithm includes: LZ4, Snappy and Zstandard; The algorithm optimization module is also used to segment large-scale data sets.
[0029] As an improved solution, the resource management module is also used to monitor the CPU usage, memory occupancy and disk I / O of the system in real time; the resource management module discovers and handles resource bottlenecks and performance problems by monitoring data; The resource management module is also used to dynamically adjust the priority of processing tasks and the amount of resources allocated according to monitoring data; the resource management module reduces the processing priority of tasks with high resource usage or allocates more resources; the resource management module increases the processing priority of tasks with idle resources or releases some resources to other tasks; The resource management module is also used to reclaim the memory and processor resources occupied by the thread or process that has completed the processing task to the system; the resource management module is also used to clear the cache data that has not been used for a long time from the memory and store it on the disk; As a better example, in practical application, the functions of this system are as follows: (1) Data source layer: This layer is responsible for receiving data from different data sources, including databases, file systems, network streams, etc. The data source layer defines a unified interface (such as JDBC, ODBC, file I / O API, etc.) so that the framework can flexibly access various types of data sources. In addition, the data source layer is also responsible for the preliminary verification and preprocessing of data to ensure data integrity and consistency.
[0030] (2) Processing layer: The processing layer is the core part of the framework and is responsible for performing specific data processing tasks. In order to support parallel processing, the processing layer is designed to be scalable and can dynamically increase or decrease processing units (such as threads or processes) as needed. Each processing unit executes the same or different data processing algorithms, which are encapsulated into reusable components and dynamically loaded into the processing layer through configuration or programming.
[0031] (3) Parallel processing mechanism: In the processing layer, multi-threading or multi-processor architecture is used to achieve parallel processing. Through task decomposition and load balancing algorithms, large data sets are divided into multiple small blocks and assigned to different processing units for parallel processing. After the processing is completed, the results are merged into the final data set.
[0032] (4) Cache layer: The cache layer is used to store temporary data and intermediate results during the processing process to reduce direct access to the data source and repeated calculations. The cache layer uses efficient memory management and data compression technology to ensure maximum storage efficiency and access speed under limited resources. At the same time, the cache layer also supports persistent storage of data so that the processing status can be quickly restored when the system is restarted or fault recovery occurs.
[0033] (5) Output layer: The output layer is responsible for outputting the processed data to the target location, such as a database, file system, network stream, etc. The output layer also supports multiple output methods by defining a unified interface, and provides functions such as data format conversion and encryption to meet the data output requirements in different scenarios.
[0034] (6) Compression algorithm: Before the data enters the processing layer, it is first pre-processed using an advanced compression algorithm. The choice of compression algorithm depends on the characteristics of the data and the processing requirements. Common compression algorithms include LZ4, Snappy, Zstandard, etc. Compression processing can significantly reduce the overhead of data transmission and storage and improve processing efficiency.
[0035] (7) Distributed computing: For the processing of large-scale data sets, distributed computing technology is introduced to improve processing efficiency. In specific implementation, the large data set can be divided into multiple small blocks and assigned to different processing units for parallel processing. The processing units communicate and exchange data through the network to jointly complete the processing task of the entire data set. Finally, the results of each processing unit are merged into the final data set.
[0036] (8) Real-time monitoring: The system has a built-in resource monitoring module that monitors the usage of system resources in real time, including CPU usage, memory usage, disk I / O, etc. By monitoring the data, resource bottlenecks and potential performance issues can be discovered and handled in a timely manner.
[0037] (9) Dynamic adjustment: Based on the monitoring results, the system can dynamically adjust the priority of processing tasks and the amount of resources allocated. For tasks with high resource usage, their priority can be lowered or more resources can be allocated; for tasks with idle resources, their priority can be increased or some resources can be released for use by other tasks. In this way, the reasonable allocation and efficient use of system resources can be ensured.
[0038] (10) Resource recycling: The system also introduces a resource recycling mechanism to promptly release resources that are no longer in use. For example, for threads or processes that have completed their processing tasks, the memory and processor resources they occupy can be recycled back to the system; for cached data that has not been used for a long time, it can be cleared from the memory and stored on the disk for subsequent use. In this way, problems such as resource waste and memory leaks can be avoided.
[0039] It should be noted that the above examples are only for explaining the present invention and cannot limit the protection scope of the present invention. Example 2
[0040] This embodiment is based on the same inventive concept as the Java-based efficient data processing system described in Embodiment 1, and provides a Java-based efficient data processing method, such as Figure 2 As shown, the following steps are included: S100, configuring a data source layer, a processing layer, a cache layer and an output layer; the data source layer, the processing layer, the cache layer and the output layer communicate and exchange data through a standardized interface; the data source layer, the processing layer, the cache layer and the output layer use a parallel processing mechanism to process data; S200, pre-processing the input data using a compression algorithm; using distributed computing technology to divide the large data set into several small blocks for parallel processing, and then merging the results; S300, monitor the usage of system resources in real time, dynamically adjust the priority of processing tasks and the amount of allocated resources; and promptly release resources that are no longer in use. Example 3
[0041] This embodiment provides a computer-readable storage medium, including: The storage medium is used to store computer software instructions used to implement the Java-based efficient data processing method described in the above-mentioned Example 2, which includes a program for executing the above-mentioned program set for the Java-based efficient data processing method; specifically, the executable program can be built into the Java-based efficient data processing system described in Example 1, so that the Java-based efficient data processing system can implement the Java-based efficient data processing method described in Example 2 by executing the built-in executable program.
[0042] In addition, the computer-readable storage medium of this embodiment may adopt any combination of one or more computer-readable storage media, wherein the computer-readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or components, or any combination thereof. Example 4
[0043] This embodiment provides an electronic device, such as Figure 3 As shown, the electronic device may include: a processor 1501 , a communication interface 1502 , a memory 1503 and a communication bus 1504 , wherein the processor 1501 , the communication interface 1502 , and the memory 1503 communicate with each other via the communication bus 1504 .
[0044] Memory 1503, used for storing computer programs; The processor 1501 is used to implement the steps of the Java-based efficient data processing method described in the above-mentioned embodiment 1 when executing the computer program stored in the memory 1503.
[0045] As an embodiment of the present invention, the communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0046] As an implementation mode of the present invention, the communication interface is used for communication between the above-mentioned terminal and other devices.
[0047] As an embodiment of the present invention, the memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0048] As an embodiment of the present invention, the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0049] Different from the prior art, the present application provides an efficient Java-based data processing system, method, device and medium, which can optimize the framework architecture, data processing algorithm and resource management mechanism, improve the speed and efficiency of data processing, and reduce system resource consumption.
[0050] It should be understood that in the various embodiments of this document, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.
[0051] It should also be understood that in the embodiments of this article, the term "and / or" is only a description of the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.
[0052] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this article.
[0053] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific logical process of the method described above can refer to the corresponding working processes of the systems, devices and units in the aforementioned method embodiments, and will not be repeated here.
[0054] In the several embodiments provided herein, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.
[0055] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this article.
[0056] In addition, each functional unit in each embodiment of this invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.
[0057] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this article is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of this article. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0058] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. An efficient data processing system based on Java, characterized in that: include: Multi-layer processing module, algorithm optimization module and resource management module; The multi-layer processing module is provided with a data source layer, a processing layer, a cache layer and an output layer; The data source layer, the processing layer, the cache layer and the output layer communicate and exchange data through a standardized interface; The data source layer, the processing layer, the cache layer and the output layer use a parallel processing mechanism to process data; The algorithm optimization module is used to pre-process the input data using a compression algorithm; The algorithm optimization module is also used to use distributed computing technology to divide a large data set into several small blocks for parallel processing and then merge the results; The resource management module is used to monitor the system resource usage in real time and dynamically adjust the priority of processing tasks and the amount of allocated resources; The algorithm optimization module is also used to release resources that are no longer used in a timely manner.
2. The Java-based efficient data processing system according to claim 1, characterized in that: The data source layer is also used to receive data from different data sources using JDBC, ODBC, file I / O, and API interfaces; The different data sources include databases, file systems and network streams.
3. The Java-based efficient data processing system according to claim 1, characterized in that: The processing layer is also used to dynamically increase or decrease processing threads or processing processes according to task processing requirements; The processing layer is also used to perform parallel data processing using a multi-threaded or multi-processor architecture; the processing layer divides the large data set into several small blocks by calling the algorithm optimization module, and allocates the several small blocks to different processing units for parallel processing; after the processing by the different processing units is completed, the processing layer merges the results of the several small blocks into a final data set.
4. The Java-based efficient data processing system according to claim 1, characterized in that: The cache layer is also used to store temporary data and intermediate results in the data processing process; the cache layer adopts efficient memory management and data compression technology; the cache layer supports persistent data storage.
5. The Java-based efficient data processing system according to claim 1, characterized in that: The output layer is used to output the processed data to a target location, which includes: a database, a file system, and a network stream; The output layer adopts a uniformly defined interface to support multiple output modes, and provides functions such as data format conversion and encryption.
6. The Java-based efficient data processing system according to claim 4, characterized in that: The algorithm optimization module is also used to pre-process the data using a compression algorithm before the data enters the processing layer; the compression algorithm includes: LZ4, Snappy and Zstandard; The algorithm optimization module is also used to segment large-scale data sets.
7. The Java-based efficient data processing system according to claim 4, characterized in that: The resource management module is also used to monitor the CPU usage, memory occupancy and disk I / O of the system in real time; the resource management module discovers and handles resource bottlenecks and performance issues by monitoring data; The resource management module is also used to dynamically adjust the priority of processing tasks and the amount of resources allocated according to monitoring data; the resource management module reduces the processing priority of tasks with high resource usage or allocates more resources; the resource management module increases the processing priority of tasks with idle resources or releases some resources to other tasks; The resource management module is also used to reclaim the memory and processor resources occupied by threads or processes that have completed processing tasks to the system; the resource management module is also used to clear cached data that has not been used for a long time from the memory and store it on the disk.
8. An efficient data processing method based on Java, characterized in that: The following steps are involved: Configure the data source layer, processing layer, cache layer, and output layer; The data source layer, the processing layer, the cache layer and the output layer communicate and exchange data through a standardized interface; The data source layer, the processing layer, the cache layer and the output layer use a parallel processing mechanism to process data; Use compression algorithms to pre-process input data; use distributed computing technology to split large data sets into several small blocks for parallel processing and then merge the results; Monitor system resource usage in real time, dynamically adjust the priority of processing tasks and the amount of allocated resources; and promptly release resources that are no longer in use.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the Java-based efficient data processing method described in claim 8 are implemented.
10. A computer device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; wherein: The memory is used to store computer programs; The processor is used to execute the steps of the Java-based efficient data processing method described in claim 8 by running the program stored in the memory.