Method and device for expanding computing power of application program
By building the actuator module and Netty-based remote communication in the big data scenario, the application computing power is improved without increasing the system complexity and hardware resources, and the application computing power is solved in the big data scenario. The problem has been solved by a large number of business scenarios.
Patent Information
- Application Number
- CN202411226142.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-17
AI Technical Summary
In the process of big data scenarios, the system complexity and hardware requirements are high, making it difficult to meet the growing business scenarios and business data needs of applications, and there is an Out Of Memory risk.
By building a computing module containing executors, encapsulate rules for batch processing or time-consuming operations in the business, and implement remote communication and load balancing between the client and the server based on Netty in the application service and executor of the application, performing Tasks compute and write results to the database or cache, and use a global thread pool to allocate tasks to reduce resource usage.
Without increasing system complexity and hardware resources, the computing power of the application can be significantly improved, which can meet the needs of massive business scenarios, and the executor batch processing service can be expanded at a level, solving the problem of insufficient computing power of the application.
Smart Images
Figure CN119149233A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of Internet big data processing technology, and in particular to a method and device for expanding the computing capabilities of an application. Background Art
[0002] In the field of big data, users have a large amount of business data, and they need to process and calculate based on these metadata information to meet various needs of users and better manage their business data. The large amount of business data and the complexity of functions put forward higher requirements for the stability and timeliness of the system and the interface. For complex business scenarios, there are mainly two traditional implementation solutions.
[0003] 1. Split the complex business system into different business subsystems according to different functions and make them perform their respective duties. The disadvantage of this solution is that it greatly increases the complexity of the system and increases the development and maintenance costs.
[0004] 2. By introducing a thread pool for asynchronous processing, due to the large amount of business data and time-consuming calculations, the system JVM memory will be occupied for a long time. As the number of business scenarios increases, there is a risk of Out Of Memory (memory overflow), which places high requirements on hardware.
[0005] How to meet the growing business scenarios and business data of applications without increasing the complexity of the system too much and without high requirements for the operating environment has become an urgent problem to be solved. Summary of the invention
[0006] The purpose of this application is to overcome the problem in the prior art that in order to conveniently expand the computing power of an application, it is necessary to split the system business and increase the machine memory resources, and to provide a method and device for expanding the computing power of an application.
[0007] In a first aspect, a method for expanding the computing capability of an application is provided, comprising:
[0008] Constructing a computing module including at least one executor, in which rules for batch processing or time-consuming operations in the business are encapsulated;
[0009] Implement remote communication and load balancing between client and server based on Netty in the application service and executor of the application;
[0010] If the executor receives the parameters of the application, it executes the task calculation and writes the calculation results of the task calculation into the database or cache;
[0011] Use the application to read the calculation results in the database or cache.
[0012] In some possible implementations, the rules for batch processing or time-consuming operations in the encapsulated business in the executor include:
[0013] Encapsulate common and time-consuming business logic operations into different methods according to business needs;
[0014] The method is encapsulated into ExecutClient to define a unified access interface as the entry for triggering task execution.
[0015] Some possible implementations of implementing remote communication and load balancing between clients and servers based on Netty in the application service and executor of the application include:
[0016] The application service initiates a remote call based on the big data component service that needs to be operated;
[0017] The client encapsulates the method and parameters into a message body that can be transmitted over the network;
[0018] The client encodes the message body and sends it to the server;
[0019] The server receives the message body and decodes it;
[0020] The server selects an executor for calculation from the calculation module through a preset load balancing strategy;
[0021] The server calls the corresponding method service in the ExecutClient of the selected executor according to the decoding result;
[0022] The server encodes the execution result of the service of the corresponding method and sends it to the client;
[0023] The client receives the message and decodes the encoded execution result to obtain the final service call result.
[0024] In some possible implementations, executing task calculations and writing calculation results of the task calculations into a database or cache include:
[0025] Define a unified access interface for the operations of the task top-level interface as the entry point for triggering task execution;
[0026] Carry out different business logic according to different methods;
[0027] Define a global thread pool;
[0028] Split large batches of data into different batches and assign them to idle threads for execution;
[0029] Resource isolation between each thread;
[0030] Write the results of the calculations performed in each thread to a database or cache.
[0031] Some possible implementations also include:
[0032] Perform secondary calculation on the read calculation result;
[0033] The result after the secondary calculation is applied.
[0034] In a second aspect, a device for expanding the computing capability of an application is provided, comprising:
[0035] A construction module, used to construct a computing module including at least one executor, in which rules for batch processing or time-consuming calculations in a business are encapsulated;
[0036] Remote call and load balancing module, which is used to implement remote communication and load balancing between the client and the server based on Netty in the application service and executor of the application;
[0037] The calculation and writing module executes the task calculation in response to the executor receiving the parameters of the application program, and writes the calculation results of the task calculation into the database or cache;
[0038] The reading module is used to use the application to read the calculation results in the database or cache.
[0039] Some possible implementations also include:
[0040] A secondary calculation module, used for performing secondary calculation on the read calculation results;
[0041] The application module is used to apply the result after the secondary calculation.
[0042] In a third aspect, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method described in the first aspect.
[0043] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes steps for executing the method in any one of the implementations of the first aspect described above.
[0044] In a fifth aspect, an electronic device is provided, comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements a method as in any one of the implementations in the first aspect above.
[0045] The present application has the following beneficial effects: the present application can meet the growing massive business scenarios and business data without splitting the system business and without increasing the machine memory resources. By abstracting and encapsulating the executor batch processing service, the time-consuming batch processing tasks are extracted from the business system, which can greatly increase the computing power of the application with minimal modification cost; and with the subsequent growth of the business, the executor batch processing service can also be horizontally expanded, which can effectively solve the problem of insufficient computing power of the application in the big data scenario. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application.
[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0048] Figure 1 is a flow chart of a method for expanding the computing capability of an application program according to Embodiment 1 of the present application;
[0049] Figure 2 It is a schematic diagram of the working process of the method for expanding the computing capability of an application program according to Embodiment 1 of the present application;
[0050] Figure 3 It is a schematic diagram of encapsulating the calculation logic by taking the desensitization table evaluation calculation as an example in the method for expanding the computing capability of an application program in Example 1 of the present application;
[0051] Figure 4 It is a schematic diagram of remote communication between a server and a client in the method for expanding the computing capability of an application program in Example 1 of the present application;
[0052] Figure 5 It is a schematic diagram of executing task calculation and writing calculation results in the method for expanding the computing capability of an application program in Example 1 of the present application;
[0053] Figure 6 It is a schematic diagram of reading calculation results and applying them in the method for expanding the computing capability of an application program in Example 1 of the present application;
[0054] Figure 7 is a structural block diagram of a device for expanding the computing capability of an application program according to Embodiment 2 of the present application;
[0055] Figure 8It is a schematic diagram of the internal structure of the electronic device of Example 5 of the present application.
[0056] Reference numerals:
[0057] 100, construction module; 200, remote call and load balancing module; 300, calculation and writing module; 400, reading module; 500, secondary calculation module; 600, application module. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] Example 1
[0060] A method for expanding the computing power of an application involved in Example 1 of the present application includes: constructing a computing module including at least one executor, encapsulating rules for batch processing or time-consuming operations in the business in the executor; implementing remote communication and load balancing between the client and the server based on Netty in the application service and executor of the application; if the executor receives the parameters of the application, it executes the task calculation and writes the calculation results of the task calculation to a database or cache; and using the application to read the calculation results in the database or cache. This method can meet the growing massive business scenarios and business data without splitting the system business and without increasing the machine memory resources.
[0061] Specifically, Figure 1 A flowchart of a method for expanding the computing capability of an application program in Embodiment 1 of the present application is shown, including:
[0062] S100: Construct a computing module including at least one executor, in which the rules for batch processing or time-consuming operations in the business are encapsulated. Figure 2 , specifically including the following steps:
[0063] S101. Taking the evaluation calculation of the desensitized table as an example, firstly, according to the multiple databases and tables selected in batches by the user, the field information of each table is traversed and queried one by one.
[0064] S102. Filter out fields that meet the requirements according to the desensitization configuration match and bind them to the corresponding desensitization rules.
[0065] S103. For the desensitized fields that have been screened out, query the field blood relationship in the business library, and also perform desensitization rule binding on the related fields in the downstream table of the desensitized fields.
[0066] S104, writing the above desensitization rules into the database.
[0067] S105 , encapsulate the above calculation logic into ExecutClient (ie, batch processing service interface) to define a unified access interface as an entry for triggering task execution.
[0068] In the above steps, time-consuming batch task processing is encapsulated into batch processing services according to business needs, so as to decouple task computing from platform functions and ensure that task computing does not affect the performance and use of the business platform.
[0069] S200, implement remote communication and load balancing between client and server based on Netty in the application service and executor of the application, please refer to Figure 3 , specifically including the following steps:
[0070] S201. TaskScheduler (i.e., application service) initiates a remote call based on the big data component service that needs to be operated.
[0071] S202. NettyRemoteClient (ie, the client) encapsulates the method and parameters into a message body that can be transmitted over the network.
[0072] S203, NettyRemoteClient (ie, the client) encodes the message and sends it to NettyRemoteServer (ie, the server).
[0073] S204. NettyRemoteServer (i.e., the server) decodes the message after receiving it.
[0074] S205. NettyRemoteServer (i.e., the server) calls the corresponding OperatorProxy (i.e., the specific execution method) service in ExecutClient according to the decoding result.
[0075] S206. NettyRemoteServer (i.e., the server) encodes the returned result and sends it to NettyRemoteClient (i.e., the client).
[0076] S207. NettyRemoteClient (ie, the client) receives the message and decodes it.
[0077] S208. TaskScheduler (ie, application service) obtains the final service call result.
[0078] In the above steps, the remote communication client NettyRemoteClient (i.e., the server-side programming framework) and the remote communication server NettyRemoteServer (i.e., the server) are implemented based on Netty (i.e., the server-side programming framework). The task scheduler TaskScheduler (i.e., the application service) initiates a remote call to the executor ExecutClient (i.e., the batch processing service) through NettyRemoteClient (i.e., the client). ExecutClient (i.e., the batch task service) accepts the remote call and processes the request operation through NettyRemoteServer (i.e., the application service).
[0079] S300. If the executor receives the parameters of the application program, it executes the task calculation and writes the calculation result of the task calculation into the database or cache. Figure 4 , specifically including the following steps:
[0080] S301. Define a unified access interface for the operation of ExecutClient (ie, the top-level interface of the task) as an entry for triggering task execution.
[0081] S302. Different business logics are followed according to different methods. Taking the desensitized table matching as an example, the parameters need to be firstly checked for idempotence. If the last execution has not been completed, it is not allowed to be executed again. Otherwise, the following logic is followed normally.
[0082] S303. Define a global thread pool to facilitate unified adjustment of parameters and avoid abuse of threads.
[0083] S304. Use the divide-and-conquer idea to split large batches of data into different batches and assign them to different threads for execution to speed up processing.
[0084] S305. Resource isolation should be performed between each thread to prevent an error in a certain part from affecting the entire calculation result.
[0085] S306: Write the calculation results executed in each thread into the DB.
[0086] In the above steps, according to the obtained parameters and methods, firstly, the parameters are checked for idempotency to prevent the execution from being completed again before the last execution. Then, the business logic is executed according to the method, and finally the calculation results are written to the DB (i.e., the database, which can also be put into the cache).
[0087] S400, use the application to read the calculation results in the database or cache, see Figure 5 , where the application can perform the following processing after reading the calculation results:
[0088] S500, perform secondary calculations such as simple aggregation statistics on the results
[0089] S600: Use the result of the secondary calculation for BI display or provide it to other business platforms.
[0090] In the above steps, in the application, the data written to the DB (i.e., database, or cache) by the executor (i.e., batch processing service) is obtained for the interface function display to obtain other operations.
[0091] In this embodiment, compared with the prior art, this embodiment abstracts and encapsulates the executor (i.e., batch processing service), which can extract time-consuming batch processing tasks from the business system, greatly increase the computing power of the application with minimal modification cost; and with the subsequent growth of the business, the executor (i.e., batch processing service) can also be horizontally expanded, which can effectively solve the problem of insufficient computing power of the application in big data scenarios.
[0092] Example 2
[0093] like Figure 6 As shown, a device for expanding the computing capability of an application program according to Embodiment 2 of the present application includes:
[0094] A construction module 100 is used to construct a computing module including at least one executor, in which rules for batch processing or time-consuming calculations in a business are encapsulated;
[0095] The remote call and load balancing module 200 is used to implement remote communication and load balancing between the client and the server based on Netty in the application service and executor of the application program;
[0096] The calculation and writing module 300 executes the task calculation in response to the executor receiving the parameters of the application program, and writes the calculation results of the task calculation into the database or cache;
[0097] The reading module 400 is used to read the calculation results in the database or cache using the application program.
[0098] In some possible embodiments, it further includes:
[0099] A secondary calculation module 500, used for performing secondary calculation on the read calculation result;
[0100] The application module 600 is used to apply the result after the secondary calculation.
[0101] It should be noted that other specific implementations of the device for expanding the computing power of an application in this embodiment can refer to the specific implementations of the method for expanding the computing power of an application described above, and will not be described again here to avoid redundancy.
[0102] Example 3
[0103] A computer program product involved in Example 3 of the present application is used to store a computer program. When the computer program runs on a computer, the method in any one of the implementation modes in Example 1 of the present application is implemented.
[0104] Example 4
[0105] A computer-readable storage medium according to Embodiment 4 of the present application, wherein the computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing a method in any one of the implementations in Embodiment 1 of the present application;
[0106] Among them, the computer-readable storage medium can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM); the computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by the processor, the processor is used to execute the steps of the method in any one of the implementation methods in Example 1 of the present application.
[0107] Example 5
[0108] like Figure 7 As shown, an electronic device involved in Embodiment 5 of the present application includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, a method in any one of the implementations in Embodiment 1 of the present application is implemented;
[0109] Among them, the processor can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method in any one of the implementation methods in Example 1 of the present application.
[0110] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of Embodiment 1 of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in software form.
[0111] The above-mentioned processor may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the functions required to be performed by the unit included in the data processing device of the embodiment of the present application in combination with its hardware, or executes the method in any one of the implementation modes in the embodiment 1 of the present application.
[0112] The above are only preferred specific implementations of the present application; however, the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved ideas of the present application within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.
Claims
1. A method for expanding the computing capability of an application, characterized in that: include: Constructing a computing module including at least one executor, in which rules for batch processing or time-consuming operations in the business are encapsulated; Implement remote communication and load balancing between client and server based on Netty in the application service and executor of the application; If the executor receives the parameters of the application, it executes the task calculation and writes the calculation results of the task calculation into the database or cache; Use the application to read the calculation results in the database or cache.
2. The method for expanding the computing capability of an application according to claim 1, characterized in that: The rules for batch processing or time-consuming operations in the business are encapsulated in the executor, including: Encapsulate common and time-consuming business logic operations into different methods according to business needs; The method is encapsulated into ExecutClient to define a unified access interface as the entry for triggering task execution.
3. The method for expanding the computing capability of an application program according to claim 2, characterized in that: In the application service and executor of the application, remote communication and load balancing between the client and the server are implemented based on Netty, including: The application service initiates a remote call based on the big data component service that needs to be operated; The client encapsulates the method and parameters into a message body that can be transmitted over the network; The client encodes the message body and sends it to the server; The server receives the message body and decodes it; The server selects an executor for calculation from the calculation module through a preset load balancing strategy; The server calls the corresponding method service in the ExecutClient of the selected executor according to the decoding result; The server encodes the execution result of the service of the corresponding method and sends it to the client; The client receives the message and decodes the encoded execution result to obtain the final service call result.
4. The method for expanding the computing capability of an application program according to claim 1 or 3, characterized in that: Execute task calculations and write the calculation results of the task calculations to the database or cache, including: Define a unified access interface for the operations of the task top-level interface as the entry point for triggering task execution; Carry out different business logic according to different methods; Define a global thread pool; Split large batches of data into different batches and assign them to idle threads for execution; Resource isolation between each thread; Write the results of the calculations performed in each thread to a database or cache.
5. The method for expanding the computing capability of an application program according to claim 1, characterized in that: Also includes: Perform secondary calculation on the read calculation result; The result after the secondary calculation is applied.
6. A device for expanding the computing capability of an application, characterized in that: include: A construction module, used to construct a computing module including at least one executor, in which rules for batch processing or time-consuming calculations in a business are encapsulated; Remote call and load balancing module, which is used to implement remote communication and load balancing between the client and the server based on Netty in the application service and executor of the application; The calculation and writing module executes the task calculation in response to the executor receiving the parameters of the application program, and writes the calculation results of the task calculation into the database or cache; The reading module is used to use the application to read the calculation results in the database or cache.
7. The device for expanding the computing capability of an application program according to claim 6, characterized in that: Also includes: A secondary calculation module, used for performing secondary calculation on the read calculation results; The application module is used to apply the result after the secondary calculation.
8. A computer program product, characterized in that The computer program product stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing the method according to any one of claims 1 to 5.
10. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the method as claimed in any one of claims 1 to 5.