A data processing method and a data processing device
By dynamically adjusting the number of subprocesses and hardware resource configuration, and using multiple subprocesses to optimize the JNIEnv channel, the problem that JVM and C library performance cannot be fully utilized is solved, and the optimal configuration and performance improvement of hardware resources are achieved.
Patent Information
- Application Number
- CN202211171189.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-23
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-09-23
AI Technical Summary
During the high concurrent request processing, the performance of Java virtual machines (JVMs) and C libraries cannot be fully utilized, mainly because the JNIEnv bridge becomes a performance bottleneck, resulting in the inability to fully utilize hardware resources, and the hardware needs to be continuously expanded to improve performance.
By dynamically adjusting the number of subprocesses and hardware resource configuration, using multiple subprocesses to establish JNIEnv channels separately, optimizing the number of subprocesses and resource allocation according to the monitoring results, ensuring that electronic devices provide optimal performance under different hardware resources.
It effectively breaks through the performance bottleneck of JNIEnv bridge, rationally configures hardware resources, improves the performance and efficiency of data processing, avoids waste of hardware resources, and realizes the optimal utilization of hardware resources.
Smart Images

Figure CN115617451B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method and a data processing device. Background Art
[0002] In the prior art, Java Native Interface (JNIEnv) technology can be used to enable mutual calls between Java and C / C++. The business logic part of the subprocess needs to be supported by the Java Virtual Machine (JVM), and the operation logic part of the subprocess needs to be supported by the C library developed based on the C language, so that reliable external encryption technology can be provided for the subprocess. However, when the subprocess handles high-concurrency requests, the more requests are processed, the more hardware resources will be consumed by the JVM and C library, such as the central processing unit (CPU) and memory. At this time, it is necessary to expand the hardware resources, such as adding CPU and memory.
[0003] As hardware resources continue to expand, the single-threaded JVM can adjust the CPU and memory it occupies to improve performance, and the C library can also allocate more CPU and memory to improve computing performance. At this time, JNIEnv, the bridge between Java and V, becomes a performance bottleneck, resulting in the inability to fully utilize the performance of the JVM and C library. Summary of the invention
[0004] The present application provides a data processing method and a data processing device, which can break through the performance bottleneck of the JNIEnv bridge channel, and can reasonably determine the optimal number of sub-processes to be started for an electronic device and formulate the optimal resource configuration for the hardware resources of the sub-processes.
[0005] In a first aspect, the present application provides a data processing method, the method comprising:
[0006] Determine an initial number of sub-processes to be started based on a first hardware resource of the electronic device, and allocate initial hardware resources to the initial number of sub-processes;
[0007] Monitoring the working status of the initial number of sub-processes in processing data requests to obtain a monitoring result, adjusting the initial number of sub-processes based on the monitoring result to obtain a target number of sub-processes, and / or reallocating target hardware resources to the target number of sub-processes based on the monitoring result;
[0008] The data request is continued to be processed by the target number of child processes based on corresponding target hardware resources.
[0009] The embodiment of the present application provides a data processing method, including: determining the initial number of subprocesses to be started based on the first hardware resource of the electronic device, and allocating initial hardware resources to the initial number of subprocesses; monitoring the working status of the initial number of subprocesses processing data requests to obtain monitoring results, adjusting the initial number of subprocesses based on the monitoring results to obtain the target number of subprocesses, and / or re-allocating the target number of subprocesses based on the monitoring results; continuing to process data requests through the target number of subprocesses based on the corresponding target hardware resources. The present application determines the initial number of subprocesses to be started according to the hardware resource information of the electronic device, and allocates initial hardware resources to each subprocess when it is started, and establishes JNIEnv channels respectively through multiple subprocesses to achieve the purpose of breaking through the performance bottleneck of the JNIEnv bridge channel; and in the process of subprocesses processing data requests, dynamically adjusting the number of subprocesses and the hardware resources of each subprocess according to the monitoring results of the subprocesses, can reasonably determine the optimal number of subprocesses to be started for the electronic device and formulate the optimal resource configuration for the hardware resources of the subprocesses.
[0010] Furthermore, the first hardware resources include total memory resources and total processor resources; determining the initial number of sub-processes to be started based on the first hardware resources of the electronic device includes: determining the size relationship between the value of the total memory resources and the value of the total processor resources; and determining the initial number of sub-processes based on the smaller of the size relationship.
[0011] Furthermore, the initial hardware resources include initial memory information; the allocating initial hardware resources to the initial number of sub-processes includes: using a preset first calculation method to calculate the value of the total memory resources and the initial number of sub-processes to obtain a first memory value; if the first memory value is greater than a preset fixed value, the preset fixed value is used as the value of the initial memory information of the sub-process; if the first memory value is not greater than the preset fixed value, the first memory value is used as the value of the initial memory information of the sub-process.
[0012] Furthermore, the method of using a preset first calculation method to calculate the value of the total memory resources and the initial number of the sub-processes to obtain a first memory value includes: performing a numerical calculation on the value of the total memory resources and the initial number of the sub-processes to obtain a first value; and performing a numerical calculation on the first value and a preset conversion factor to obtain the first memory value.
[0013] Furthermore, the initial hardware resource information also includes initial processor information; allocating initial hardware resources to the initial number of sub-processes includes: determining whether the value of the total processor resources and the initial number of sub-processes are divisible; if so, using the quotient after the division as the value of the initial processor information of the sub-process; if not, determining the value of the initial processor information of the sub-process based on the quotient and remainder after the division.
[0014] Further, when the monitoring result shows that the initial number of sub-processes are in the first state, the initial number of sub-processes are adjusted based on the monitoring result to obtain the target number of sub-processes, and the initial hardware resources of the sub-processes are adjusted based on the monitoring result to obtain the target hardware resources, including: determining the value of the total processor resources as the target number; determining the number of newly opened sub-processes based on the difference between the target number and the initial number to obtain the target number of sub-processes, wherein the target number is greater than the initial number; and reallocating target hardware resources to the target number of sub-processes based on the value of the total processor resources and the target number.
[0015] Furthermore, the target hardware resources include target memory information and target processor information; the reallocation of target hardware resources to the target number of sub-processes based on the value of the total processor resources and the target number includes: determining a memory adjustment parameter based on the value of the total processor resources, and calculating the value of the total memory resources and the memory adjustment parameter to obtain a second memory value; correcting the second memory value, and using the corrected second memory value as the value of the target memory information of the sub-process; and determining the target processor information for the sub-process based on the value of the total processor resources and the target number.
[0016] Furthermore, the second memory value is corrected, and the corrected second memory value is used as the value of the target memory information of the child process, including: if the second memory value exceeds the first value, the first value is used as the value of the target memory information; if the second memory value does not exceed the second value, the second value is used as the value of the target memory information, and the target quantity is corrected according to the value of the target memory information to obtain the corrected target quantity, and the operation of determining the target processor information for the child process based on the value of the total processor resources and the target quantity is re-executed, and the first value is greater than the second value.
[0017] Furthermore, when the monitoring result shows that the initial number of sub-processes are in the second state, the initial number is used as the target number; target hardware resources are reallocated to the target number of sub-processes based on the monitoring result, including: using a preset second calculation method to calculate the value of the total memory resources and the target number of sub-processes to obtain a third memory value; verifying the third memory value, and using the verified third memory value as the value of the target memory information of the sub-process.
[0018] Furthermore, the monitoring of the working status of the initial number of sub-processes in processing data requests to obtain monitoring results includes: in the process of the sub-processes processing data requests, determining the concurrency status of the data requests, and determining the data calculation status required for the data requests; if the concurrency status is high concurrency and the data calculation status is small in amount of calculation, then determining that the monitoring result is that the initial number of sub-processes are in the first state; if the concurrency status is low concurrency and the data calculation status is large in amount of calculation, then determining that the monitoring result is that the initial number of sub-processes are in the second state.
[0019] In a second aspect, the present application provides a data processing device, the device comprising:
[0020] A resource allocation module, configured to determine an initial number of sub-processes to be started based on a first hardware resource of the electronic device, and allocate initial hardware resources to the initial number of sub-processes;
[0021] a resource adjustment module, configured to monitor the working status of the initial number of sub-processes in processing data requests to obtain a monitoring result, adjust the initial number of sub-processes based on the monitoring result to obtain a target number of sub-processes, and / or reallocate target hardware resources to the target number of sub-processes based on the monitoring result;
[0022] The data processing module is used to continue processing the data request through the target number of sub-processes based on the corresponding target hardware resources.
[0023] In a third aspect, the present application provides an electronic device, the electronic device comprising:
[0024] at least one processor; and
[0025] a memory communicatively connected to the at least one processor; wherein,
[0026] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing method described in any embodiment of the present application.
[0027] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing method described in any embodiment of the present application when executed.
[0028] It should be noted that the above computer instructions may be stored in whole or in part on a computer-readable storage medium, wherein the computer-readable storage medium may be packaged together with the processor of the data processing device, or may be packaged separately from the processor of the data processing device, which is not limited in this application.
[0029] The description of the second, third and fourth aspects in this application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third and fourth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.
[0030] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description.
[0031] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, and usage scenarios of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0033] Figure 1 A first flow chart of a data processing method provided in an embodiment of the present application;
[0034] Figure 2 A second flow chart of a data processing method provided in an embodiment of the present application;
[0035] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0036] Figure 4 It is a block diagram of an electronic device used to implement a data processing method in an embodiment of the present application. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.
[0038] It should be noted that the terms "first", "second", "target" and "original" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0039] It should be noted that the units listed for the total memory resources, initial memory information, and target memory information in the embodiments of the present application are only used to explain the data processing method of the present application. The units of the total memory resources, initial memory information, and target memory information are not limited to the cases listed in the embodiments of the present application. When other units are used, they must first be converted according to the unit conversion rate, and then the sub-process is resource configured according to the data processing method of the present application. Preferably, the units of the total memory resources, initial memory information, and target memory information can be converted to GB.
[0040] Figure 1 This is a first flow chart of a data processing method provided in an embodiment of the present application. This embodiment can be applied to how to optimize resource allocation when Java and C / C++ call each other through JNIEnv. A data processing method provided in this embodiment can be executed by a data processing device provided in an embodiment of the present application. The device can be implemented in software and / or hardware and integrated in an electronic device that executes the method.
[0041] See also Figure 1 The method of this embodiment includes but is not limited to the following steps:
[0042] S110: Determine an initial number of sub-processes to be started based on a first hardware resource of the electronic device, and allocate initial hardware resources to the initial number of sub-processes.
[0043] The first hardware resource includes total memory resources and total processor resources. The total memory resources can be the memory size of the electronic device, such as in GB; the total processor resources can be the number of CPU cores (referred to as core number) of the electronic device. Common CPU core numbers include dual core, quad core, hexa core, octa core, twelve core, etc. The child process is built based on the JVM, and the child process calls the C language library through the JNIEnv technology.
[0044] In an embodiment of the present application, after the main process is started, it actively sends a hardware resource acquisition command, such as a shell command, to the electronic device to obtain the hardware resources of the electronic device. The main process determines the initial number of sub-processes to be started based on the first hardware resource and allocates the initial hardware resources occupied by each sub-process to ensure that the initial sub-process can provide services to the outside in the optimal state under different hardware resources.
[0045] Furthermore, the initial number of sub-processes to be started is determined based on the first hardware resource of the electronic device, including: determining the size relationship between the value of the total memory resources (i.e., the number of GB of memory) and the value of the total processor resources (i.e., the number of CPU cores); and determining the initial number of sub-processes based on the smaller of the size relationship.
[0046] Specifically: when the size relationship is less than or equal to, the initial number of child processes is half of the total memory resource value. For example, for an electronic device with 16 cores and 8GB, the number of GB of memory is <= the number of CPU cores, and the initial number of child processes is = the number of GB of memory / 2, that is, 4 child processes; when the size relationship is greater than, the initial number of child processes is half of the total processor resource value. For example, for an electronic device with 8 cores and 16GB, the number of GB of memory is > the number of CPU cores, and the initial number of child processes is = the number of CPU cores / 2, that is, 4 child processes.
[0047] In a preferred embodiment, in combination with business scenarios, in order to prevent unnecessary waste of resources, the first hardware resource information of the electronic device is matched in a 1:1 ratio according to the values of the total memory resources and the total processor resources, for example: a 4-core, 4GB electronic device, an 8-core, 8GB electronic device.
[0048] The initial hardware resources include initial memory information. Since the child process is built based on the JVM, the initial memory information may be the size of the JVM memory, such as in MB (abbreviated as M).
[0049] Further, allocating initial hardware resources to the initial number of child processes includes: using a preset first calculation method to calculate the value of the total memory resources (i.e., the number of GB of memory) and the initial number of child processes to obtain a first memory value; if the first memory value is greater than a preset fixed value, the preset fixed value is used as the value of the initial memory information of the child process (i.e., the initial memory number of the JVM); if the first memory value is not greater than the preset fixed value, the first memory value is used as the value of the initial memory information of the child process. The present application does not specifically limit the value of the preset fixed value, and the preferred preset fixed value is 1536M.
[0050] Further, the first memory value is calculated by using a preset first calculation method to calculate the value of the total memory resources and the initial number of child processes, including: performing numerical calculation on the value of the total memory resources and the initial number of child processes to obtain the first value; performing numerical calculation on the first value and a preset conversion factor to obtain the first memory value. Preferably, the preset conversion factor here can be 8.
[0051] Specifically, the calculation formula for the initial memory of a single child process JVM is: the initial memory of a single child process JVM = the number of GB of memory / the initial number of child processes / the preset conversion factor. For example, for an electronic device with 16 cores and 8GB, after the calculation using the above calculation formula for the initial memory of a single child process JVM, the initial memory of the JVM is 256M, and 256M is actually allocated to the child process. Another example: for an electronic device with 16 cores and 64GB, after the calculation using the above calculation formula for the initial memory of a single child process JVM, the initial memory of the JVM is 2048M. Since 2048M>1536M, 1536M is actually allocated to the child process.
[0052] The initial hardware resource information also includes initial processor information, which is the number of cores of the child process CPU.
[0053] Specifically, allocating initial hardware resources to an initial number of child processes includes: determining whether the value of total processor resources (i.e., the number of CPU cores) and the initial number of child processes are divisible by each other; if so, taking the quotient after dividing the two as the value of initial processor information of the child process (which can be recorded as JVM-initial number of CPU cores); if not, determining the value of the initial processor information of the child process according to the quotient and remainder after the division.
[0054] Specifically, the calculation formula for the initial number of cores of a single subprocess JVM-CPU is: JVM-CPU initial number of cores = CPU cores / initial number of subprocesses, and the remainder is configured one by one in the order of the initial number of subprocesses. For example: for an electronic device with 8 cores, 8GB, and an initial number of subprocesses of 4, then each subprocess is configured with 2 cores. Another example: for an electronic device with 10 cores, 8GB, and an initial number of subprocesses of 4, then the first 2 subprocesses are configured with 3 cores, and the last 2 subprocesses are configured with 2 core processor resources.
[0055] S120. Monitor the working status of the initial number of sub-processes processing data requests to obtain monitoring results, adjust the initial number of sub-processes based on the monitoring results to obtain a target number of sub-processes, and / or reallocate target hardware resources to the target number of sub-processes based on the monitoring results.
[0056] In an embodiment of the present application, the electronic device is connected to the external controller for communication, the main process provides a unified entrance for receiving external business processing requests, the external controller sends a data request to the electronic device through the business processing request entrance, and the electronic device receives and responds to the data request. The electronic device distributes multiple data requests to the opened sub-processes, so that the initial number of sub-processes can jointly process multiple data requests based on the corresponding initial hardware resources, so that multiple sub-processes can uniformly provide external services, and JNIEnv channels can be established through multiple sub-processes to achieve the purpose of breaking through the performance bottleneck of the JNIEnv bridge channel.
[0057] Furthermore, the working status of the initial number of sub-processes in processing data requests is monitored to obtain monitoring results, including: in the process of the sub-processes processing data requests, the concurrency status of the data requests is determined by the response time, query rate per second and number of transactions per second to the data requests, and the data calculation amount required for a single data request is determined; if the concurrency status is high concurrency and the data calculation amount required for a single data request is small, then the monitoring result is determined to be that the initial number of sub-processes are in the first state; if the concurrency status is low concurrency and the data calculation amount required for a single data request is large, then the monitoring result is determined to be that the initial number of sub-processes are in the second state; if the concurrency status is high concurrency and the data calculation amount required for a single data request is large, then the monitoring result is determined to be that the initial number of sub-processes are in the third state; if the concurrency status is low concurrency and the data calculation amount required for a single data request is small, then the monitoring result is determined to be that the initial number of sub-processes are in the fourth state.
[0058] When the monitoring result shows that the initial number of subprocesses is in the first state, it is necessary to open as many subprocesses as possible (i.e., establish multiple JNIEnv channels) to process concurrent data requests. Therefore, the initial number of subprocesses is adjusted to obtain the target number of subprocesses, and the initial hardware resources of the subprocesses are adjusted to obtain the target hardware resources. The process of determining the target number and the specific allocation process of the target hardware resources will be described below. Figure 2 The corresponding embodiments are explained in detail.
[0059] When the monitoring result shows that the initial number of sub-processes is in the second state, the sub-process needs to increase the memory of a single sub-process as much as possible based on the initial processor information (i.e., the initial number of JVM-CPU cores) to improve the storage of big data, thereby improving performance. Therefore, the initial number is used as the target number, and the target hardware resources are re-allocated to the target number of sub-processes. The specific allocation process of the target hardware resources is: the value of the total memory resources and the target number of sub-processes are calculated using the preset second calculation method to obtain a third memory value; the third memory value is verified, and the third memory value after verification is used as the value of the target memory information of the sub-process (i.e., the JVM target memory number). Specifically, the calculation formula for the target memory number of a single sub-process JVM is: single sub-process JVM target memory number = memory GB number / target number of sub-processes / preset conversion factor, preferably, the preset conversion factor here can be 2. After the calculation formula for the target memory number of a single sub-process JVM is used, if the JVM target memory number>1536M, 1536M is actually allocated to the sub-process at this time.
[0060] Optionally, the initial number of sub-processes may be adjusted to obtain a target number of sub-processes.
[0061] When the monitoring result shows that the initial number of subprocesses is in the third state or the fourth state, there is no need to adjust the number of subprocesses and the hardware resources of the subprocesses. That is, the initial number is used as the target number, and the initial hardware resources are used as the target hardware resources.
[0062] S130 . Continue to process the data request through a target number of child processes based on corresponding target hardware resources.
[0063] In the embodiment of the present application, the initial number of sub-processes is adjusted based on the monitoring result to obtain the target number of sub-processes in step S120, and / or the target number of sub-processes are re-allocated with the target hardware resources based on the monitoring result, thereby determining the optimal number of sub-processes to be started and formulating the optimal resource configuration for the hardware resources of the sub-processes. Then, the data request is continued to be processed by the target number of sub-processes based on the corresponding target hardware resources.
[0064] The technical solution provided in this embodiment determines the initial number of subprocesses to be started based on the first hardware resource of the electronic device, and allocates initial hardware resources to the initial number of subprocesses; monitors the working status of the initial number of subprocesses in processing data requests to obtain monitoring results, adjusts the initial number of subprocesses based on the monitoring results to obtain the target number of subprocesses, and / or re-allocates target hardware resources to the target number of subprocesses based on the monitoring results; and continues to process data requests through the target number of subprocesses based on the corresponding target hardware resources. This application determines the initial number of subprocesses to be started based on the hardware resource information of the electronic device, and allocates initial hardware resources to each subprocess when it is started, and establishes JNIEnv channels respectively through multiple subprocesses to achieve the purpose of breaking through the performance bottleneck of the JNIEnv bridge channel; and in the process of subprocesses processing data requests, dynamically adjusts the number of subprocesses and the hardware resources of each subprocess according to the monitoring results of the subprocesses, so as to reasonably determine the optimal number of subprocesses to be started for the electronic device and formulate the optimal resource configuration for the hardware resources of the subprocesses.
[0065] The data processing method provided by the embodiment of the present invention is further described below. Figure 2 A second flow chart of a data processing method provided in an embodiment of the present application. The embodiment of the present application is optimized on the basis of the above embodiment, and the specific optimization is as follows: when the monitoring result is that the initial number of sub-processes are in the first state, the process of adjusting the number of sub-processes and adjusting the hardware resources of the sub-processes is explained in detail.
[0066] See also Figure 2 The method of this embodiment includes but is not limited to the following steps:
[0067] S210: Determine the value of the total processor resources as the target quantity.
[0068] In the embodiment of the present application, the main process monitors the working status of each sub-process in processing data requests during operation, that is, the business operation status. If the monitoring result shows that the sub-process is in a high concurrency state and the amount of data calculation required for a single data request is small, then it is necessary to increase the number of opened sub-processes and establish more JNIEnv channels to cope with concurrent data requests. Therefore, the value of the total processor resources in the electronic device is determined as the number of opened sub-processes.
[0069] S220: Determine the number of newly started sub-processes based on the difference between the target number and the initial number to obtain the target number of sub-processes.
[0070] Among them, the target quantity is greater than the initial quantity.
[0071] In an embodiment of the present application, the difference between the target number and the initial number is the number of sub-processes that need to be reopened, and then the main process reopens the said number of sub-processes, so that the number of sub-processes currently in the open state is the target number.
[0072] S230, determining a memory adjustment parameter based on the value of the total processor resources, calculating the value of the total memory resources and the memory adjustment parameter to obtain a second memory value; correcting the second memory value, and using the corrected second memory value as the value of the target memory information of the child process.
[0073] In the embodiment of the present application, target hardware resources are re-allocated to the target number of sub-processes based on the value of the total processor resources and the target number. The target hardware resources include target memory information and target processor information. This step is the first step of the specific allocation process of the target hardware resources, namely, allocating the target memory information.
[0074] Specifically, the value of the total processor resources is compared with the preset experience value. If the value of the total processor resources is less than or equal to the preset experience value, the preset multiple of the preset experience value is used as the memory adjustment parameter; if the value of the total processor resources is greater than the preset experience value, the preset multiple of the value of the total processor resources is used as the memory adjustment parameter. Among them, the preferred preset experience value is 8; the preferred preset multiple is 2. Then, the value of the total memory resources is divided by the memory adjustment parameter, and the quotient after the division is used as the second memory value.
[0075] Specifically, the second memory value is corrected, and the corrected second memory value is used as the value of the target memory information of the child process, including: if the second memory value exceeds the first value, the first value is used as the value of the target memory information, that is, only the target memory information is corrected. If the second memory value does not exceed the second value, the second value is used as the value of the target memory information; at this time, it is also necessary to correct the target quantity according to the value of the target memory information to obtain the corrected target quantity, and the specific correction formula is: target quantity of the child process = the value of the total memory resources / 2*8; in addition, since the target value has been corrected, it is necessary to determine the target processor information for the child process based on the value of the total processor resources and the corrected target quantity (that is, re-execute the following step S240), that is, it is necessary to correct the target quantity, target memory information and target processor information.
[0076] Among them, the first value is greater than the second value, the first value is the maximum limit of the child process JVM memory number, and the second value is the minimum limit of the child process JVM memory number. The preferred first value can be 1536M, and the preferred second value can be 128M.
[0077] S240: Determine target processor information for the child process based on the value of the total processor resources and the target quantity.
[0078] In the embodiment of the present application, this step is the second step of the specific allocation process of the target hardware resources, that is, allocating the target processor information. Since the value of the total processor resources is determined as the target number in step S210, that is, the number of child processes has changed, so for the above Figure 1 The formula for calculating the initial number of cores of a single subprocess JVM-CPU in step S110 of the embodiment becomes: JVM-CPU target number of cores = CPU number of cores / subprocess target number, and the remainder is configured one by one in the order of the target number of subprocesses.
[0079] The technical solution provided by this embodiment is to determine the target number by taking the value of the total processor resources as the target number; determine the number of newly started sub-processes based on the difference between the target number and the initial number to obtain the target number of sub-processes; determine the memory adjustment parameter based on the value of the total processor resources, calculate the value of the total memory resources and the memory adjustment parameter to obtain the second memory value; correct the second memory value, and use the corrected second memory value as the value of the target memory information of the sub-process; determine the target processor information for the sub-process based on the value of the total processor resources and the target number. This application determines the initial number of sub-processes to be started according to the hardware resource information of the electronic device, and allocates initial hardware resources to each sub-process when starting, and establishes JNIEnv channels respectively through multiple sub-processes to achieve the purpose of breaking through the performance bottleneck of the JNIEnv bridge channel; and in the process of processing data requests by the sub-process, the sub-process is in a high concurrency state and the data operation state is a small amount of operation, and dynamically adjusts the number of sub-processes and the hardware resources of each sub-process, so as to reasonably determine the optimal number of sub-processes to be started for the electronic device and formulate the optimal resource configuration for the hardware resources of the sub-process.
[0080] Figure 3 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application, the device is integrated into an electronic device, such as Figure 3 As shown, the device 300 may include:
[0081] A resource allocation module 310, configured to determine an initial number of sub-processes to be started based on a first hardware resource of the electronic device, and allocate initial hardware resources to the initial number of sub-processes;
[0082] The resource adjustment module 320 is used to monitor the working status of the initial number of sub-processes processing data requests to obtain a monitoring result, adjust the initial number of sub-processes based on the monitoring result to obtain a target number of sub-processes, and / or reallocate target hardware resources to the target number of sub-processes based on the monitoring result;
[0083] The data processing module 330 is used to continue processing the data request through the target number of sub-processes based on the corresponding target hardware resources.
[0084] Furthermore, the resource allocation module 310 may include a sub-process number determination unit, a memory information determination unit, and a processor information determination unit;
[0085] Optionally, the first hardware resources include total memory resources and total processor resources;
[0086] The sub-process quantity determination unit can be used to determine the size relationship between the value of the total memory resources and the value of the total processor resources; and determine the initial number of the sub-processes based on the smaller of the size relationship.
[0087] Optionally, the initial hardware resources include initial memory information;
[0088] The memory information determination unit can be used to calculate the value of the total memory resources and the initial number of the sub-processes using a preset first calculation method to obtain a first memory value; if the first memory value is greater than a preset fixed value, the preset fixed value is used as the value of the initial memory information of the sub-process; if the first memory value is not greater than the preset fixed value, the first memory value is used as the value of the initial memory information of the sub-process.
[0089] Furthermore, the memory information determination unit may be specifically used to: perform numerical calculation on the value of the total memory resource and the initial number of the sub-processes to obtain a first value; and perform numerical calculation on the first value and a preset conversion factor to obtain the first memory value.
[0090] Optionally, the initial hardware resource information also includes initial processor information;
[0091] The processor information determination unit can be used to determine whether the value of the total processor resources and the initial number of the child processes are divisible by each other; if so, the quotient after the division is used as the value of the initial processor information of the child process; if not, the value of the initial processor information of the child process is determined based on the quotient and remainder after the division.
[0092] Furthermore, the resource adjustment module 320 may include a status monitoring unit, a first resource adjustment unit, and a second resource adjustment unit;
[0093] When the monitoring result is that the initial number of sub-processes are in the first state, the first resource adjustment unit can be used to: determine the value of the total processor resources as the target number; determine the number of newly opened sub-processes based on the difference between the target number and the initial number to obtain the target number of sub-processes, and the target number is greater than the initial number; and reallocate target hardware resources to the target number of sub-processes based on the value of the total processor resources and the target number.
[0094] Optionally, the target hardware resources include target memory information and target processor information;
[0095] Furthermore, the above-mentioned first resource adjustment unit can be specifically used to: determine a memory adjustment parameter based on the value of the total processor resources, and calculate the value of the total memory resources and the memory adjustment parameter to obtain a second memory value; correct the second memory value, and use the corrected second memory value as the value of the target memory information of the child process; determine the target processor information for the child process based on the value of the total processor resources and the target number.
[0096] Furthermore, the above-mentioned first resource adjustment unit can also be specifically used for: if the second memory value exceeds the first value, then the first value is used as the value of the target memory information; if the second memory value does not exceed the second value, then the second value is used as the value of the target memory information, and the target quantity is corrected according to the value of the target memory information to obtain the corrected target quantity, and the operation of determining the target processor information for the sub-process based on the value of the total processor resources and the target quantity is re-executed, and the first value is greater than the second value.
[0097] When the monitoring result is that the initial number of sub-processes are in the second state, the second resource adjustment unit can be used to: take the initial number as the target number, and use a preset second calculation method to calculate the value of the total memory resources and the target number of sub-processes to obtain a third memory value; verify the third memory value, and use the verified third memory value as the value of the target memory information of the sub-process.
[0098] The status monitoring unit can be used to determine the concurrency status of the data request and the data calculation status required for the data request during the process of the sub-process processing the data request; if the concurrency status is high concurrency and the data calculation status is small in amount of calculation, then the monitoring result is determined to be that the initial number of sub-processes are in the first state; if the concurrency status is low concurrency and the data calculation status is large in amount of calculation, then the monitoring result is determined to be that the initial number of sub-processes are in the second state.
[0099] The data processing device provided in this embodiment can be applied to the data processing method provided in any of the above embodiments, and has corresponding functions and beneficial effects.
[0100] Figure 4 It is a block diagram of an electronic device for implementing a data processing method of an embodiment of the present application. Electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0101] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0102] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0103] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a data processing method.
[0104] In some embodiments, the data processing method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method in any other appropriate manner (e.g., by means of firmware).
[0105] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0106] The computer programs for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0107] In the context of the present application, a computer readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device or equipment. A computer readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium may be a machine readable signal medium. A more specific example of a machine readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0108] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0109] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0110] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0111] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of this application can be achieved, and this document is not limited here.
[0112] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A data processing method, characterized in that: Applied to the main process of the electronic device, the method comprises: Determine an initial number of sub-processes to be started based on a first hardware resource of the electronic device, and allocate initial hardware resources to the initial number of sub-processes; the first hardware resource includes a total processor resource; Monitoring the working status of the initial number of sub-processes in processing data requests to obtain a monitoring result, adjusting the initial number of sub-processes based on the monitoring result to obtain a target number of sub-processes, and / or reallocating target hardware resources to the target number of sub-processes based on the monitoring result; Continue to process the data request based on the corresponding target hardware resources by using the target number of sub-processes; The monitoring of the working status of the initial number of sub-processes in processing data requests to obtain the monitoring result includes: in the process of the sub-processes processing data requests, determining the concurrency status of the data requests and determining the data computing status required for the data requests; if the concurrency status is high concurrency and the data computing status is small computing amount, determining that the monitoring result is that the initial number of sub-processes are in the first state; if the concurrency status is low concurrency and the data computing status is large computing amount, determining that the monitoring result is that the initial number of sub-processes are in the second state; When the monitoring result shows that the initial number of sub-processes is in the first state, the initial number of sub-processes is adjusted based on the monitoring result to obtain a target number of sub-processes, and the initial hardware resources of the sub-processes are adjusted based on the monitoring result to obtain target hardware resources, including: determining the value of the total processor resources as the target number; determining the number of newly opened sub-processes based on the difference between the target number and the initial number to obtain the target number of sub-processes, wherein the target number is greater than the initial number; and reallocating target hardware resources to the target number of sub-processes based on the value of the total processor resources and the target number.
2. The data processing method according to claim 1, characterized in that: The first hardware resource also includes total memory resources; and determining the initial number of sub-processes to be started based on the first hardware resource of the electronic device includes: Determine the magnitude relationship between the value of the total memory resources and the value of the total processor resources; The initial number of the child processes is determined based on the smaller one of the size relationships.
3. The data processing method according to claim 2, characterized in that: The initial hardware resources include initial memory information; the allocating initial hardware resources to the initial number of sub-processes includes: Calculating the value of the total memory resource and the initial number of the child processes by using a preset first calculation method to obtain a first memory value; If the first memory value is greater than a preset fixed value, the preset fixed value is used as the value of the initial memory information of the child process; If the first memory value is not greater than a preset fixed value, the first memory value is used as the value of the initial memory information of the child process.
4. The data processing method according to claim 3, characterized in that: The step of calculating the total memory resource value and the initial number of the child processes by using a preset first calculation method to obtain the first memory value includes: Performing numerical calculation on the value of the total memory resource and the initial number of the child processes to obtain a first value; The first memory value is obtained by performing numerical calculation on the first value and a preset conversion factor.
5. The data processing method according to claim 2, characterized in that: The initial hardware resource information also includes initial processor information; the allocating initial hardware resources to the initial number of sub-processes includes: Determining whether the value of the total processor resources is divisible by the initial number of the child processes; If it is divisible, the quotient after the division is used as the value of the initial processor information of the child process; If it is not divisible, the value of the initial processor information of the child process is determined according to the quotient and the remainder after the division.
6. The data processing method according to claim 2, characterized in that: The target hardware resources include target memory information and target processor information; The reallocating target hardware resources to the target number of sub-processes based on the value of the total processor resources and the target number includes: Determining a memory adjustment parameter based on the value of the total processor resources, and calculating the value of the total memory resources and the memory adjustment parameter to obtain a second memory value; Correcting the second memory value, and using the corrected second memory value as the value of the target memory information of the child process; Target processor information is determined for the child process based on the value of the total processor resources and the target quantity.
7. The data processing method according to claim 6, characterized in that: Correcting the second memory value and using the corrected second memory value as the value of the target memory information of the child process includes: If the second memory value exceeds the first value, the first value is used as the value of the target memory information; If the second memory value does not exceed the second numerical value, the second numerical value is used as the value of the target memory information, and the target quantity is corrected according to the value of the target memory information to obtain the corrected target quantity, and the operation of determining the target processor information for the child process based on the value of the total processor resources and the target quantity is re-executed, and the first numerical value is greater than the second numerical value.
8. The data processing method according to claim 2, characterized in that: When the monitoring result shows that the initial number of sub-processes is in the second state, taking the initial number as the target number; Re-allocating target hardware resources to the target number of sub-processes based on the monitoring result includes: The third memory value is obtained by calculating the value of the total memory resource and the target number of the sub-processes using a preset second calculation method; The third memory value is verified, and the verified third memory value is used as the value of the target memory information of the child process.
9. A data processing device, characterized in that: Integrated in an electronic device, the device comprises: a resource allocation module, configured to determine an initial number of sub-processes to be started based on a first hardware resource of the electronic device, and allocate initial hardware resources to the initial number of sub-processes; the first hardware resource includes a total processor resource; a resource adjustment module, configured to monitor the working status of the initial number of sub-processes in processing data requests to obtain a monitoring result, adjust the initial number of sub-processes based on the monitoring result to obtain a target number of sub-processes, and / or reallocate target hardware resources to the target number of sub-processes based on the monitoring result; A data processing module, configured to continue processing the data request through the target number of sub-processes based on corresponding target hardware resources; The resource adjustment module includes a state monitoring unit, and the state monitoring unit is used to determine the concurrency status of the data request and the data computing status required for the data request during the process of the sub-process processing the data request; if the concurrency status is high concurrency and the data computing status is small computing amount, then determine that the monitoring result is that the initial number of sub-processes are in the first state; if the concurrency status is low concurrency and the data computing status is large computing amount, then determine that the monitoring result is that the initial number of sub-processes are in the second state; The resource adjustment module includes a first resource adjustment unit, which is used to determine the value of the total processor resources as the target number when the monitoring result is that the initial number of sub-processes is in a first state; determine the number of newly opened sub-processes based on the difference between the target number and the initial number to obtain the target number of sub-processes, and the target number is greater than the initial number; and reallocate target hardware resources to the target number of sub-processes based on the value of the total processor resources and the target number.
