Data processing method and device and storage medium

Abstract

The invention relates to the field of big data, and discloses a data processing method and device and a storage medium, and the method comprises the steps: determining a data concurrent transmission parameter according to the resource condition of a test press; concurrently sending to-be-sent data to a big data server according to the data concurrency sending parameter; acquiring data processing parameters during transmission of the to-be-transmitted data in real time, and adjusting the data concurrent transmission parameters according to the data processing parameters and a preset data processing parameter warning range until the data processing parameters are within the preset data processing parameter warning range; wherein the preset data processing parameter warning range is used for indicating high data processing efficiency, the data concurrent sending parameters can be adaptively adjusted according to the current resource condition of the test press, namely, the concurrent sending condition of the data flow is adaptively adjusted according to the test environment, so that the appropriate high concurrent efficiency is achieved, and the test efficiency is improved. The big data testing method and device can flexibly cope with big data testing of various high-concurrency pressure requirements.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of big data, and in particular to a data processing method, device and storage medium. Background Art

[0002] With the rapid development of cloud computing, the Internet and the Internet of Things, and the widespread application of big data technology, the amount of big data has grown exponentially in recent years, and the importance and complexity of testing big data have also increased. The amount of big data is very large, and big data technology has a huge data set, complex and diverse data types and structures, and the speed and change rate of data are also very high. In addition, big data relies on a complex cluster environment.

[0003] However, traditional data testing cannot effectively handle large, diverse, high-speed, distributed and complex big data. Therefore, traditional data testing is only suitable for partial data testing and is not suitable for big data environments. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a data processing method, device and storage medium to flexibly respond to various big data tests with high concurrency pressure requirements.

[0005] To solve the above technical problems, an embodiment of the present application provides a data processing method, including:

[0006] Determine the data concurrent sending parameters according to the test press resource situation;

[0007] Concurrently sending the data to be sent to the big data server according to the data concurrent sending parameter;

[0008] Acquire in real time the data processing parameters for the data to be sent during transmission, and adjust the data concurrency transmission parameters according to the data processing parameters and a preset data processing parameter warning range until the data processing parameters are within the preset data processing parameter warning range; wherein the preset data processing parameter warning range is used to indicate high data processing efficiency.

[0009] The embodiment of the present application further provides a data processing method, wherein the method for determining the data to be sent includes:

[0010] The data to be sent is generated by presetting a big data model, a presetting data structure and a presetting data volume.

[0011] The embodiment of the present application further provides a data processing method, wherein the data processing parameters include: a data accumulation value; the real-time acquisition of the data processing parameters when transmitting the data to be sent includes:

[0012] Real-time acquisition of the amount of data sent to the big data server per unit time and the amount of data consumed and processed by the big data server per unit time;

[0013] The data accumulation value is determined according to the amount of data sent to the big data server within the unit time and the amount of data consumed and processed by the big data server within the unit time.

[0014] The embodiment of the present application further provides a data processing method, wherein the preset data processing parameter warning range includes: a data accumulation value warning range; and adjusting the data concurrent sending parameter according to the data processing parameter and the preset data processing parameter warning range until the data processing parameter is within the preset data processing parameter warning range includes:

[0015] According to the data accumulation value and the data accumulation value warning range, the data concurrent sending parameter is adjusted until the data accumulation value is within the data accumulation value warning range.

[0016] The embodiment of the present application further provides a data processing method, wherein the data processing parameters further include: performance indicators; the real-time acquisition of the data processing parameters when the data is to be sent further includes:

[0017] Obtaining services of the big data server in real time;

[0018] According to the service, the performance indicator is determined.

[0019] The embodiment of the present application further provides a data processing method, wherein the preset data processing parameter warning range further includes: a performance indicator warning range; and adjusting the data concurrent sending parameter according to the data processing parameter and the preset data processing parameter warning range until the data processing parameter is within the preset data processing parameter warning range includes:

[0020] The data concurrent sending parameter is adjusted according to the data accumulation value, the data accumulation value warning range, the performance indicator, and the performance indicator warning range until the data accumulation value is within the data accumulation value warning range and the performance indicator is within the performance indicator warning range.

[0021] An embodiment of the present application further provides a data processing method, wherein the method of adjusting the data concurrent sending parameter according to the data accumulation value, the data accumulation value warning range, the performance indicator, and the performance indicator warning range until the data accumulation value is within the data accumulation value warning range and the performance indicator is within the performance indicator warning range comprises:

[0022] According to the data accumulation value and the data accumulation value warning range, adjusting the data concurrent sending parameter until the data accumulation value is within the data accumulation value warning range;

[0023] When the data to be sent is concurrently sent to the big data server by using the adjusted data concurrent sending parameter, if the performance indicator is within the performance indicator warning range, the adjusted data concurrent sending parameter is not adjusted.

[0024] The embodiment of the present application also provides a data processing method, the method further comprising:

[0025] When sending the data to be sent to the big data server by using the adjusted concurrent data sending parameter, if the performance indicator is not within the performance indicator warning range, adjusting the adjusted concurrent data sending parameter until the performance indicator is within the performance indicator warning range;

[0026] When the data to be sent is concurrently sent to the big data server using the adjusted data concurrent sending parameters, if the data accumulation value is within the data accumulation value warning range, the adjusted data concurrent sending parameters are not adjusted.

[0027] An embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the data processing methods described above.

[0028] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program implements any of the above-mentioned data processing methods when executed by a processor.

[0029] In this application, the data concurrent sending parameters can be adaptively adjusted according to the current resource situation of the test press, that is, according to the test environment, the concurrent sending of the data stream can be adaptively adjusted to achieve appropriate high concurrency efficiency. This method can flexibly respond to various big data tests with high concurrency pressure requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0031] Figure 1 It is a structural diagram of a data processing system provided in an embodiment of the present application;

[0032] Figure 2is a flow chart of a first data processing method provided in an embodiment of the present application;

[0033] Figure 3 is a flow chart of a second data processing method provided in an embodiment of the present application;

[0034] Figure 4 is a flowchart of a third data processing method provided in an embodiment of the present application;

[0035] Figure 5 is a flowchart of a fourth data processing method provided in an embodiment of the present application;

[0036] Figure 6 is a flowchart of a fifth data processing method provided in an embodiment of the present application;

[0037] Figure 7 is a flowchart of a sixth data processing method provided in an embodiment of the present application;

[0038] Figure 8 is a flow chart of a seventh data processing method provided in an embodiment of the present application;

[0039] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0041] The embodiment of the present application relates to a data processing system. Figure 1 is a structural diagram of a data processing system provided in an embodiment of the present application, such as Figure 1 As shown, the data processing system includes: a test press and a big data server.

[0042] Among them, the communication connection between the press and the data server is tested.

[0043] When performing concurrent stress testing, the test stress machine can send data to the big data server concurrently to evaluate the performance of the big data server under high load. This type of stress testing is usually called load testing or performance testing. Its main purpose is to simulate the behavior of actual users or devices in order to measure key performance indicators such as system stability, response time, throughput, and resource utilization under stress.

[0044] Based on the data processing system provided in the above embodiment, the embodiment of the present application further provides a data processing method. Figure 2 is a flow chart of a first data processing method provided in an embodiment of the present application, such as Figure 2 As shown, the data processing method can be applied to the test press in the above data processing system, and specifically includes the following steps.

[0045] Step 201, determining data concurrent transmission parameters according to the resource conditions of the test press.

[0046] The concurrent data transmission parameters may also be referred to as concurrent startup parameters, and the concurrent data transmission parameters may include: the number of processes, the number of threads, the amount of data sent by each thread, the concurrent transmission time interval, and other parameters.

[0047] Before sending data, set the data concurrent sending parameters according to the resource situation of the local machine (test stress machine), including the number of processes, number of threads, amount of data sent by each thread, concurrent sending time interval, etc. This setting reserves the space for improving and reducing concurrent efficiency, which can be used to adjust the concurrent efficiency as needed later.

[0048] The number of processes refers to the number of test program instances running simultaneously. Usually, a process can control multiple threads or virtual users. The number of processes should be selected based on the CPU (Central Processing Unit) and memory resources of the test machine, as well as the size of the target load. It is usually recommended to start the test with a few to dozens of processes. The specific number needs to be determined based on the performance of the test machine and the requirements of the test load.

[0049] The number of threads indicates the number of concurrently running threads in each process. Each thread can simulate a user or device sending a data request. The setting of the number of threads depends on the support capabilities of the stress test tool and the number of CPU cores and memory of the test stress machine. Generally, the more threads there are, the higher the concurrency of the test, but it will also increase the resource consumption of the stress test machine.

[0050] When setting the data volume for each thread, you need to ensure that the network bandwidth of the test press and the processing power of the big data server can withstand such a load to avoid network bottlenecks or server overload.

[0051] The time interval should be selected taking into account the response time of the system and the test objectives, and is usually set to a small interval in milliseconds to ensure that the test can generate high load in a short period of time.

[0052] By properly setting the data concurrency sending parameters and adjusting and optimizing them according to the actual situation, you can effectively perform stress testing on the big data server and evaluate its performance and stability under high load conditions.

[0053] Based on the performance data collected during the test, parameters such as the number of processes, number of threads, amount of data sent by each thread, and concurrent sending time interval can be dynamically adjusted to optimize the authenticity and effect of the test. During the test, the resource utilization of the test press and big data server is continuously monitored and adjusted in time to avoid resource saturation or performance degradation.

[0054] Step 202: according to the data concurrent sending parameters, concurrently send the data to be sent to the big data server.

[0055] Through the data concurrent sending parameters, the concurrent sending process of the test press to concurrently send the data to be sent to the big data server is controlled.

[0056] Step 203, obtaining data processing parameters for the data to be sent in real time, and adjusting data concurrency transmission parameters according to the data processing parameters and a preset data processing parameter warning range until the data processing parameters are within the preset data processing parameter warning range.

[0057] The data processing parameters may be partial parameters corresponding to the test press, in which case the data processing parameters are obtained from the test press. The data processing parameters may also be partial parameters corresponding to the big data server, in which case the data processing parameters are obtained from the big data server.

[0058] The data processing parameter is used to indicate the data processing efficiency of the data processing system for the data to be sent, and the preset data processing parameter warning range is used to indicate the high efficiency of the data processing for the data to be sent.

[0059] By real-time monitoring of data processing parameters, the data processing parameters can be adjusted according to the preset data processing parameter warning range, so that the data to be sent is sent concurrently to the big data server according to the data concurrent sending parameters, and the data processing parameters during the transmission of the data to be sent are within the preset data processing parameter warning range, so as to achieve the most appropriate and high concurrent data processing efficiency of the data to be sent.

[0060] In an embodiment of the present application, the data concurrent sending parameters can be adaptively adjusted according to the current resource situation of the test press, that is, according to the test environment, the concurrent sending of the data stream can be adaptively adjusted to achieve a suitable high concurrency efficiency. This method can flexibly respond to various big data tests with high concurrency pressure requirements.

[0061] In the above Figure 2 Based on the data processing method shown, an embodiment of the present application further provides another data processing method. The method for determining the data to be sent in the above step 202 specifically includes the following steps.

[0062] The data to be sent is generated by presetting a big data model, a presetting data structure and a presetting data volume.

[0063] In an embodiment of the present application, a default big data data model and data structure are provided, and user-defined big data data models and data structures are also supported. When the user does not customize the big data model, the default big data data model and data structure are used to generate the data to be sent.

[0064] A preset big data data model, a preset data structure and a preset data volume are used to generate the data to be sent. In terms of the mode of generating the data to be sent, according to the preset data volume and the complexity of the big data data model, a multi-threaded or multi-process mode can be used to improve the efficiency of generating big data (data to be sent).

[0065] The preset big data model specifies the logical structure of the data, including data types, field names, and constraints. These models define how the data is organized and the meaning of each field. The preset data structure defines the physical structure and organization of data when it is stored and processed. It can include data tables, key-value pairs, documents, and other forms, as well as the association between data. The preset data volume represents the number of specific data entries or records generated according to the defined preset big data model and preset data structure. The larger the data volume, the more data is generated. This is usually used to simulate big data scenarios in real systems.

[0066] The generated data to be sent conforms to the predefined preset big data model and preset data structure, ensuring that the type, value range and constraints of each field are consistent with the model. The amount of data generated to be sent matches the preset amount of data required for testing. If it is necessary to test the performance and stability of the system under large amounts of data, the amount of data generated should be large enough to reflect these situations.

[0067] In an embodiment of the present application, the required data to be sent is generated by presetting a big data data model, a preset data structure, and a preset data volume. The data to be sent is big data to meet the big data input of the big data test.

[0068] In the above Figure 2 Based on the data processing method shown in FIG. 1 , the embodiment of the present application further provides another data processing method. Figure 3 is a flow chart of a second data processing method provided in an embodiment of the present application, such as Figure 3 As shown, the real-time acquisition of data processing parameters for data transmission to be sent in the above step 203 specifically includes the following steps.

[0069] Step 301, obtaining in real time the amount of data sent to the big data server per unit time and the amount of data consumed and processed by the big data server per unit time.

[0070] During the data transmission process, the amount of concurrent data can be adjusted as needed. The actual resource usage of the remote machine (big data server) can be monitored in real time, and the data concurrent transmission parameters can be adjusted in a timely manner to achieve the most appropriate concurrent transmission efficiency.

[0071] The data processing parameters include: data accumulation value, which is the data accumulation value in the big data server. Specifically, if the data sending rate of the test press is higher than the consumption processing rate of the big data server, the data in the big data server will accumulate. This accumulation may cause increased latency or increased system resource pressure, thereby affecting the performance and stability of the overall system.

[0072] The data accumulation value is related to the amount of data sent by the test press to the big data server and the amount of data consumed and processed by the big data server. Therefore, it is necessary to first obtain in real time the amount of data sent to the big data server per unit time and the amount of data consumed and processed by the big data server per unit time.

[0073] Step 302, determining the data accumulation value according to the amount of data sent to the big data server per unit time and the amount of data consumed and processed by the big data server per unit time.

[0074] The difference between the amount of data sent to the Kafka system in the big data server per unit time and the amount of data consumed and processed by the Kafka system per unit time is the data accumulation value. This difference reflects the data processing capability of the big data server.

[0075] Correspondingly, the preset data processing parameter warning range in the above step 203 includes: data accumulation value warning range. Define the data accumulation value warning range in the Kafka data stream processing system, and the data accumulation value cannot exceed the data accumulation value warning range. When the data accumulation value exceeds the data accumulation value warning range, it is considered that data accumulation occurs and concurrency needs to be controlled. If it does not exceed, it is considered safe and concurrent data sending can continue. If the user does not customize the data accumulation value warning range, the default data accumulation value warning range provided by the system is used.

[0076] In step 203, according to the data processing parameters and the preset data processing parameter warning range, the data concurrency sending parameters are adjusted until the data processing parameters are within the preset data processing parameter warning range, which specifically includes the following steps.

[0077] According to the data accumulation value and the data accumulation value warning range, adjust the data concurrent sending parameters until the data accumulation value is within the data accumulation value warning range.

[0078] If the data accumulation value exceeds the highest point of the data accumulation value warning range, it means that the amount of concurrent data of big data is too large and has exceeded the processing capacity of the remote machine (big data server). It is necessary to adjust and reduce the amount of concurrent data sent by big data, that is, it is necessary to adjust the data concurrent sending parameters to reduce the amount of data to be sent.

[0079] If the data accumulation value does not reach the lowest point of the data accumulation value warning range, it means that the remote machine (big data server) still has sufficient big data processing performance resources. If you need to achieve maximum concurrency efficiency, you can also appropriately increase the amount of concurrent data to increase the big data test pressure. You need to adjust and increase the amount of big data concurrently sent data, that is, you need to adjust the data concurrent sending parameters to increase the amount of data to be sent.

[0080] If the data accumulation value is within the data accumulation value warning range, it means that the big data processing efficiency of the remote machine (big data server) has reached the maximum value, and there is no need to adjust the concurrent data volume of big data to maintain the concurrent sending efficiency.

[0081] In the embodiment of the present application, the data accumulation of the data processing system is appropriately achieved through the data accumulation value and the data accumulation value warning range, thereby realizing the appropriate maximum concurrency efficiency of the data processing system.

[0082] In the above Figure 3 Based on the data processing method shown in FIG. 1 , the embodiment of the present application further provides another data processing method. Figure 4 is a flow chart of a third data processing method provided in an embodiment of the present application, such as Figure 4 As shown, the real-time acquisition of data processing parameters for data transmission to be sent in the above step 203 specifically includes the following steps.

[0083] Step 401, obtaining services of a big data server in real time.

[0084] The data processing parameters include: performance indicators, which are performance indicators in the big data server.

[0085] The performance indicators are related to the services of the big data server, so it is necessary to obtain the services of the big data server in real time.

[0086] Step 402, determining performance indicators based on the service.

[0087] In the embodiment of the present application, different types of services have different performance indicators.

[0088] According to the service nature of different services of the remote machine (big data server), set performance indicator reference items and set the performance indicator warning range. For example, the machine memory usage rate is between 80% and 85%, which is the warning range. Within this range, it is considered that the appropriate pressure has been reached and concurrency testing can continue at this pressure. If it exceeds the highest point of 85%, it is considered that the machine system performance bottleneck has been reached. If the pressure continues to be increased, it may cause damage to the machine.

[0089] Correspondingly, the preset data processing parameter warning range in the above step 203 includes: a performance indicator warning range. A performance indicator warning range is defined, and the performance indicator cannot exceed the performance indicator warning range. If the user does not define the performance indicator warning range, the default performance indicator warning range provided by the system is used.

[0090] In step 203, according to the data processing parameters and the preset data processing parameter warning range, the data concurrency sending parameters are adjusted until the data processing parameters are within the preset data processing parameter warning range, which specifically includes the following steps.

[0091] According to the performance indicator and the warning range of the performance indicator, adjust the data concurrency sending parameters until the performance indicator is within the warning range of the performance indicator.

[0092] When the performance indicator exceeds the warning range of the performance indicator, it is considered that the performance of the remote machine (big data server) has reached a bottleneck. The current amount of concurrent big data data has exceeded the processing capacity of the remote machine. It is necessary to adjust and reduce the amount of concurrent big data data, that is, it is necessary to adjust the data concurrent sending parameters to reduce the performance indicator.

[0093] If the performance indicator does not reach the lowest point of the performance indicator warning range, it means that the remote machine (big data server) still has sufficient big data processing performance resources. If you want to achieve maximum concurrency efficiency, you can also appropriately improve the performance indicator, that is, you need to adjust the data concurrency sending parameters to improve the performance indicator.

[0094] If the performance indicator is within the performance indicator warning range, it means that the big data processing efficiency of the remote machine (big data server) has reached the maximum value and no further adjustment is required to maintain the concurrent sending efficiency.

[0095] When the data processing parameters include: a data accumulation value and a performance index, the preset data processing parameter warning range includes: a data accumulation value warning range and a performance index warning range.

[0096] In step 203, according to the data processing parameters and the preset data processing parameter warning range, the data concurrency sending parameters are adjusted until the data processing parameters are within the preset data processing parameter warning range, which specifically includes the following steps.

[0097] According to the data accumulation value, the data accumulation value warning range, the performance index, and the performance index warning range, adjust the data concurrent sending parameters until the data accumulation value is within the data accumulation value warning range and the performance index is within the performance index warning range.

[0098] In the embodiment of the present application, the data accumulation and performance indicators of the data processing system are made appropriate through the data accumulation value and the data accumulation value warning range, the performance indicators and the performance indicator warning range, so as to achieve the appropriate maximum concurrency efficiency of the data processing system.

[0099] Based on the data processing method shown in the above embodiment, the embodiment of the present application also provides another data processing method. Figure 5 is a flowchart of a fourth data processing method provided in an embodiment of the present application, such as Figure 5 As shown, the above embodiment adjusts the data concurrent sending parameters according to the data accumulation value, the data accumulation value warning range, the performance index, and the performance index warning range until the data accumulation value is within the data accumulation value warning range and the performance index is within the performance index warning range, which specifically includes the following steps.

[0100] Step 501, according to the data accumulation value and the data accumulation value warning range, adjust the data concurrent sending parameter until the data accumulation value is within the data accumulation value warning range.

[0101] First ensure that the data accumulation value is within the data accumulation value warning range, and then determine the performance indicators, which can ensure that the system can run stably during the gradual increase in load, and will not crash or abnormal due to excessive load. If the system is unstable or has problems under high load, the results of the performance indicator test may be inaccurate or unreliable. Therefore, verifying the stability of the system first (first ensure that the data accumulation value is within the data accumulation value warning range) can improve the effectiveness and credibility of subsequent performance indicators. Therefore, first ensuring that the data accumulation value is within the data accumulation value warning range is a key step to ensure that the system can run stably under high load, and lays a solid foundation for subsequent performance indicator testing. This step-by-step testing method can effectively discover and solve various problems that may arise when the system faces high loads, thereby improving the overall reliability and performance of the system.

[0102] Step 502, when the data to be sent is concurrently sent to the big data server using the adjusted data concurrent sending parameters, if the performance indicator is within the performance indicator warning range, the adjusted data concurrent sending parameters are not adjusted.

[0103] The adjusted data concurrent sending parameters can ensure that the data accumulation value is within the data accumulation value warning range. At this time, continue to judge whether the performance indicator is within the performance indicator warning range. If the performance indicator is within the performance indicator warning range at this time, it means that the adjusted data concurrent sending parameters can meet both the data accumulation value warning range and the performance indicator warning range, indicating that the big data processing efficiency of the remote machine (big data server) has reached the maximum value. There is no need to adjust the big data concurrent data volume, and maintain the concurrent sending efficiency corresponding to the adjusted data concurrent sending parameters.

[0104] In an embodiment of the present application, by determining whether the data accumulation value is within the data accumulation value warning range and then judging whether the performance indicator is within the performance indicator warning range, it is possible to ensure stable operation of the system under high load and will not cause crashes or abnormalities due to excessive load.

[0105] In the above Figure 5 Based on the data processing method shown in FIG. 1 , the embodiment of the present application further provides another data processing method. Figure 6 is a flowchart of a fifth data processing method provided in an embodiment of the present application, such as Figure 6 As shown, the above method also includes the following steps.

[0106] Step 601, when sending the data to be sent to the big data server through the adjusted data concurrent sending parameters, if the performance indicator is not within the performance indicator warning range, adjust the adjusted data concurrent sending parameters until the performance indicator is within the performance indicator warning range.

[0107] At this time, the adjusted data concurrent sending parameters can satisfy the performance indicator within the performance indicator warning range.

[0108] Step 602, when the data to be sent is concurrently sent to the big data server using the adjusted data concurrent sending parameters, if the data accumulation value is within the data accumulation value warning range, the adjusted data concurrent sending parameters are not adjusted.

[0109] Since the data concurrent sending parameters have changed, it is uncertain whether the data accumulation value is still within the data accumulation value warning range. Therefore, it is necessary to re-judge whether the data accumulation value at this time is within the data accumulation value warning range. If the data accumulation value is within the data accumulation value warning range, the adjusted data concurrent sending parameters do not need to be adjusted, which means that the adjusted data concurrent sending parameters can meet both the data accumulation value warning range and the performance indicator warning range, indicating that the big data processing efficiency of the remote machine (big data server) has reached the maximum value. There is no need to adjust the big data concurrent data volume, and the concurrent sending efficiency corresponding to the adjusted data concurrent sending parameters can be maintained.

[0110] If the data accumulation value is not within the data accumulation value warning range at this time, continue to adjust the data concurrent sending parameters until the data accumulation value is within the data accumulation value warning range; use the adjusted data concurrent sending parameters to concurrently send the data to be sent to the big data server. If the performance indicator is within the performance indicator warning range, do not adjust the adjusted data concurrent sending parameters.

[0111] In the embodiment of the present application, it is ensured that the final data concurrent sending parameters can meet both the data accumulation value warning range and the performance indicator warning range. At this time, the big data processing efficiency of the remote machine (big data server) has reached the maximum value, and the concurrent sending efficiency corresponding to the adjusted data concurrent sending parameters can be maintained.

[0112] Based on the above embodiments, the present application also provides a data processing method. Figure 7 is a flowchart of a sixth data processing method provided in an embodiment of the present application, Figure 8 It is a flowchart of the seventh data processing method provided in an embodiment of the present application.

[0113] like Figure 7 As shown, the big data model is a preset big data model, the data format is a preset data structure, and the generated data volume is a preset data volume. The local system resource status is the test press resource status.

[0114] like Figure 8 As shown, the expected bottleneck value warning range includes: data accumulation value warning range, performance indicator warning range, the remote machine is a big data server, and the expected bottleneck value includes: the highest point of the data accumulation value warning range, and the highest point of the performance indicator warning range.

[0115] In big data model testing, the present application can automatically generate huge and complex data. At the same time, the system can adapt the remote (big data server) and local (test pressure machine) resources to adjust the sending of data streams according to the current situation, so as to achieve the highest data stream sending efficiency based on the minimum resource overhead of the local machine, realize data quality assurance and high concurrency and scalability requirements, and flexibly respond to big data tests with various pressure requirements.

[0116] This application can perform big data related tests with large data volumes, meet big data models of various sizes, and can flexibly schedule system resources to achieve appropriate concurrency effects. It can adapt to all test environments, has strong compatibility and high practicality.

[0117] In the present application, a big data stream is dynamically generated by default, customization, and specified types. During the sending of the big data stream, the frequency of data stream sending of the test press is controlled in real time by continuously and dynamically monitoring the data processing capabilities of the specific business of the current big data server, the resource overhead (memory) of the big data server's services, and the resource consumption of each local resource of the test press. By adjusting the system's scheduling of each user's data concurrency rate, as well as the total number of system processes and the number of threads of each process, the highest data stream sending efficiency is achieved based on the minimum resource overhead of the test press, thereby achieving data quality assurance as well as high concurrency and scalability requirements.

[0118] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0119] The present application embodiment relates to an electronic device, Fig. 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, it includes: at least one processor 901; and a memory 902 that is communicatively connected to the at least one processor 901; wherein the memory 902 stores instructions that can be executed by the at least one processor 901, and the instructions are executed by the at least one processor 901 so that the at least one processor 901 can execute the data processing method in the above-mentioned embodiments.

[0120] The memory 902 and the processor 901 are connected in a bus manner. The bus may include any number of interconnected buses and bridges. The bus connects various circuits of one or more processors and memories together.

[0121] In an embodiment of the present application, the electronic device may be a test press.

[0122] The embodiment of the present application relates to a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the above method embodiment is implemented.

[0123] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0124] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Determine the data concurrent sending parameters according to the test press resource situation; Concurrently sending the data to be sent to the big data server according to the data concurrent sending parameter; Acquire in real time the data processing parameters for the data to be sent during transmission, and adjust the data concurrency transmission parameters according to the data processing parameters and a preset data processing parameter warning range until the data processing parameters are within the preset data processing parameter warning range; wherein the preset data processing parameter warning range is used to indicate high data processing efficiency.

2. The data processing method according to claim 1, characterized in that: The method for determining the data to be sent includes: The data to be sent is generated by presetting a big data model, a presetting data structure and a presetting data volume.

3. The data processing method according to claim 1, characterized in that: The data processing parameters include: data accumulation value; the data processing parameters for real-time acquisition of the data transmission to be sent include: Real-time acquisition of the amount of data sent to the big data server per unit time and the amount of data consumed and processed by the big data server per unit time; The data accumulation value is determined according to the amount of data sent to the big data server within the unit time and the amount of data consumed and processed by the big data server within the unit time.

4. The data processing method according to claim 3, characterized in that: The preset data processing parameter warning range includes: a data accumulation value warning range; the adjusting the data concurrent sending parameter according to the data processing parameter and the preset data processing parameter warning range until the data processing parameter is within the preset data processing parameter warning range includes: According to the data accumulation value and the data accumulation value warning range, the data concurrent sending parameter is adjusted until the data accumulation value is within the data accumulation value warning range.

5. The data processing method according to claim 4, characterized in that: The data processing parameters also include: performance indicators; the real-time acquisition of the data processing parameters when the data is to be sent also includes: Obtaining services of the big data server in real time; According to the service, the performance indicator is determined.

6. The data processing method according to claim 5, characterized in that: The preset data processing parameter warning range also includes: a performance indicator warning range; the adjusting the data concurrency sending parameter according to the data processing parameter and the preset data processing parameter warning range until the data processing parameter is within the preset data processing parameter warning range includes: The data concurrent sending parameter is adjusted according to the data accumulation value, the data accumulation value warning range, the performance indicator, and the performance indicator warning range until the data accumulation value is within the data accumulation value warning range and the performance indicator is within the performance indicator warning range.

7. The data processing method according to claim 6, characterized in that: The adjusting the data concurrent sending parameter according to the data accumulation value, the data accumulation value warning range, the performance indicator, and the performance indicator warning range until the data accumulation value is within the data accumulation value warning range and the performance indicator is within the performance indicator warning range includes: According to the data accumulation value and the data accumulation value warning range, adjusting the data concurrent sending parameter until the data accumulation value is within the data accumulation value warning range; When the data to be sent is concurrently sent to the big data server by using the adjusted data concurrent sending parameter, if the performance indicator is within the performance indicator warning range, the adjusted data concurrent sending parameter is not adjusted.

8. The data processing method according to claim 7, characterized in that: The method further comprises: When sending the data to be sent to the big data server by using the adjusted concurrent data sending parameter, if the performance indicator is not within the performance indicator warning range, adjusting the adjusted concurrent data sending parameter until the performance indicator is within the performance indicator warning range; When the data to be sent is concurrently sent to the big data server using the adjusted data concurrent sending parameters, if the data accumulation value is within the data accumulation value warning range, the adjusted data concurrent sending parameters are not adjusted.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.

Images