A data acquisition back pressure control method and device

Abstract

The present invention provides a data acquisition backpressure control method and device, the data acquisition backpressure control method comprises: S1, constructing a backpressure prediction model using the time series data features constructed by the basic parameters of the sending end and the response information returned by the server end; S2, the sending end learning the backpressure prediction model, predicting the maximum number of requests and the amount of requests accepted by the server end in the next time period, and generating a prediction service API; S3, the sending end actively controls the number of requests sent by the server end in the next time period according to the prediction service API, and completes the backpressure control of data acquisition. The method of the present invention is used to solve the problem of data loss after the data cache of the sender and the receiver is full. The data acquisition backpressure control method of the present invention can improve the efficiency of data transmission and reduce the resource consumption caused by processing errors.

Description

Technical Field

[0001] The invention belongs to the field of software technology, and in particular to a data acquisition back pressure control method and device. Background Art

[0002] For the monitoring system, data collection on the end side is the responsibility of an agent. After the agent collects the data, it sends the data to the cloud side. However, when the agent collects data, there are two conflicting requirements: first, data collection must be timely, and second, the agent resource occupancy must be low. However, for most business systems, the monitoring system is required not to have a negative impact on the performance of the business system, so low resource occupancy is the first choice.

[0003] In order to ensure low resource utilization and timely data collection, the usual strategy is sampling. In order to ensure data integrity while requiring low agent resource utilization, the usual strategy is backpressure control. For pull mode collection, backpressure control is simply controlled by the agent. When the agent resource utilization reaches the critical value, collection is stopped and collection is resumed after the resource utilization rate decreases. This method is a relatively rough backpressure control strategy, which gives users a discontinuous feeling.

[0004] For push mode collection, when the receiver and the server use a non-backpressure communication protocol, data will be lost after the data cache of the sender and the server is full.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] In view of this, the present invention discloses a data acquisition back pressure control method and device to solve the problem of data loss after the data buffer of the sender and the receiver is full. The data acquisition back pressure control method of the present invention can improve the efficiency of data transmission and reduce resource consumption caused by processing errors.

[0007] Specifically, the present invention is achieved through the following technical solutions:

[0008] In a first aspect, the present invention discloses a data acquisition back pressure control method, comprising the following steps:

[0009] S1. Construct a back pressure prediction model using the time series data features constructed by the basic parameters of the sending end and the response information returned by the server end;

[0010] S2, the sending end learns the back pressure prediction model, predicts the maximum number and amount of requests accepted by the server in the next time period, and generates a prediction service API;

[0011] S3. The sending end actively controls the number of requests sent by the server in the next time period according to the prediction service API to complete the back pressure control of data collection.

[0012] Furthermore, in the step S1, the method for constructing the back pressure prediction model includes:

[0013] The sending end describes the amount of data transmitted this time, generates a transmission data event, and adds the response information returned by the response header of the server to add a response feature, thereby forming the time series data feature;

[0014] The failed requests returned by the response header are used as negative samples, and a back pressure prediction model is constructed in combination with the time series data features.

[0015] Furthermore, the response characteristics are characteristics of whether the transmission data event is collected successfully and characteristics of the request duration.

[0016] Furthermore, in the S1 step, the basic parameters include request ID, target recipient, request initiation time, request data volume, request return time, whether the request is successful, and request duration.

[0017] Furthermore, in the step S3, the active control method includes comparing the current amount of data accumulated and sent by the sending end with the predicted amount of data that can be sent, and determining the final amount of data to be sent according to the predicted result.

[0018] Furthermore, the sending end needs to train the backpressure prediction model before learning the backpressure prediction model.

[0019] Furthermore, in the S1 step, the timing characteristics include the number of requests in the past 5 minutes, the number of failures in the past 5 minutes, the total transmission volume in the past 5 minutes, the number of requests in the past 15 minutes, the number of failures in the past 15 minutes, and the total transmission volume in the past 15 minutes.

[0020] In a second aspect, the present invention discloses a data acquisition back pressure control device, comprising:

[0021] Back pressure prediction model building module: constructs a back pressure prediction model using the time series data features constructed by the basic parameters of the sending end and the response information returned by the server;

[0022] Prediction service API generation module: the sending end learns the back pressure prediction model, predicts the maximum number and amount of requests accepted by the server in the next time period, and generates a prediction service API;

[0023] Data collection module: The sending end actively controls the number of requests sent by the server in the next time period according to the prediction service API to complete the back pressure control of data collection.

[0024] In a third aspect, the present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the data acquisition backpressure control method as described in the first aspect are implemented.

[0025] In a fourth aspect, the present invention discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data acquisition back pressure control method as described in the first aspect are implemented.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The data acquisition back pressure control method and device proposed in the present invention can improve the efficiency of data transmission and reduce the resource consumption caused by processing errors. During data transmission, when the sending end and the server use the HTTP protocol as a stateless transmission protocol, a back pressure control prediction model is constructed without reconstructing the transmission protocol to predict the maximum number and amount of requests that the server can accept in the next time period, and actively control the number of requests sent in the next time period, thereby reducing the number of errors generated by sending requests, improving the sending efficiency, and reducing the resource consumption of the sending end. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0029] Figure 1 A flow chart of a data acquisition back pressure control method provided by an embodiment of the present invention;

[0030] Figure 2 A schematic diagram of a prediction code provided by an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of a traditional data collection architecture provided by an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of a data acquisition back pressure control architecture provided by an embodiment of the present invention;

[0033] Figure 5 A schematic diagram of a data acquisition back pressure control device provided in an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and specific embodiments, but those skilled in the art will understand that the embodiments described below are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to illustrate the present invention, and should not be regarded as limiting the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0036] In order to more clearly explain the technical solution of the present invention, it is described in the form of specific embodiments below.

[0037] Example

[0038] Reference Figure 1 As shown, the present invention discloses a data acquisition back pressure control method, comprising the following steps:

[0039] S1. Construct a back pressure prediction model using the time series data features constructed by the basic parameters of the sending end and the response information returned by the server end;

[0040] S2, the sending end learns the back pressure prediction model, predicts the maximum number and amount of requests accepted by the server in the next time period, and generates a prediction service API;

[0041] S3. The sending end actively controls the number of requests sent by the server in the next time period according to the prediction service API to complete the back pressure control of data collection.

[0042] The actual operation of the present invention is as follows:

[0043] The sender uses the size of each data packet sent, the number of times sent per second, and the average return time per second as basic parameters to construct some time series features, and uses the response information returned by the server to build a back pressure prediction model. Specifically, without intruding the transmission protocol, it interacts with the sender three times. Before each data transmission at the sender, the sender sends an event to the data management module through the sender's sending statistics module to describe the amount of data transmitted this time and generate a transmission data event; after each transmission at the sender, the sender's callback statistics module updates the information in the sender's sending statistics module based on the server's response header. Specifically, the response features returned by the server's response header are added to form time series data features, where the response features include whether the transmission data event is successfully collected and the request duration features. The back pressure prediction model is constructed based on the time series data features of the sender's sending statistics module and the sender's callback statistics module and the request with failed response header return as negative samples.

[0044] The time series data features formed by the entire data module are as follows: request ID, target recipient, request initiation time, request data volume, request return time, request success, and request duration. Specifically, through the basic data, the following additional features can be constructed: number of requests in the past 5 minutes, number of failures in the past 5 minutes, total transmission volume in the past 5 minutes, number of requests in the past 15 minutes, number of failures in the past 15 minutes, and total transmission volume in the past 15 minutes. Among them, the construction principle of the overall data feature of the number of requests is that when the transmission request quickly reaches a peak, the receiver rejects the service and directly rejects the next request from the sender.

[0045] Based on the above time series data, enter the back pressure prediction model training module, train the back pressure prediction model, and obtain the trained back pressure prediction model. The sender learns the back pressure prediction model, predicts the maximum number of requests and request volume accepted by the server in the next time period, and generates a prediction service API. Specifically, the number of requests in the next 5 seconds is predicted. The fitting model uses prophet. The prediction code is shown in the attached Figure 2 The sender actively controls the number of requests sent by the server in the next time period according to the prediction service API to complete the back pressure control of data collection.

[0046] For specific use, see Figure 3 The figure shows a common data acquisition processing architecture. In this architecture, agent C will become a bottleneck. Once it is busy, it will constantly reject the data sending requests of agents A and B. The traditional architecture is upgraded by the data acquisition back pressure control method of the present invention. Figure 4In the upgrade architecture shown, the data of link 1 and link 2 are collected to one place through the sending statistics module and the sending callback statistics module, and managed through the data management module. The pressure resistance of agent c is obtained through the back pressure prediction model, the tolerance of agent c is predicted in real time, and the rhythm of real-time data sending by the sending end is adjusted.

[0047] The data acquisition back pressure control method of the present invention is stress tested. Specifically, the test case is as follows: two agents are used to simulate the generation of 100M and 10G data in 1 minute and send data by code, and the data is sent to the third agent at the same time. The cgroup is used to limit the resource usage of the three agents to 2C4G.

[0048] Scenario 1: The sending end keeps retrying when encountering an error;

[0049] Scenario 2: The data acquisition back pressure control method of the present invention is used to dynamically adjust the sending amount of the sender; stress tests are performed through different scenarios to examine the time it takes to complete data transmission and the agent resource occupancy, and the experiment is repeated 3 times. The test structure proves that after using the data acquisition back pressure control method of the present invention, the average sending efficiency of the sender is increased by 5%, and the average CPU occupancy rate is reduced by 3%.

[0050] To sum up, the data acquisition back pressure control method proposed in the present invention can improve the efficiency of data transmission, reduce the resource consumption caused by processing errors, and build a back pressure control prediction model without reconstructing the transmission protocol to predict the maximum number of requests and request volume that the server can accept in the next time period, and actively control the number of requests sent in the next time period, thereby reducing the number of errors generated by sending requests, improving sending efficiency, and reducing resource consumption on the sending end.

[0051] The present invention also provides a data acquisition back pressure control device, such as Figure 5 As shown, specifically including:

[0052] Back pressure prediction model building module: constructs a back pressure prediction model using the time series data features constructed by the basic parameters of the sending end and the response information returned by the server;

[0053] Prediction service API generation module: the sending end learns the back pressure prediction model, predicts the maximum number and amount of requests accepted by the server in the next time period, and generates a prediction service API;

[0054] Data collection module: The sending end actively controls the number of requests sent by the server in the next time period according to the prediction service API to complete the back pressure control of data collection.

[0055] The device is mainly composed of the above three modules. Through the construction of the system, the purpose of parallel operation can be well achieved by mounting the same file system at the same time.

[0056] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined and implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0057] Figure 6 This is a schematic diagram of the structure of a computer device disclosed in the present invention. Figure 6 As shown, the computer device 400 includes at least a memory 402 and a processor 401; the memory 402 is connected to the processor via a communication bus 403, and is used to store computer instructions executable by the processor 401, and the processor 401 is used to read computer instructions from the memory 402 to implement the steps of the data acquisition back pressure control method described in the above embodiment.

[0058] For the above-mentioned device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0059] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0060] Finally, it should be noted that although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of a sub-combination.

[0061] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or requiring that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0062] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the particular order or sequential order shown to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0063] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data acquisition back pressure control method, characterized in that: The following steps are involved: S1. Use the time series data features constructed by the basic parameters of the sender and the response information returned by the server to build a backpressure prediction model; S2, the sending end learns the back pressure prediction model, predicts the maximum number and amount of requests accepted by the server in the next time period, and generates a prediction service API; S3, the sending end actively controls the number of requests sent by the server in the next time period according to the prediction service API, and completes the back pressure control of data collection; In the step S1, the method for constructing the back pressure prediction model includes: The sending end describes the amount of data transmitted this time, generates a transmission data event, and adds a response feature returned by the response header of the server to form the time series data feature; Based on the failed requests returned by the response header as negative samples, a back pressure prediction model is constructed in combination with the time series data features; The response characteristics are characteristics of whether the transmission data event is collected successfully and characteristics of the request duration; In the step S3, the active control method includes comparing the current amount of data accumulated and sent by the sending end with the predicted amount of data that can be sent, and determining the final amount of data to be sent according to the prediction result.

2. The data acquisition back pressure control method according to claim 1, characterized in that: In the S1 step, the basic parameters include request ID, target recipient, request initiation time, request data volume, request return time, whether the request is successful, and request duration.

3. The data acquisition back pressure control method according to claim 1, characterized in that: The sending end needs to train the backpressure prediction model before learning the backpressure prediction model.

4. The data acquisition back pressure control method according to claim 1, characterized in that: In the S1 step, the time series data features include the number of requests in the past 5 minutes, the number of failures in the past 5 minutes, the total transmission volume in the past 5 minutes, the number of requests in the past 15 minutes, the number of failures in the past 15 minutes, and the total transmission volume in the past 15 minutes.

5. A data acquisition back pressure control device, using the method according to any one of claims 1 to 4, characterized in that: include: Backpressure prediction model building module: builds a backpressure prediction model using the time series data features built by the basic parameters of the sender and the response information returned by the server; Prediction service API generation module: the sending end learns the back pressure prediction model, predicts the maximum number and amount of requests accepted by the server in the next time period, and generates a prediction service API; Data collection module: The sending end actively controls the number of requests sent by the server in the next time period according to the prediction service API to complete the back pressure control of data collection.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the steps of the data acquisition back pressure control method according to any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data acquisition back pressure control method according to any one of claims 1 to 4 are implemented.

Images