A load reduction adjustment method for converged terminal high concurrent data scenario

By using the delay interval statistics module and request queue in the intelligent converged terminal in the station area, the maximum delay interval is analyzed and estimated, and asynchronous and batch processing strategies are adopted to solve the problem of repeated data processing operations in high concurrent data scenarios, and the network is reduced IO load and the effect of improving data center performance.

CN114416361BActive Publication Date: 2024-12-06深圳市力合微电子股份有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210027801.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-11
Publication Date
2024-12-06
Estimated Expiration
2042-01-11

AI Technical Summary

Technical Problem

In the high concurrent data scenario, the inter-process data interaction method of traditional concentrator terminal devices leads to untimely repeated data processing operations, resulting in excessive system IO resources occupied, performance bottlenecks, and data that really needs to be updated cannot be obtained in time.

Method used

In the intelligent fusion terminal of the station area, the delay interval statistics module and request queue are initialized through the front-end data processing APP, the maximum delay interval of the estimated data type is analyzed and estimated, and the timer and message queue are used for asynchronous processing and batch processing to reduce repeated operation notifications. , delay loading of data and optimize network IO utilization.

Benefits of technology

It effectively reduces network IO load, reduces data update notification interaction, improves network IO utilization, avoids high concurrent load in the data center, and ensures business stability and data consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114416361B_ABST
    Figure CN114416361B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for reducing load in a high concurrent data scenario of a fusion terminal: S1, initialization; S2, writing a data type A to a request W A Put it into the request queue Q A ; S3, according to the estimated maximum delay interval MT1, start the first timer to monitor whether there is a similar request, if there is, input MT1 to M A After estimating the maximum delay interval of the next operation, return to S2; No, the first timer has finished, if not, continue monitoring, and if finished, go to S5; S5, take out Q A All request tasks of A are written into the data center, and after writing, an update notification of A is sent to each subscribing APP; S6, after receiving the update notification, the subscribing APP sets the validity flag of A to expire and starts the second timer to monitor whether this APP has any read operation on A. If yes, proceed to S9, otherwise proceed to S7; S7, whether the second timer has expired, if not, continue monitoring, if yes, proceed to S8; S8, determine whether the current data center is idle, if not, continue monitoring, if idle, proceed to S9; S9, the subscribing APP reloads A's data from the data center.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of smart meter communication technology, and in particular to a load reduction adjustment method in a high-concurrency data scenario of a fusion terminal and a related computer-readable storage medium. Background Art

[0002] The intelligent fusion terminal of the substation (hereinafter referred to as SCU) continues the design concept of "hardware modularization and software APPization" of the energy controller. It is developed and designed in combination with physical hardware with high main frequency and large storage. It has many advanced features such as distributed architecture, data and business separation, matrix application expansion, cross-platform business interaction, etc. The hardware modular design can realize the flexible access of metering and sensing equipment on the customer side and the distribution side, and has business functions such as data collection, intelligent fee control, clock synchronization, accurate metering, orderly charging, energy management, circuit status inspection, household-to-variable relationship identification, and power outage event reporting; the software APP design supports the interchange of the same APP from different manufacturers and supports the expansion and installation of new business APPs. The product can be widely used in the fields of electricity consumption information collection, power quality analysis, comprehensive energy metering and collection, distributed energy management, and orderly charging of electric vehicles.

[0003] For traditional concentrator terminal devices, they usually work in a multi-process mode. The data interaction between processes is realized by sharing memory or files. Whenever there is data update, the process of writing data sends a notification to the process of reading data through inter-process communication. The disadvantage of this is that it can only realize point-to-point data interaction, and this operation type is a completely synchronous mode. When a large number of operations of the same type occur, it will inevitably lead to the problem of repeated data processing between various processes and untimely operations. For example: the user continuously adds 100 meter devices to the terminal through the master station. According to the original plan, on the new intelligent fusion terminal of the substation area, every time the user adds a device, the terminal APP will operate a data center write operation, and the data center will send a notification to other subscribed APPs. In this way, this round of operation will operate the data center 100 times, send 100 notifications to each APP, and each APP will load the data of the data center 100 times; however, in fact, each APP often does not need to use the updated data immediately at this moment. If all APPs are loaded at the same time, a large amount of system IO resources will be occupied, resulting in a tidal effect and performance bottlenecks, which will cause the APP that really needs to update data to fail to obtain data in time. Summary of the invention

[0004] In view of this, the present invention proposes an adjustment method for reducing the load in a high-concurrency data scenario of an intelligent fusion terminal in an area, so as to solve the technical problems existing in the above-mentioned prior art.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for reducing load in a high-concurrency data scenario of a fusion terminal, comprising the following steps: S1, a front-end data processing APP initializes a delay interval statistics module and a request queue corresponding to each data type; wherein the APP refers to an application running on a terminal; S2, for any data type A among the data types, the front-end data processing APP receives a write request W for the data type A from the front-end. A Place it in the corresponding request queue Q A S3, the front-end data processing APP according to the data type A delay interval statistics module M A Analyze the estimated maximum delay interval of this time and start the first timer, and monitor whether there are continuous write requests of data type A next; if yes, execute step S4.1; if not, execute step S4.2; S4.1, input the delay interval sample of this operation into the corresponding delay interval statistics module M A , delay interval statistics module M A Analyze and estimate the maximum delay interval of the next operation, and then return to step S2; S4.2, determine whether the first timer has expired; if so, proceed to step S5; if not, continue to monitor whether there are continuous write requests of data type A; S5, the front-end data processing APP takes out the request queue Q A All request tasks of data type A in the data center are processed, and all the retrieved request tasks are written to the data center in a unified manner using transaction batch processing. After writing is completed, a data type A update notification is sent to each APP that subscribes to data type A through a message queue telemetry transmission method; S6. After receiving the data type A update notification, the APP that subscribes to data type A sets the valid flag of data type A to expire, and starts a second timer to monitor whether this APP has a read operation on data type A next; if a read operation on data type A is detected, step S9 is executed; if no read operation on data type A is detected, step S7 is executed; S7. Determine whether the second timer has expired; if not, continue monitoring; if so, execute step S8; S8. Determine whether the current data center APP is idle; if not, continue monitoring; if so, execute step S9; S9. The APP that receives the data type A update notification reloads the data of data type A from the data center.

[0007] Furthermore, in step S1, the terminal is initialized when it leaves the factory, including setting an initial default delay interval, and the initial number of statistical samples of each module of the front-end data processing APP is initialized to 0.

[0008] Furthermore, in step S2, write requests of various data types enter corresponding request queues using a first-in-first-out strategy to ensure that various tasks are executed in an orderly manner.

[0009] Furthermore, in step S2, the write request of each data type is not limited to the random copy request of the master station front end, but also includes the LCD interface operation of the terminal, the scheduled copy task or the event report.

[0010] Furthermore, in step S3, the front-end data processing APP uses an asynchronous method to monitor the front end, suspends the current thread, and wakes up when a write request is monitored to improve CPU utilization.

[0011] Furthermore, in step S3, one request queue corresponds to only one timer; for a certain request queue, when a new request of the same type arrives, the timer of this queue does not end, but restarts according to the new delay value.

[0012] Furthermore, in step S41, the delay interval statistics module M A Analyze and estimate the maximum delay interval of the next operation, including:

[0013] Delay interval statistics module M A The operation delay interval sample of each input follows the standard normal distribution, that is,

[0014]

[0015]

[0016] Statistical module M for calculating the delay interval A The expectation is μ and the standard deviation is σ, then the maximum delay interval MT2 of the next operation is calculated as follows:

[0017] MT2=μ+3σ+MT1*k

[0018] Among them, MT1 is the input delay interval sample of this operation, and k is the correlation factor with the previous operation.

[0019] Furthermore, in step S41, the delay interval statistics module M A When analyzing and estimating the maximum delay interval of the next operation, if it is the first time to input the sample, since the expected μ, standard deviation σ and MT1 do not exist yet, the calculation formula of MT2 is:

[0020] MT2=MT

[0021] Among them, MT is the default delay interval of the terminal when it leaves the factory.

[0022] Furthermore, in step S5, the batch processing interface of the data center is used to write and submit once. If multiple attempts fail, the request queue Q is retained. A All tasks in the process are completed, and the timing of the first timer is restarted, and the process returns to step S3.

[0023] The present invention further proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned adjustment method for reducing the load in a high-concurrency data scenario of a converged terminal can be implemented.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention estimates the delay interval of each front-end operation through intelligent analysis of the terminal, filters a large number of repeated operations of the same type, and greatly reduces the network IO load of the terminal in high concurrent data scenarios. It can also accurately estimate the maximum delay interval of the next front-end operation through historical statistical values ​​and the last actual value analysis, and then achieve the purpose of deduplication by detecting whether the current operation is the same type of operation as before. Therefore, it greatly reduces the data update notification interaction between terminal APPs, improves the utilization rate of network IO, and provides guarantees for other terminal services such as meter reading and event reporting.

[0026] In addition, the present invention uses a strategy of delayed loading of data by the terminal to fully utilize the idle time of each APP reading data and the idle time of requesting the data center, thereby avoiding the high concurrency behavior of a large number of APPs operating the same data type in the data center at the same time, and greatly reducing the load on the data center. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a method for adjusting the load reduction in a high concurrent data scenario of a converged terminal proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings and specific implementation methods.

[0029] like Figure 1 As shown, the embodiment of the present invention proposes a method for reducing load in a high concurrent data scenario of a converged terminal, including the following steps S1 to S9:

[0030] S1. The front-end data processing APP initializes the delay interval statistics module and request queue corresponding to each data type; wherein APP refers to the application program running on the terminal, and the terminal refers to the intelligent fusion terminal of the station area, on which three types of APPs are running: the front-end data processing APP, the data center APP, and other business APPs that subscribe to the data update notification of the data center APP. Among them, the front-end data processing APP is responsible for receiving and processing the front-end request.

[0031] The data type of the intelligent fusion terminal of the substation area may include various data types according to the actual business situation, such as daily curve collection plan, minute curve collection task or meter file information, etc., but not limited thereto. The subsequent steps are applicable to any data type.

[0032] The specific performance of initialization in step S1 is as follows: when the terminal leaves the factory, the initial statistical sample number of each module of the front-end data processing APP is 0, and the initial default delay interval is recorded as MT, which is generally set to 2 seconds. Of course, you can also customize and load existing configuration files to modify it. In addition, the statistical related values ​​of each module will be serialized to the file in real time. Even if the terminal has power outages and other events, it can continue to be used when it is powered on again, so it has strong recoverability.

[0033] S2. For any data type A among the data types, the front-end data processing APP receives a write request W for data type A from the front-end. A Place it in the corresponding request queue Q A Specifically, when the terminal leaves the factory, each request queue has no tasks. As time goes by, various types of requests will be continuously stored in the corresponding queues, using the first-in-first-out strategy to ensure the orderly execution of each task. It should be noted that the request is not limited to the random copy request of the master station front end, but may also be the terminal's LCD interface operation, scheduled copy task and event reporting, etc.

[0034] S3, the front-end data processing APP is based on the delay interval statistics module M of the data type A A Analyze the estimated maximum delay interval and start the first timer, and monitor whether there are continuous write requests of data type A next; if yes, execute step S4.1; if not, execute step S4.2. Specifically, the front-end data processing APP uses asynchronous mode to monitor the front-end, suspends the current thread, and wakes up immediately when a request comes, thereby greatly improving the CPU utilization; in addition, it should be noted that a request queue only corresponds to one timer Timer. When a new request of the same type arrives, the timer of this queue does not end, but restarts according to the new delay value.

[0035] S4.1. Input the delay interval sample of this operation into the corresponding delay interval statistics module M A , delay interval statistics module M A Analyze and estimate the maximum delay interval of the next operation, and then return to step S2. The calculation method for analyzing and estimating the maximum delay interval MT2 of the next operation is:

[0036] Delay interval statistics module M A The operation delay interval samples input each time can be considered to obey the standard normal distribution, that is:

[0037]

[0038] P(a k |y i )=ɡ(a k ,η yi ,σ yi )

[0039] Statistical module M for calculating the delay interval A The expectation is μ and the standard deviation is σ, then the maximum delay interval MT2 of the next operation is calculated as follows:

[0040] MT2=μ+3σ+MT1*k

[0041] Among them, MT1 is the input delay interval sample of this operation, k is the correlation factor with the previous operation, the general default value is 0.1, and it can also be customized according to the data type.

[0042] It should be noted that if the sample is input for the first time, since there is no expected μ, standard deviation σ and MT1, the calculation formula of MT2 is:

[0043] MT2=MT

[0044] MT is the default delay interval of the terminal when it leaves the factory, which is usually 2 seconds.

[0045] S4.2, determine whether the first timer has expired; if so, proceed to step S5; if not, continue to monitor whether there are continuous write requests of data type A;

[0046] S5: The front-end data processing APP takes out the request queue Q A All request tasks of data type A in the data center are written to the data center in a unified manner using transaction batch processing. After writing, data type A update notification is sent to each APP that subscribes to data type A through message queue telemetry transmission. Specifically, the batch processing interface of the data center is used to write and submit once. If multiple attempts fail, the queue Q is retained. AAll tasks in the process are executed, and the timing of the first timer is restarted, and the process returns to step S3. It should be noted that the maximum delay interval value in step S3 will not change.

[0047] S6. After receiving the update notification of data type A, the APP subscribing to data type A sets the valid flag of data type A to expire, and starts the second timer to monitor whether the APP has a read operation on data type A next; if a read operation on data type A is detected, step S9 is executed, which fundamentally ensures that each APP can obtain the correct and latest data in time while achieving data load balancing, avoiding the generation of dirty data; if no read operation on data type A is detected, step S7 is executed;

[0048] S7, determining whether the second timer has expired; if not, continuing monitoring; if yes, executing step S8;

[0049] S8, determine whether the current data center APP is idle; specifically, the terminal service APP will check whether there are unfinished tasks being executed in the current data center operation queue. If so, it is considered that the data center operation is busy, that is, "not idle", and continue monitoring; if it is detected that the current data center APP is idle, execute step S9;

[0050] S9. The APP that receives the update notification of the data type A reloads the data of the data type A from the data center and updates the memory.

[0051] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the load reduction adjustment method in the high-concurrency data scenario of the converged terminal of the aforementioned embodiment can be implemented.

[0052] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A method for reducing load in a high concurrent data scenario of a converged terminal, characterized in that: The steps include: S1. The front-end data processing APP initializes the delay interval statistics module and request queue corresponding to each data type; wherein APP refers to the application running on the terminal; S2. For any data type A among the data types, the front-end data processing APP receives a write request W for data type A from the front-end. A Place it in the corresponding request queue Q A middle; S3, the front-end data processing APP is based on the delay interval statistics module M of the data type A A Analyze the estimated maximum delay interval and start the first timer, and monitor whether there are consecutive write requests of data type A; if yes, execute step S4.1; if no, execute step S4.2; S4.

1. Input the delay interval sample of this operation into the corresponding delay interval statistics module M A , delay interval statistics module M A Analyze and estimate the maximum delay interval of the next operation, and then return to step S2; wherein the delay interval statistics module M A Analyze and estimate the maximum delay interval of the next operation, including: delay interval statistics module M A Each input operation delay interval sample follows the standard normal distribution, and the delay interval statistics module M is counted A The expectation is μ and the standard deviation is σ, then the maximum delay interval MT2 of the next operation is calculated as follows: MT2=μ+3σ+MT1*k Among them, MT1 is the input delay interval sample of this operation, and k is the correlation factor with the previous operation; Delay interval statistics module M A When analyzing and estimating the maximum delay interval of the next operation, if it is the first time to input the sample, since the expected μ, standard deviation σ and MT1 do not exist yet, the calculation formula of MT2 is: MT2=MT Among them, MT is the default delay interval when the terminal leaves the factory; S4.2, determine whether the first timer has expired; if so, proceed to step S5; if not, continue to monitor whether there are continuous write requests of data type A; S5: The front-end data processing APP takes out the request queue Q A All request tasks of data type A in the data center are collected and written to the data center in a unified manner using transaction batch processing. After writing is completed, data type A update notification is sent to each APP that subscribes to data type A through message queue telemetry transmission; S6. After receiving the update notification of data type A, the APP subscribing to data type A sets the valid flag of data type A to expire, and starts the second timer to monitor whether the APP has a read operation on data type A next; if a read operation on data type A is detected, step S9 is executed; if no read operation on data type A is detected, step S7 is executed; S7, determining whether the second timer has expired; if not, continuing monitoring; if yes, executing step S8; S8, determine whether the current data center APP is idle; if not, continue monitoring; if yes, execute step S9; S9. The APP that receives the update notification of the data type A reloads the data of the data type A from the data center.

2. The method for reducing load in a high concurrent data scenario of a converged terminal according to claim 1, characterized in that: In step S1, the terminal is initialized when it leaves the factory, including setting an initial default delay interval, and the initial statistical sample numbers of each module of the front-end data processing APP are initialized to 0.

3. The method for reducing load in a high concurrent data scenario of a converged terminal according to claim 1, characterized in that: In step S2, write requests of various data types enter the corresponding request queue using a first-in-first-out strategy to ensure that various tasks are executed in an orderly manner.

4. The method for reducing load in a high concurrent data scenario of a converged terminal according to claim 1, characterized in that: In step S2, the write request of each data type is not limited to the random copy request of the master station front end, but also includes the LCD interface operation of the terminal, the scheduled copy task or the event report.

5. The method for reducing load in a high concurrent data scenario of a converged terminal according to claim 1, characterized in that: In step S3, the front-end data processing APP uses an asynchronous method to monitor the front end, suspends the current thread, and wakes up when a write request is monitored to improve CPU utilization.

6. The method for reducing load in a high concurrent data scenario of a converged terminal according to claim 5, characterized in that: In step S3, one request queue corresponds to only one timer; for a certain request queue, when a new request of the same type arrives, the timer of this queue does not end, but restarts according to the new delay value.

7. The method for reducing load in a high concurrent data scenario of a converged terminal according to claim 1, characterized in that: In step S5, the batch processing interface of the data center is used to write and submit once. If multiple attempts fail, the request queue Q is retained. A All tasks in the process are completed, and the timing of the first timer is restarted, and the process returns to step S3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for reducing the load in a high-concurrency data scenario of a converged terminal as described in any one of claims 1 to 7 can be implemented.