Data processing method and device
By monitoring the change trend of data request success rate, the data received by the server is cached and the response from the cache area is preferred, which solves the problem of high response pressure from the server when receiving a large number of requests, and improves response efficiency and service availability.
Patent Information
- Application Number
- CN202310686785.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-09
- Publication Date
- 2024-12-10
AI Technical Summary
When a large number of client requests are received by the server, the response pressure of the data request from the data source is high, resulting in a reduced response efficiency and may cause the services provided by the server to be unavailable.
By monitoring the change trend of data request success rate, determine whether there is a traffic burst trend. If it exists, the target data will be cached on the server side and priority will be responded from the cache area when subsequent data requests are made to reduce the request pressure on the data source.
It effectively reduces the response pressure of data requests of data sources, improves the response efficiency of data sources, avoids unavailability caused by excessive data source pressure on the server, and improves user experience.
Smart Images
Figure CN119109985A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method and device. Background Art
[0002] The server can receive requests sent by the client. After receiving the requests sent by the client, the server can request the corresponding data from the data source, thereby feeding the data back to the client. In some scenarios, the server will receive a large number of requests from the client in a short period of time. For each request received, the server needs to request the corresponding data from the data source, which leads to a high pressure on the data source to respond to data requests. Accordingly, the data source's data request response efficiency will also be reduced accordingly.
[0003] Therefore, a solution is urgently needed to solve the above problems. Summary of the invention
[0004] In order to solve or at least partially solve the above technical problems, an embodiment of the present application provides a data processing method and device.
[0005] In a first aspect, an embodiment of the present application provides a data processing method, the method comprising:
[0006] Obtaining a data request success rate change trend monitored in the current monitoring period, wherein the data request success rate change trend is used to record a success rate of a data source responding to a data request from a server in chronological order of a preset number of monitoring periods;
[0007] Determine whether there is a traffic burst trend according to the change trend of the data request success rate;
[0008] When it is determined that there is a traffic burst trend, the server responds to the client's data request, obtains target data from the data source and feeds back the target data to the client, and synchronously caches the target data in the cache area of the server; the cache area is used for the server to preferentially obtain data from the cache area to respond to other data requests when it is determined that there is a traffic burst trend.
[0009] Optionally, determining whether there is a traffic burst trend according to the data request success rate change trend includes:
[0010] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing, it is determined that there is a traffic burst trend.
[0011] Optionally, determining whether there is a traffic burst trend according to the data request success rate change trend includes:
[0012] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold, it is determined that there is a traffic burst trend.
[0013] Optionally, determining whether there is a traffic burst trend according to the data request success rate change trend includes:
[0014] When the success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually decreasing and the time parameter of the data source responding to the data request of the server monitored in the current monitoring period is greater than or equal to a preset time parameter threshold, it is determined that there is a traffic burst trend.
[0015] Optionally, determining whether there is a traffic burst trend according to the data request success rate change trend includes:
[0016] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold and the time parameter of the data source in responding to the data request of the server monitored in the current monitoring period is greater than or equal to the preset time parameter threshold, it is determined that there is a traffic burst trend.
[0017] Optionally, the time parameter is used to characterize the proportion of requests whose response time of the data source in responding to the data request of the server exceeds a preset time threshold; then the preset time parameter threshold includes a timeout response request ratio threshold.
[0018] Optionally, the preset success rate threshold and / or the preset time parameter threshold supports dynamic adjustment.
[0019] Optionally, the method further includes:
[0020] When it is determined that there is no traffic burst trend, the target data is obtained from the data source in response to a data request from the client, and the target data is fed back to the client without caching the target data.
[0021] In a second aspect, an embodiment of the present application provides a data processing device, the device comprising:
[0022] An acquisition unit, used to acquire a data request success rate change trend monitored in a current monitoring period, wherein the data request success rate change trend is used to record a success rate of a data source responding to a data request of a server end in a time sequence of a preset number of monitoring periods;
[0023] A determination unit, configured to determine whether there is a traffic burst trend according to the change trend of the data request success rate;
[0024] A cache unit is used to synchronously cache the target data in a cache area of the server when the server obtains the target data from the data source and feeds back the target data to the client in response to a data request from the client when it is determined that there is a traffic burst trend; the cache area is used for the server to preferentially obtain data from the cache area in response to other data requests when it is determined that there is a traffic burst trend.
[0025] Optionally, the determining unit is used to:
[0026] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing, it is determined that there is a traffic burst trend.
[0027] Optionally, the determining unit is used to:
[0028] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold, it is determined that there is a traffic burst trend.
[0029] Optionally, the determining unit is used to:
[0030] When the success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually decreasing and the time parameter of the data source responding to the data request of the server monitored in the current monitoring period is greater than or equal to a preset time parameter threshold, it is determined that there is a traffic burst trend.
[0031] Optionally, the determining unit is used to:
[0032] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold and the time parameter of the data source in responding to the data request of the server monitored in the current monitoring period is greater than or equal to the preset time parameter threshold, it is determined that there is a traffic burst trend.
[0033] Optionally, the time parameter is used to characterize the proportion of requests whose response time of the data source in responding to the data request of the server exceeds a preset time threshold; then the preset time parameter threshold includes a timeout response request ratio threshold.
[0034] Optionally, the preset success rate threshold and / or the preset time parameter threshold supports dynamic adjustment.
[0035] Optionally, the device further comprises:
[0036] A feedback unit is used to obtain target data from the data source in response to a data request from a client when it is determined that there is no traffic burst trend, and to feed back the target data to the client without caching the target data.
[0037] In a third aspect, an embodiment of the present application provides a device, the device comprising a processor and a memory;
[0038] The processor is used to execute instructions stored in the memory so that the device executes the method described in any one of the first aspects above.
[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, wherein the instructions instruct a device to execute any of the methods described in the first aspect above.
[0040] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any of the methods described in the first aspect above.
[0041] Compared with the prior art, the embodiments of the present application have the following advantages:
[0042] The embodiment of the present application provides a data processing method, in one example, the method can be applied to a server. The server can obtain the data request success rate change trend monitored in the current monitoring period, and the data request success rate change trend is used to record the success rate of the data source responding to the data request of the server in the time sequence of a preset number of monitoring periods. After obtaining the data request success rate change trend, it can be determined whether there is a traffic burst trend according to the data request success rate change trend. If it is determined that there is a traffic burst trend, the server can obtain the target data from the data source and feedback the target data to the client in response to the client's data request, and synchronously cache the target data in the cache area of the server, thereby realizing active degradation of the dependent data source. Among them, the cache area is used for the server to obtain data from the cache area in priority to respond to other data requests when it is determined that there is a traffic burst trend. In this way, when responding to other data requests, the server does not need to request the corresponding data from the data source, thereby reducing the data request response pressure of the data source, and correspondingly, it can also improve the data request response efficiency of the data source. Moreover, this solution can avoid the unavailability of services provided by the server due to failures of the data source due to excessive request response pressure. Accordingly, it improves the availability of services provided by the server, thereby providing users with a better service experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0044] Figure 1 A schematic diagram of an exemplary application scenario provided for an embodiment of the present application;
[0045] Figure 2 A flowchart of a data processing method provided in an embodiment of the present application;
[0046] Figure 3 A schematic diagram of a statistical cycle provided in an embodiment of the present application;
[0047] Figure 4 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;
[0048] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0049] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0050] The inventor of the present application has found through research that with the development of network technology, more and more applications (applications, APPs) have emerged, and applications can also be called clients. Users can download and install clients on their devices. When a user triggers an operation on a client, the client can send a request to the server, and accordingly, the server feeds back data to the client in response to the request sent by the client. When the client receives the data fed back by the server, it can perform corresponding operations, for example, refreshing the content displayed on the client page.
[0051] In one example, after receiving the request sent by the client, the server can request the corresponding data from the data source accordingly, thereby feeding the data back to the client. In some scenarios, such as when a new function is launched or a major event occurs, a large number of users will use the client in the same time period (for example, a large number of users experience the new function), and accordingly, the server will receive a large number of requests from the client in a short period of time. For each request received, the server needs to request the corresponding data from the data source, that is, the server will also send a large number of data requests to the data source in a short period of time. This leads to a large pressure on the data source to respond to data requests, and accordingly, the data source's data request response efficiency will also become lower accordingly. Among them, the phenomenon that the server receives a large number of requests from the client in a short period of time can be called a "traffic burst", and accordingly, the phenomenon that the server sends a large number of data requests to the data source in a short period of time can also be called a "traffic burst".
[0052] The data source mentioned in the embodiments of the present application may be any device or system that can provide data. For example, the data source may be a device such as a server, a file system running on a device such as a server, a database running on a device such as a server, a service running on a server, etc., which will not be listed one by one here.
[0053] At present, when a traffic burst is found, the data source can be expanded to ensure the data request response efficiency of the data source. For example, the data source can be expanded horizontally or vertically. However, this method has a high maintenance cost for the data source. For example, after the traffic burst phenomenon ends, the data source needs to be reduced. On the other hand, considering that it takes a certain amount of time to expand the data source, the data request response efficiency of the data source cannot be improved during this period.
[0054] In order to solve the above problems, the embodiments of the present application provide a data processing method and device, which can effectively ensure the data request response efficiency of the data source without expanding the data source.
[0055] Various non-limiting implementations of the present application are described in detail below in conjunction with the accompanying drawings.
[0056] Exemplary Methods
[0057] See also Figure 1 , which is a schematic diagram of an exemplary application scenario provided in an embodiment of the present application.
[0058] like Figure 1 As shown, the client 110 and the client 120 can interact with the server 130, and the server 130 can interact with the data source 140. It should be noted that the server 120 can interact with multiple clients. Figure 1 Only client 110 and client 120 are shown. In addition, server 130 can interact with multiple data sources. Figure 1 Only data source 140 is shown.
[0059] In one example, the client 110 may send a request 1 to the server 130. Accordingly, after receiving the request 1, the server 130 may send a request 2 to the data source 140 to query the data source 140 for the data requested by the client 110. After receiving the request 2, the data source 140 may respond to the request 2 and return the data corresponding to the request 2 to the server 130. The server 130 further sends the data returned by the data source 140 to the client 110. In an embodiment of the present application, the server 130 may record the success rate of the data source 140 responding to the data request of the server 130 according to the time sequence of a preset number of monitoring cycles, obtain the change trend of the data request success rate monitored in the current monitoring cycle, and further determine whether there is a traffic burst trend according to the change trend of the data request success rate. When it is determined that there is a traffic burst trend, the data returned by the data source 140 is cached in the cache area. In this way, when the server 130 receives a request from the client 120 later, the data requested by the client 120 can be obtained from the cache area first, thereby reducing the data request response pressure of the data source 140 and, correspondingly, improving the data request response efficiency of the data source 140.
[0060] In one example, the server 130 may include a sliding time window statistics module 131 and a business processing module 132. The sliding time window statistics module 131 is used to obtain parameters corresponding to each monitoring period. For example, the success rate of the data source 140 responding to the data request of the server 130 can be obtained by statistics of the sliding time window statistics module 131. For another example, the time parameters of the data source responding to the data request of the server in the current monitoring period mentioned below can also be obtained by statistics of the sliding time window statistics module 131. The business processing module 132 is used to process specific business logic. For example, the business processing module 132 is used to parse the data request 1 from the client 110 and send a request 2 to the data source 140 to query the data source 140 for the data requested by the client 110.
[0061] Next, combine Figure 2 , the data processing method provided in the embodiment of the present application is introduced.
[0062] See also Figure 2 , which is a flow chart of a data processing method provided in an embodiment of the present application. Figure 2 The method shown can be applied to the server. In one example, the server can correspond to Figure 1 The server 130 is shown.
[0063] In a specific example, Figure 2 The method shown may include the following S101-S103.
[0064] In one example, Figure 2 The first client in the method shown may correspond to Figure 2 Client 110 is shown; Figure 2 The first data source in the method shown may correspond to Figure 1 A data source 140 is shown.
[0065] S101: Obtain a data request success rate variation trend monitored in a current monitoring period, wherein the data request success rate variation trend is used to record a success rate of a data source responding to a data request from a server in chronological order of a preset number of monitoring periods.
[0066] S102: Determine whether there is a traffic burst trend according to the data request success rate change trend.
[0067] In one example, considering that if there is a traffic burst trend, the efficiency of the data source responding to the data request from the server will be low. Therefore, it is possible to determine whether there is a traffic burst trend based on the efficiency of the data source responding to the data request from the server. In a specific example, the success rate change trend of the data source responding to the data request from the server can reflect the efficiency of the data source responding to the data request from the server, or in other words, the success rate change trend of the data source responding to the data request from the server can reflect the change trend of the data source responding to the data request from the server. Therefore, the server can obtain the data request success rate change trend monitored in the current monitoring period, and use the data request success rate change trend to determine whether there is a traffic burst trend.
[0068] The data request success rate mentioned in the embodiment of the present application may be the ratio of the number of data requests to which the data source successfully responds to and the total number of data requests sent by the server to the data source. Wherein, after receiving the data request, if the data source returns the data corresponding to the data request to the server, it means that the data source has successfully responded to the data request; if the data source does not return the data corresponding to the data request to the server, it means that the data source has not successfully responded to the data request. This can be understood in conjunction with formula (1).
[0069] Ps=Cq_suc / Cq_win_n Formula (1)
[0070] In formula (1):
[0071] Cq_suc is the number of data requests that the data source successfully responds to within a certain monitoring period;
[0072] Cq_win_n is the total number of data requests sent by the server to the data source during the monitoring period;
[0073] Ps is the data request success rate corresponding to the monitoring period. In an embodiment of the present application, the data request success rate change trend is used to record the success rate of the data source responding to the data request of the server in the time sequence of a preset number of monitoring periods. The data request success rate change trend can indicate the change trend of the success rate of the data source responding to the data request of the server. In an embodiment of the present application, the server can record the time when it sends a data request to the data source and the time when the data source returns the data corresponding to the data request, so as to obtain the success rate of the data source responding to the data request of the server corresponding to each monitoring period based on the recorded content, thereby obtaining the prime number data request success rate change trend based on the data request success rate corresponding to the preset number of monitoring periods.
[0074] The present application embodiment does not specifically limit the preset number, and the preset number can be set according to actual conditions. For example, the preset number can be 3. Figure 3 To understand, Figure 3 A schematic diagram of a monitoring cycle provided for an embodiment of the present application. The data request success rate change trend can be determined by the data request success rate corresponding to three consecutive monitoring cycles, assuming that monitoring cycle 301, monitoring cycle 302 and monitoring cycle 303 are three consecutive monitoring cycles, wherein monitoring cycle 303 can be the current statistical cycle, monitoring cycle 302 is the previous monitoring cycle, and monitoring cycle 301 is the previous monitoring cycle of monitoring cycle 302. Then the data request success rate change trend can be determined by the data request success rate corresponding to monitoring cycle 301, the data request success rate corresponding to monitoring cycle 302, and the data request success rate corresponding to monitoring cycle 303.
[0075] In the embodiment of the present application, considering that once there is a traffic burst trend, the success rate of the data source responding to the data request of the server will be correspondingly lowered, therefore, in one example, when the data request success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually decreasing, it is determined that there is a traffic burst trend. Figure 3 For understanding, assuming that the data request success rate corresponding to monitoring period 301, the data request success rate corresponding to monitoring period 302, and the data request success rate corresponding to monitoring period 303 satisfy the following formula (2), it can be determined that there is a traffic burst trend.
[0076] Ps 301 >Ps 302 >Ps 303 Formula (2)
[0077] In formula (2):
[0078] Ps301 The data request success rate corresponding to the monitoring period 301;
[0079] Ps 302 The data request success rate corresponding to the monitoring period 302;
[0080] Ps 303 It is the data request success rate corresponding to the monitoring period 303.
[0081] Correspondingly, if the data request success rate variation trend does not indicate that the success rate of the data source in responding to the data requests of the server is gradually decreasing, for example, if the data request success rate variation trend indicates that the success rate of the data source in responding to the data requests of the server fluctuates, then it can be determined that there is no traffic burst trend.
[0082] In one example, considering the data request success rate corresponding to the current monitoring period, it can reflect the response efficiency of the data source to the data request from the server. Therefore, in one example, S102 can also be combined with the data request success rate corresponding to the current monitoring period when it is implemented. In a specific example, if the data request success rate corresponding to the current monitoring period is less than or equal to the preset success rate threshold, it means that the data source has a low response efficiency to the data request from the server, and in this case, there is a high possibility of a traffic burst trend. Therefore, when S102 is implemented, it can be determined that there is a traffic burst trend when the data request success rate change trend indicates that the success rate of the data source responding to the data request from the server is gradually decreasing, and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold.
[0083] The data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold, which can be expressed as the following formula (3).
[0084] Ps n ≤Ps valid Formula (3)
[0085] In formula (3):
[0086] Ps n The data request success rate monitored in the current monitoring period;
[0087] Ps valid is the preset success rate threshold.
[0088] Correspondingly, if the data request success rate change trend does not indicate that the success rate of the data source in responding to the data request of the server is gradually decreasing, or the data request success rate monitored in the current monitoring period is greater than the preset success rate threshold, it is determined that there is no traffic burst trend.
[0089] The embodiments of the present application do not specifically limit the preset success rate threshold, and the preset success rate threshold can be set according to actual conditions. In one example, the preset success rate threshold can be a fixed value. In another example, the preset success rate threshold can also support dynamic adjustment. For example, the preset success rate threshold can be adaptively adjusted according to the performance of the data source. For another example, the preset success rate threshold can be adaptively adjusted according to the type of service provided by the client.
[0090] In another example, considering the response time of the data source responding to the data request of the server in the current monitoring period, the response efficiency of the data source responding to the data request of the server can also be reflected. For example, when the response efficiency is high, the response time is relatively short; when the response efficiency is low, the response time is relatively long. Therefore, in one example, when S102 is implemented specifically, it can also be combined with the time parameter of the data source responding to the data request of the server monitored in the current monitoring period. In a specific example, if the time parameter is greater than or equal to the preset time parameter threshold, it means that the response efficiency of the data source to the data request from the server is low, and in this case, the possibility of a traffic burst trend is high. Therefore, when S102 is implemented specifically, it can be determined that there is a traffic burst trend when the success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually reduced, and when the time parameter of the data source responding to the data request of the server monitored in the current monitoring period is greater than or equal to the preset time parameter threshold.
[0091] Correspondingly, if the trend of the change in the data request success rate does not indicate that the success rate of the data source in responding to the data request of the server is gradually decreasing, or the time parameter of the data source in responding to the data request of the server monitored in the current monitoring period is less than the preset time parameter threshold, it is determined that there is no traffic burst trend.
[0092] The embodiments of the present application do not specifically limit the preset time parameter threshold, and the preset time parameter threshold can be set according to actual conditions. In one example, the preset time parameter threshold can be a fixed value. In another example, the preset time parameter threshold can also support dynamic adjustment. For example, the preset time parameter threshold can be adaptively adjusted according to the performance of the data source. For another example, the preset time parameter threshold can be adaptively adjusted according to the type of service provided by the client.
[0093] In another example, when S102 is implemented, it can also comprehensively consider the data request success rate monitored in the current monitoring period and the time parameter of the data source responding to the data request of the server monitored in the current monitoring period, so as to make the judgment of the traffic burst trend more accurate. In a specific example, when S102 is implemented, it can determine the presence of a traffic burst trend when the data request success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually decreasing, and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold, and the time parameter of the data source responding to the data request of the server monitored in the current monitoring period is greater than or equal to the preset time parameter threshold.
[0094] Correspondingly, if the trend of the data request success rate change does not indicate that the success rate of the data source in responding to the data request of the server is gradually decreasing, or the data request success rate monitored in the current monitoring period is greater than the preset success rate threshold, or the time parameter of the data source in responding to the data request of the server monitored in the current monitoring period is less than the preset time parameter threshold, it is determined that there is no traffic burst trend.
[0095] The time parameter mentioned in the embodiment of the present application may be a parameter related to the response time, which may be determined based on the response time of the data source in response to multiple data requests of the server in the current monitoring period. For a certain data request, the response time corresponding to the data request may be the difference between the second time when the server receives the data returned by the data source for the data request and the first time when the server sends the data request to the data source.
[0096] In one example, the time parameter may be the average response time of the data source responding to the data request of the server in the current monitoring period. Accordingly, in this case, the preset time parameter threshold may be the average response time threshold.
[0097] In another example, the time parameter can be used to characterize the proportion of requests in which the response time of the data source to the data request of the server exceeds the preset time threshold. Correspondingly, for this case, the preset time parameter threshold is the timeout response request ratio threshold, and the timeout response request ratio threshold can be a number less than 1 and greater than 0. The specific value of the timeout response request ratio threshold is not specifically limited in the embodiment of the present application.
[0098] Specifically, it can be understood by combining formula (4) and formula (5).
[0099] Ptn=Ct_over / Cq_win_n Formula (4)
[0100] Ptn≥Ptvalid formula (5)
[0101] In formula (4) and formula (5):
[0102] Ptn is the proportion of requests in which the response time of the data source to the data request of the server exceeds the preset time threshold in the current monitoring period;
[0103] Ct_over is the number of data requests whose response time exceeds the preset time threshold in the current monitoring period;
[0104] Cq_win_n is the total number of data requests in the current monitoring period;
[0105] Ptvalid is the timeout response request ratio threshold.
[0106] The embodiments of the present application do not specifically limit the preset time threshold, and the preset time threshold can be set according to actual conditions. In one example, the preset time threshold can be a fixed value. In another example, the preset time threshold can also support dynamic adjustment. For example, the preset time threshold can be adaptively adjusted according to the performance of the data source. For another example, the preset time threshold can be adaptively adjusted according to the type of service provided by the client.
[0107] In another example, considering that when there is a traffic burst, the data requests sent by the server to the data source will surge, and when the increment of the data requests corresponding to a certain monitoring period is relatively large relative to the previous monitoring period, and the increment ratio continues to increase, it can be determined that the data requests sent by the server to the data source are in a surge state. For this case, in the specific implementation of S102, if the success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually decreasing, and the data requests sent by the server to the data source are in a surge state, it is determined that there is a traffic burst trend. In one example, the data request sent by the server to the data source is in a surge state, which can be reflected by the following formula (6).
[0108] Pq_1≥Pq_2≥Pq_valid Formula (6)
[0109] In formula (6):
[0110] Pq_1 is the increment ratio corresponding to the current monitoring period, which can be obtained by formula (7);
[0111] Pq_2 is the increment ratio corresponding to the previous monitoring period, which can be obtained by formula (8);
[0112] Pq_valid is the preset increment ratio threshold.
[0113] in:
[0114] Pq_1=Cq_win_n / Cq_win_n-1 Formula (7)
[0115] Pq_2=Cq_win_n -1 / Cq_win_n-2 Formula (8)
[0116] In formula (7) and formula (8):
[0117] Cq_win_n is the total amount of data requests sent by the server to the data source during the current monitoring period;
[0118] Cq_win_n-1 is the total number of data requests sent by the server to the data source in the previous monitoring cycle of the current monitoring cycle;
[0119] Cq_win_n-2 is the total amount of data requests sent by the server to the data source in the first two monitoring cycles of the current monitoring cycle.
[0120] S103: When it is determined that there is a traffic burst trend, the server obtains target data from the data source in response to the client's data request and feeds back the target data to the client, and synchronously caches the target data in the cache area of the server; the cache area is used for the server to preferentially obtain data from the cache area to respond to other data requests when it is determined that there is a traffic burst trend.
[0121] In an embodiment of the present application, when it is determined that there is a traffic burst, in order to reduce the data request response pressure of the data source in the traffic burst scenario, the server, when responding to the data request of the client, can synchronously cache the target data in the cache area of the server when obtaining the target data from the data source and feeding back the target data to the client. In one example, the server can cache the target data in a key-value manner.
[0122] Regarding the buffer, it should be noted that in the embodiment of the present application, the buffer is used to cache data, and the buffer is also used for the server to preferentially obtain data from the buffer in response to other data requests when it is determined that there is a trend of traffic bursts. In other words, when the server determines that there is a trend of traffic bursts, if it receives a data request from the client, the server can preferentially search for data from the cache in response to the data request, rather than directly requesting the data requested by the client from the data source, thereby reducing the pressure on the data source to respond to data requests.
[0123] In one example, if the server finds the data requested by the client in the cache, it will feed the data back to the client. If the server does not find the data requested by the client in the cache, the server can further request the data from the data source and feed it back to the client.
[0124] It can be seen that, using this solution, when it is determined that there is a traffic burst trend, the server can cache the data from the data source, thereby realizing active degradation of the dependent data source, thereby effectively reducing the data request response pressure of the data source in the traffic burst scenario, and correspondingly improving the data request response efficiency of the data source in the traffic burst scenario, thereby avoiding the unavailability of the service provided by the server due to the failure of the data source due to excessive request response pressure, and accordingly improving the availability of the service provided by the server, thereby providing users with a better service experience.
[0125] In one example, if after executing S102, it is determined that there is no traffic burst trend, in order to improve the efficiency of the server in responding to the client's data request, the real-time nature of the data returned to the client is guaranteed. When responding to the client's data request, the server can directly feed back the target data to the client without caching the target data obtained from the data source. In other words, after the server obtains the target data from the data source, it directly feeds back the target data to the client without performing a cache operation, thereby avoiding performing a cache operation and reducing the efficiency of feeding back the target data to the client, and accordingly, ensuring the real-time nature of the data returned to the client.
[0126] Exemplary Devices
[0127] Based on the method provided in the above embodiment, the embodiment of the present application further provides a device, which is described below in conjunction with the accompanying drawings.
[0128] See also Figure 4 , which is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application, Figure 4 The data processing device 400 shown can be used to execute the data processing method provided by the above method embodiment.
[0129] In an example, the data processing device 400 includes: an acquisition unit 401 , a determination unit 402 , and a cache unit 403 .
[0130] An acquisition unit 401 is used to acquire a data request success rate change trend monitored in a current monitoring period, wherein the data request success rate change trend is used to record a success rate of a data source responding to a data request from a server in a time sequence of a preset number of monitoring periods;
[0131] A determination unit 402 is used to determine whether there is a traffic burst trend according to the data request success rate change trend;
[0132] The cache unit 403 is used to synchronously cache the target data in the cache area of the server when the server obtains the target data from the data source and feeds back the target data to the client in response to the data request of the client when it is determined that there is a traffic burst trend; the cache area is used for the server to preferentially obtain data from the cache area to respond to other data requests when it is determined that there is a traffic burst trend.
[0133] Optionally, the determining unit 402 is configured to:
[0134] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing, it is determined that there is a traffic burst trend.
[0135] Optionally, the determining unit 402 is configured to:
[0136] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold, it is determined that there is a traffic burst trend.
[0137] Optionally, the determining unit 402 is configured to:
[0138] When the success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually decreasing and the time parameter of the data source responding to the data request of the server monitored in the current monitoring period is greater than or equal to a preset time parameter threshold, it is determined that there is a traffic burst trend.
[0139] Optionally, the determining unit 402 is configured to:
[0140] When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold and the time parameter of the data source in responding to the data request of the server monitored in the current monitoring period is greater than or equal to the preset time parameter threshold, it is determined that there is a traffic burst trend.
[0141] Optionally, the time parameter is used to characterize the proportion of requests whose response time of the data source in responding to the data request of the server exceeds a preset time threshold; then the preset time parameter threshold includes a timeout response request ratio threshold.
[0142] Optionally, the preset success rate threshold and / or the preset time parameter threshold supports dynamic adjustment.
[0143] Optionally, the device further comprises:
[0144] A feedback unit is used to obtain target data from the data source in response to a data request from a client when it is determined that there is no traffic burst trend, and to feed back the target data to the client without caching the target data.
[0145] Since the device 400 is a device corresponding to the method provided in the above method embodiment, the specific implementation of each unit of the device 400 is based on the same concept as the above method embodiment. Therefore, for the specific implementation of each unit of the device 400, reference can be made to the description part of the above method embodiment, which will not be repeated here.
[0146] The embodiment of the present application also provides a device, the device comprising a processor and a memory;
[0147] The processor is used to execute instructions stored in the memory so that the device executes the data processing method described in any one of the above method embodiments.
[0148] Reference below Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present application. The electronic devices in the embodiment of the present application may include but are not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable android devices), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs (televisions), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0149] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0150] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0151] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 509, or installed from the storage device 508, or installed from the ROM502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present application are executed.
[0152] The electronic device provided in the embodiment of the present application and the method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.
[0153] An embodiment of the present application provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any of the above embodiments is implemented.
[0154] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0155] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.
[0156] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0157] The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device executes the method.
[0158] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0159] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0160] The units involved in the embodiments described in this application may be implemented by software or hardware. The name of a unit / module does not limit the unit itself in some cases. For example, a voice data acquisition module may also be described as a "data acquisition module".
[0161] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0162] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. An embodiment of the present application provides a computer program product that, when the computer program product is run on a computer, causes the computer to execute the data processing method described in any one of the above method embodiments.
[0163] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0164] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
[0165] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A data processing method, characterized in that: The method comprises: Obtaining a data request success rate change trend monitored in the current monitoring period, wherein the data request success rate change trend is used to record a success rate of a data source responding to a data request from a server in chronological order of a preset number of monitoring periods; Determine whether there is a traffic burst trend according to the change trend of the data request success rate; When it is determined that there is a traffic burst trend, the server responds to the client's data request, obtains target data from the data source and feeds back the target data to the client, and synchronously caches the target data in the cache area of the server; the cache area is used for the server to preferentially obtain data from the cache area to respond to other data requests when it is determined that there is a traffic burst trend.
2. The method according to claim 1, characterized in that The determining whether there is a traffic burst trend according to the data request success rate change trend includes: When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing, it is determined that there is a traffic burst trend.
3. The method according to claim 1, characterized in that The determining whether there is a traffic burst trend according to the data request success rate change trend includes: When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold, it is determined that there is a traffic burst trend.
4. The method according to claim 1, characterized in that The determining whether there is a traffic burst trend according to the data request success rate change trend includes: When the success rate change trend indicates that the success rate of the data source responding to the data request of the server is gradually decreasing and the time parameter of the data source responding to the data request of the server monitored in the current monitoring period is greater than or equal to a preset time parameter threshold, it is determined that there is a traffic burst trend.
5. The method according to claim 1, characterized in that The determining whether there is a traffic burst trend according to the data request success rate change trend includes: When the data request success rate variation trend indicates that the success rate of the data source in responding to the data request of the server is gradually decreasing and the data request success rate monitored in the current monitoring period is less than or equal to the preset success rate threshold and the time parameter of the data source in responding to the data request of the server monitored in the current monitoring period is greater than or equal to the preset time parameter threshold, it is determined that there is a traffic burst trend.
6. The method according to claim 4 or 5, characterized in that: The time parameter is used to characterize the proportion of requests in which the response time of the data source to the data request of the server exceeds a preset time threshold; the preset time parameter threshold includes a timeout response request ratio threshold.
7. The method according to claim 5, characterized in that The preset success rate threshold and / or the preset time parameter threshold support dynamic adjustment.
8. The method according to claim 1, characterized in that The method further comprises: When it is determined that there is no traffic burst trend, the target data is obtained from the data source in response to a data request from the client, and the target data is fed back to the client without caching the target data.
9. A data processing device, characterized in that: The device comprises: An acquisition unit, used to acquire a data request success rate change trend monitored in a current monitoring period, wherein the data request success rate change trend is used to record a success rate of a data source responding to a data request of a server end in a time sequence of a preset number of monitoring periods; A determination unit, configured to determine whether there is a traffic burst trend according to the change trend of the data request success rate; A cache unit is used to synchronously cache the target data in a cache area of the server when the server obtains the target data from the data source and feeds back the target data to the client in response to a data request from the client when it is determined that there is a traffic burst trend; the cache area is used for the server to preferentially obtain data from the cache area in response to other data requests when it is determined that there is a traffic burst trend.
10. A device, characterized in that: The device comprises a processor and a memory; The processor is configured to execute instructions stored in the memory, so that the device performs the method according to any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that: The method comprises instructions for instructing a device to execute the method according to any one of claims 1 to 8.