Data processing method and apparatus, and electronic device

Abstract

Embodiments of the present application provide a data processing method and device and electronic equipment. The method comprises: receiving a data reading request sent by a target consumer, determining target data from a data source according to the data reading request, and determining the state of the target data. If the state of the target data is a first state, the target consumer is subjected to flow limiting processing according to a flow limiting rule. The present application reduces the case that the first state data is read in a large amount in a relatively concentrated time, thereby improving the efficiency of data reading and ensuring the normal implementation of transactions.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of data processing, and in particular, to a data processing method and device and electronic equipment. BACKGROUND

[0002] With the development of network technology, transactions are also more diversified, and different consumers can obtain relevant data to implement different transactions. As the number of transaction types and transactions increases, the amount of data required by consumers is also increasing. In order to better manage data, the application of distributed message queues has become more and more popular.

[0003] In a distributed message queue, a producer and a consumer can be included. The producer is used to generate data, and the consumer is used to consume the data generated by the producer, thereby implementing related transactions.

[0004] However, when the consumer consumes data, there is a situation of a sharp increase in the amount of data obtained by the consumer, resulting in most of the data obtained by the consumer not existing in the cache (i.e., the state of the data is a first state), that is, the data needs to be obtained directly from the disk. This greatly increases the demand for disk IO and increases the data request duration, thereby reducing the efficiency of data acquisition by the consumer and data writing by the producer, and affecting the normal implementation of transactions. SUMMARY

[0005] Embodiments of the present application provide a data processing method, device and electronic equipment to improve the efficiency of data acquisition by the consumer and data writing by the producer.

[0006] In a first aspect, embodiments of the present application provide a data processing method, comprising:

[0007] receiving a data read request sent by a target consumer;

[0008] determining target data from a data source according to the data read request, and determining a state of the target data;

[0009] if the state of the target data is a first state, performing a flow limiting process on the target consumer according to a flow limiting rule.

[0010] Optionally, the determination of the state of the target data comprises:

[0011] obtaining a data and data state mapping relationship;

[0012] determining the state of the target data according to the data and data state mapping relationship, wherein the state of the target data is a first state or a second state, the second state is a state corresponding to data stored in a cache, and the first state is a state corresponding to data in the data source other than data stored in the cache.

[0013] Optionally, the determining the state of the target data according to the mapping relationship between data and data states comprises:

[0014] If the target data is abnormal data in the data source, the state of the target data is determined according to the mapping relationship between data and data states, wherein the abnormal data in the data source is data stored in the data source for a duration exceeding a first preset duration threshold.

[0015] Optionally, the flow limiting processing of the target consumer according to the flow limiting rule comprises:

[0016] If the number of the first state data acquired by the target consumer within a second preset duration exceeds a preset number threshold, the target consumer is limited to acquire new first state data within a third preset duration.

[0017] After the third preset duration, the number of the first state data acquired by the target consumer within the second preset duration is re-accumulated.

[0018] Optionally, the method further comprises:

[0019] If the target consumer does not acquire new first state data within a fourth preset duration, the flow limiting processing of the target consumer is released.

[0020] Optionally, before the determining the state of the target data according to the mapping relationship between data and data states, the method further comprises:

[0021] scanning data in the data source and determining whether each scanned data has been cached;

[0022] For each target data, if the scanned target data has been cached, the state of the target data is set as a second state, and if the scanned target data has not been cached, the state of the target data is set as a first state, to obtain the mapping relationship between data and data states.

[0023] Optionally, the scanning data in the data source and determining whether each scanned data has been cached comprises:

[0024] the data in the data source is scanned at a timing, and from each scanned data, a data is selected every preset number to determine whether the data has been cached.

[0025] Optionally, the determining the target data from the data source according to the data reading request comprises:

[0026] determining whether the target consumer is a flow limiting consumer;

[0027] If not, target data is determined from a data source according to the data reading request.

[0028] In a second aspect, an embodiment of the present application provides a data processing apparatus, comprising:

[0029] a receiving module configured to receive a data reading request sent by a target consumer;

[0030] a processing module configured to determine target data from a data source according to the data reading request, and determine a state of the target data;

[0031] The processing module is further configured to, if the state of the target data is a first state, perform flow limiting processing on the target consumer according to a flow limiting rule.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, and a memory connected with the processor in communication;

[0033] The memory stores computer execution instructions;

[0034] The processor executes the computer execution instructions stored in the memory, and implements the data processing method as described in the first aspect and various possible designs of the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, implements the data processing method as described in the first aspect and various possible designs of the first aspect.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product, which comprises a computer program, and when a processor executes the computer program, implements the data processing method as described in the first aspect and various possible designs of the first aspect.

[0037] The embodiment of the present application provides a data processing method, device and electronic equipment, and after the above scheme, a data reading request sent by a target consumer can be received, then target data is determined from a data source according to the message reading request, and the state of the target data is determined, if the state of the target data is a first state, the target consumer is subjected to flow limiting processing according to a flow limiting rule. Through the control logic that determines the state of the data read by the consumer first, and then processes the data in different ways according to the state of the data, the separation control of the first state data and the second state data is realized in a software manner, the popularity of the separation control process of different state data is improved, and when the state of the data read by the consumer is determined to be the first state, the consumer is subjected to flow limiting processing according to the flow limiting rule, so that the first state data is read as much as possible in a relatively concentrated time, the occupation of the disk IO is reduced, and then the efficiency of data reading and the efficiency of data writing are improved, and the normal implementation of a transaction is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0039] Figure 1 The application scenario schematic diagram provided for the embodiment of the present application is shown in the figure.

[0040] Figure 2 The flow schematic diagram of the data processing method provided for the embodiment of the present application is shown in the figure.

[0041] Figure 3 The flow schematic diagram of the mapping relationship construction process provided for the embodiment of the present application is shown in the figure.

[0042] Figure 4 The principle schematic diagram of the mapping relationship construction process provided for the embodiment of the present application is shown in the figure.

[0043] Figure 5 The principle schematic diagram of the data processing method provided for the embodiment of the present application is shown in the figure.

[0044] Figure 6 The structure schematic diagram of the data processing device provided for the embodiment of the present application is shown in the figure.

[0045] Figure 7 The hardware structure schematic diagram of the electronic equipment provided for the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0046] With reference to the drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0047] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can also include other order instances in addition to those illustrated or described. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0048] First, the terms related to the present application are explained:

[0049] Distributed message queue: a distributed message system based on a queue model, which can provide data sending, storage, consumption and the like.

[0050] Producer: used for producing data, which can be a data producer of different transactions or different application systems.

[0051] Consumer: used for consuming data generated by the producer, which can be a message processing party of different transactions or different application systems. For example, a keyboard input program can be a producer, a screen display program can be a consumer, and characters are data produced by the keyboard input program, which can be read by the screen display program and displayed.

[0052] The application scenarios and inventive concepts of the present application are explained below.

[0053] In the process of distributed message queue application, the producer is used to generate data, and the consumer is used to consume data generated by the producer, thereby realizing related transactions. In order to improve the processing efficiency of data, the producer can store the data into a data source after generating the data. In order to improve the data reading efficiency, the data in the data source can be stored into a cache before the data is applied to realize related transactions, and then the data is read from the cache and consumed to realize related transactions.

[0054] However, in some scenarios, there may be a surge in consumers, or consumers that have been stopped for a long time suddenly start consuming. For example, in the scenario of a holiday promotion activity, there are many users participating in the promotion activity, and users who generally do less shopping may also participate in the promotion activity, resulting in a surge in consumers who need to implement promotion-related transactions (such as ordering, payment, refund, adding to the shopping cart, etc.). Or, in the scenario of a large holiday program live broadcast, there are many users participating in the program viewing, and users who generally do not watch live broadcasts frequently may also participate in the program live broadcast, resulting in a surge in consumers who need to implement program live broadcast-related transactions (such as liking, commenting, sharing the live broadcast link, and reviewing, etc.). After the surge in consumers, the amount of data consumed also increases significantly, resulting in most of the data consumed being data that does not exist in the cache (i.e., the data state is the first state), which needs to be obtained from the memory through disk IO for consumption. The demand for disk IO increases significantly, and the data request time is also increased, thereby reducing the efficiency of the consumer obtaining data and the efficiency of the producer writing data, affecting the stability of the transaction.

[0055] In addition, in the prior art, there is also a way of artificially controlling the consumer to read data to limit the total amount of data consumed by the consumer, or increasing the cache by increasing the hardware, which has a large control cost and is not highly popular.

[0056] Based on the above technical problems, the present application determines the state of the data read by the consumer first, and then uses different processing methods to process the data according to the state of the data, to realize the separation control of the first state data and the second state data in a software manner, improve the popularity of the separation control process of different state data, and when the state of the data read by the consumer is determined to be the first state, the consumer is processed according to the flow limiting rule, which reduces the amount of first state data read in a relatively concentrated time as much as possible, thereby improving the efficiency of data reading. At the same time, due to sufficient disk IO, the efficiency of the producer writing data can also be improved, the stability of the transaction implementation process is improved, and the normal implementation of the transaction is ensured.

[0057] Figure 1 The application scenario provided by the embodiment of the present application is shown in the following figure: Figure 1As shown, in this embodiment, the target consumer, the server and the data source can be included. The target consumer can be multiple, and can implement corresponding transactions by consuming data in the data source. For example, the target consumer can be an ordering system, a payment system and an after-sales system, the ordering system implements an ordering transaction by consuming data in the data source, the payment system implements a payment transaction by consuming data in the data source, and the after-sales system implements an order signing, return, exchange and other transactions by consuming data in the data source. Optionally, when the target consumer needs to consume data, the target consumer can first generate a data reading request, and then send the data reading request to the server, so that the server obtains data from the data source according to the data reading request, and returns the data to the target consumer for consumption. The server can also obtain data corresponding to the target consumer from the data source every preset time interval, and send the obtained data to the corresponding target consumer for consumption.

[0058] In the data source, the data can be data generated by the producer. After generating the data, the producer can store the generated data directly into the data source for subsequent consumption by the consumer. For example, the data source can be a data storage module CommitLog.

[0059] In addition, the data in the data source can have two states. One is the second state, that is, the state corresponding to the data stored in the cache. The other is the first state, that is, the state corresponding to the data in the data source except the data stored in the cache, that is, the state corresponding to the data not stored in the cache, for example, the data stored in the memory. Since the running speed of the cache is much faster than that of the memory, the reading speed of the data in the second state is obviously higher than that of the data in the first state.

[0060] In addition, after the server receives the data reading request sent by the target consumer, the server can determine the target data from the data source according to the data reading request. Since the target data can be data in the first state or data in the second state, after obtaining the target data, the state of the target data can be determined first, and then different processing methods can be used according to the determined state of the target data. Optionally, if the state of the target data is the second state, the target data can be directly sent to the target consumer for consumption to implement the related transaction. If the state of the target data is the first state, the target consumer can be throttled according to the throttling rule to reduce the number of target consumers reading the data in the first state, thereby improving the reading efficiency of the data, and separating the control of the data in the first state and the data in the second state in a software manner, thereby improving the popularity of the separation control process of the data in different states.

[0061] The server can be a separately deployed server or a server cluster.

[0062] The technical solutions of the present application will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes can not be described in detail in some examples.

[0063] Figure 2 The flowchart of the data processing method provided in the embodiment of the present application can be executed by a server. As shown in the figure, the method of the present embodiment can include: Figure 2

[0064] S201: receiving a data reading request sent by a target consumer.

[0065] In the present embodiment, the target consumer may need to obtain corresponding data when implementing a related transaction, and then implement the related transaction according to the obtained data. Optionally, the target consumer can obtain the corresponding data by sending a data reading request to the server. In addition to the source address, destination address and other information, the data reading request can also contain identification information of the data to be obtained, and the server can determine the data required by the target consumer according to the identification information.

[0066] In addition, the server can also send newly updated data to the corresponding target consumer for consumption processing every preset time interval, thereby implementing the related transaction. The preset time interval can be customized according to the actual application scenario, and will not be limited in detail here.

[0067] S202: determining target data from the data source according to the data reading request, and determining the state of the target data.

[0068] In the present embodiment, the server can determine the target data from the data source according to the identification information contained in the data reading request after receiving the data reading request sent by the target consumer. After determining the target data, the state of the target data can be further determined.

[0069] Optionally, the identification information contained in the data reading request can be the address of the target data or the index number corresponding to the target data.

[0070] Further, the state of the data can be divided into two types, one is the first state, and the other is the second state. Determining the state of the target data can specifically include:

[0071] Obtaining a mapping relationship between data and data state.

[0072] ​According to the data and data state mapping relationship, the state of the target data is determined, wherein the state of the target data is a first state or a second state, the second state is a state corresponding to data stored in the cache, and the first state is a state corresponding to data other than the data stored in the cache.

[0073] Specifically, when the state of the target data is determined, the data and data state mapping relationship constructed in advance can be acquired first, and then the state corresponding to the target data is searched in the data and data state mapping relationship.

[0074] Optionally, the data source can include a plurality of sub-components, each sub-component corresponding to a sub-component name. When the data generated by the storage producer is stored, whether the existing sub-component can store the generated data can be determined according to the size of the generated data and the remaining space of the existing sub-component. If yes, the generated data can be directly stored in the existing sub-component. If not, a new sub-component can be generated, and the remaining memory of the existing component and the new sub-component are combined to store the generated data. Further, when the data is stored in the sub-component, the data is stored in units of pages, that is, the data can be divided into a plurality of pages in units of pages, each target data can correspond to one or more pages of data, and the size of the page can be configured according to the actual situation of the system. Correspondingly, the data and data state mapping relationship can be stored in the form of a mapping Map key-value pair, Key is a sub-component name, Value is a Byte array, and each element in the Byte array represents the state of one page of data. For example, 1 can indicate that the state of the data is the second state, and 0 can indicate that the state of the data is the first state. The mapping Map key-value pair can be (CommitLog1, Byte[1,1,1,1,0,0]), which indicates that the sub-component name is CommitLog1, and there are six pages of data in CommitLog1, the states of the first four pages of data are the second state, and the states of the last two pages of data are the first state.

[0075] In summary, by determining the state of the target data according to the data and data state mapping relationship, the query efficiency of the state of the target data is improved.

[0076] S203: If the state of the target data is the first state, the target consumer is throttled according to the throttling rule.

[0077] In this embodiment, if it is determined that the state of the target data is the first state, it indicates that the target data does not exist in the cache. Since the reading efficiency of the data not in the cache is low, when the target consumer reads the data in the first state, the target consumer can be throttled according to the throttling rule, so as to reduce the number of the target consumer reading the data in the first state, thereby improving the reading efficiency of the data and ensuring the stable operation of the system.

[0078] In addition, if it is determined that the state of the target data is the second state, the target data can be directly read from the cache and sent to the target consumer, so that the target consumer implements a corresponding transaction according to the target data.

[0079] After the above scheme is adopted, the data read request sent by the target consumer can be received, and then the target data is determined from the data source according to the message read request, and the state of the target data is determined. If the state of the target data is the first state, the target consumer is throttled according to the throttling rule. By first determining the state of the data read by the consumer, and then processing the data in different ways according to the state of the data, the control logic of the first state data and the second state data is realized in a software manner, improving the popularity of the separation control process of different state data. When the state of the data read by the consumer is determined to be the first state, the consumer is throttled according to the throttling rule, which reduces the amount of first state data read in a relatively concentrated time as much as possible, reduces the occupation of disk IO, and further improves the efficiency of data read and data write, and ensures the normal implementation of the transaction.

[0080] Based on the method, Figure 2 The embodiments of the present specification also provide some specific embodiments of the method, which are described below.

[0081] In another embodiment, before the state of the target data is determined according to the data and data state mapping relationship, the method can further include:

[0082] Scanning the data in the data source, and determining whether each scanned data has been cached.

[0083] For each target data, if the scanned target data has been cached, the state of the target data is set to the second state, and if the scanned target data has not been cached, the state of the target data is set to the first state, to obtain the data and data state mapping relationship.

[0084] In this embodiment, the data and data state mapping relationship can be constructed by scanning the data in the data source and judging whether each scanned data has been cached. When scanning the data in the data source, the scanning can be performed in units of pages, and it can be determined whether each data exists in the cache in units of pages. If it exists in the cache, the state of the data can be set to the second state, and if it does not exist in the cache, the state of the data can be set to the first state, and then the data and data state mapping relationship is obtained.

[0085] Further, Figure 3A flowchart of the mapping relationship construction process provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, in this embodiment, the mincore function can be used to determine whether the data exists in the cache. Specifically, the vec parameter in the mincore(void*start, size_t length, unsigned char*vec) function can be used to determine whether the data has been cached. Among them, each page (i.e., each page) of data can be judged as a minimum unit. If the return value of the mincore function is 0, it indicates that the mincore function is executed successfully. Then, it can be checked whether the data under the specified index in vec is 0. If it is not 0, it can be determined that the page data exists in the cache, that is, the data status is the second state; if the vec data under the specified index is 0, it can be determined that the page data does not exist in the cache, that is, the data status is the first state. If the return value of the mincore function is not 0, it indicates that there is a problem with the function call, and a function exception prompt can be generated to remind relevant personnel that there is a problem with the function call, so that the exception can be handled in time.

[0086] In summary, by using a function to determine the state of data, the efficiency of determining the state of data is improved, and the cost of determining the state of data is reduced.

[0087] Furthermore, scanning the data in the data source and determining whether each scanned data has been cached may specifically include:

[0088] The data in the data source is scanned periodically, and a data item is selected every preset number of items from the scanned data to determine whether it has been cached.

[0089] Specifically, since the producer generates new data at irregular intervals and stores the newly generated data in the data source, and the data in the cache is also constantly changing, the data in the data source can be scanned once every preset time period to determine whether the data in the data source has been cached, and the scanning interval can be dynamically configured based on the specific transaction. By scanning the data in the data source at regular intervals, the server's computing power is reduced, thereby saving the server's computing resources. Of course, the data in the data source can also be scanned in real time, which will not be further specified here.

[0090] Optionally, the data in the scanned data source is data in an abnormal state, that is, data other than data in a normal state in the data source.

[0091] In addition, since the data in the data source is relatively large, the proportion of the occupied heap memory is large, in order to save storage space, a data sampling strategy can be adopted, from each data scanned, every preset number of data is selected to determine whether the data has been cached, that is, every preset number of pages of data is selected, and it is determined whether the selected page of data has been cached, and then the data and data state mapping relationship can be constructed according to the data caching condition.

[0092] Exemplarily, Figure 4 The principle schematic diagram of the mapping relationship construction process provided by the embodiment of the application is shown in Figure 4 In the embodiment, the data source can include a plurality of sub-components, which can be CommitLog1, CommitLog2, CommitLog3, and CommitLogn. The abnormal data in all CommitLogs is scanned by a timing task thread, and the mincore function is called to determine whether the data in each sub-component is in the cache. The process of determining whether the data is in the cache can be performed every 3 pages. Then the data and data state mapping relationship can be constructed according to the determination result. The data and data state mapping relationship can be stored in the form of a mapping Map key-value pair, the key is the sub-component name (that is, it can be CommitLog1, CommitLog2, or CommitLog3, etc.), and the value is a Byte array, each element in the Byte array represents the state of a page of data.

[0093] In summary, by selecting one data every preset number to determine whether it has been cached, the sampling step is increased, the determination frequency is reduced, the size of the data occupied memory is reduced, and the efficiency of determining the data state is improved.

[0094] In another embodiment, the state of the target data can be determined according to the data and data state mapping relationship, and the state of the target data can be determined according to the data and data state mapping relationship.

[0095] If the target data is abnormal data in the data source, the state of the target data is determined according to the data and data state mapping relationship, wherein the abnormal data in the data source is data stored in the data source for more than a first preset time threshold.

[0096] In the embodiment, before the state of the target data is determined according to the data and data state mapping relationship, it can be determined whether the target data is abnormal data in the data source, and if the target data is abnormal data in the data source, the subsequent process of determining the state of the target data according to the data and data state mapping relationship is executed. If the target data is not abnormal data in the data source, the target data can be directly obtained, and the obtained target data is sent to the target consumer, so that the target consumer can implement the corresponding transaction according to the target data.

[0097] The setting process of the abnormal data can be set according to actual transaction requirements. Optionally, the data corresponding to the transaction can be filtered according to the execution priority or importance of the transaction to obtain a certain proportion of data as normal data. This part of data should be normally read by the consumer at the system logic level, and even if the disk is directly read, it will not be controlled. The remaining data in the data source can be used as abnormal data (in addition, the abnormal data can also be referred to as logical cold data, and the normal data can also be referred to as logical hot data).

[0098] Optionally, the data can also be divided according to the storage duration of the data in the data source to obtain abnormal data and normal data. For example, if the storage duration of the data in the data source exceeds a first preset duration threshold, it indicates that the frequency of use of this part of data is low, and this data can be set as abnormal data. The first preset duration threshold can be set according to actual application scenarios, and will not be limited in detail here.

[0099] In summary, since most of the data read by the consumer is normal data, the number of times of judging the cold second state of the data is significantly reduced, and the efficiency of data reading is improved. The process of judging whether the data is in a normal state is much more efficient than the process of judging the cold second state of the data, so the efficiency of data reading is further improved.

[0100] In another embodiment, the flow limiting processing of the target consumer according to the flow limiting rule can specifically include:

[0101] If the number of first state data obtained by the target consumer within a second preset duration exceeds a preset number threshold, the target consumer is limited to obtain new first state data within a third preset duration.

[0102] After the third preset duration, the number of first state data obtained by the target consumer within the second preset duration is re-accumulated.

[0103] In this embodiment, after determining the target data to be obtained by the target consumer, the state of the target data can be determined first, that is, it is determined whether the target data is in a first state or a second state. If it is determined that the state of the target data is the second state, the server can directly obtain the target data from the cache and send the target data to the target consumer, so that the target consumer can implement the corresponding transaction according to the target data.

[0104] Optionally, if it is determined that the state of the target data is the first state, the number of the first state data obtained by the target consumer within the second preset time period can be determined first. If the number does not exceed the preset number threshold, the server can directly obtain the target data from the memory and send the target data to the target consumer, so that the target consumer can implement the corresponding transaction according to the target data. If the number exceeds the preset number threshold, it indicates that the target consumer reads the first state data too frequently, which may reduce the reading efficiency of the data. Therefore, the target consumer can be limited to obtain new first state data within a third preset time period, or the target consumer can be limited to obtain new data (i.e., not limited to data in the first state or the second state), and after the third preset time period, the number of the first state data obtained by the target consumer within the second preset time period is accumulated again. The second preset time period, the third preset time period, and the number threshold can be set according to actual application scenarios, and will not be limited in detail here. In addition, the second preset time period and the third preset time period can be the same or different.

[0105] For example, the second preset time period can be 5s, and the third preset time period can be 6s, that is, the number of the first state data obtained within 5s can be determined. If the number exceeds the number threshold, the target consumer is not allowed to obtain new first state data within 6s, and after 6s, the number of the first state data obtained by the target consumer is cleared and the number of the first state data obtained by the target consumer is accumulated again.

[0106] In addition, the total amount of resources occupied by the data can also be used as a dimension for judgment, that is, whether the total amount of resources occupied by the first state data obtained by the target consumer within the second preset time period exceeds the preset resource total amount threshold.

[0107] In addition, the method can further include:

[0108] If the target consumer does not obtain new first state data within a fourth preset time period, the flow limiting processing on the target consumer is released.

[0109] Specifically, if the target consumer does not obtain new first state data within a period of time, it indicates that the frequency of the target consumer reading the first state data is low, and the flow limiting processing on the target consumer can be released. The fourth preset time period can be the same as the second preset time period or the third preset time period, or can be different, and the specific value of the fourth preset time period can be set according to actual application scenarios.

[0110] Optionally, the target consumer can be monitored in a single consumer dimension, and can also be monitored in a global consumer dimension. When monitored in the global consumer dimension, the total amount of data read by all consumers or the total amount of resources occupied by the data can be judged to determine whether all consumers need to be throttled.

[0111] In addition, the consumer can also be throttled in an artificial control manner, that is, the number threshold corresponding to the first state data of a single or all consumers can be flexibly configured, thereby improving the flexibility of throttling the consumer and further improving the efficiency of data reading.

[0112] In summary, by throttling the consumer when the data read by the consumer is the first state data, the situation that the consumer always reads the first state data is prevented, the frequency of the consumer reading the first state data is reduced, the efficiency of the consumer reading data is improved, and the stable operation of each transaction is ensured.

[0113] In another embodiment, the target data is determined from the data source according to the data read request, including:

[0114] It is judged whether the target consumer is a throttled consumer.

[0115] If not, the target data is determined from the data source according to the data read request.

[0116] In this embodiment, before the target data is determined from the data source, it can be judged whether the target consumer is a throttled consumer. If not, the target data can be directly determined from the data source according to the data read request. If yes, the target consumer can be throttled according to the throttling rule.

[0117] Further, when it is judged whether the target consumer is a throttled consumer, it can be judged by judging whether the target consumer is in the throttled consumer list. If the target consumer is in the throttled consumer list, it can be determined that the target consumer is a throttled consumer. If the target consumer is not in the throttled consumer list, it can be determined that the target consumer is not a throttled consumer.

[0118] In summary, by judging whether the target consumer is a throttled consumer first, and then taking different processing procedures according to the judgment result, the flexibility of data acquisition is improved.

[0119] For example, Figure 5 The principle diagram of the data processing method provided by the embodiment of the present application is as follows: Figure 5As shown, in the embodiment, the target consumer can first send a data reading request to the server. After receiving the data reading request, the server can determine whether the target consumer is a flow limiting consumer. If yes, the flow limiting process is directly entered for flow limiting processing. Otherwise, the target data is determined through the data reading request, and then it is determined whether the target data is abnormal data or normal data (in addition, abnormal data can also be referred to as logical cold data, and normal data can also be referred to as logical hot data). If it is determined that the target data is abnormal data, the state of the target data can be further determined to be the first state or the second state according to the constructed data and data state mapping relationship. The data and data state mapping relationship can be stored in the form of a mapping Map key-value pair. The Key is a sub-component name, the Value is a Byte array, and each element in the Byte array represents the state of a page of data. In addition, after determining that the state of the target data is the first state, the target consumer can be set as a flow limiting consumer, and the flow limiting process is entered for flow limiting processing. If it is determined that the state of the target data is the second state, the target data can be sent to the target consumer for processing, and then related transactions are implemented.

[0120] In addition, the flow limiting processing rule can be to cancel the flow limiting state of the flow limiting consumer that has not read the first state data for a long time (the specific time length can be set according to actual conditions), to accumulate the total amount of the first state data read by the consumer, and the like. Optionally, the flow limiting rule can also be self-defined in an artificial control manner, so as to improve the flexibility of the flow limiting rule setting.

[0121] Based on the same idea, the embodiment of the present specification also provides a device corresponding to the above method, Figure 6 The structure diagram of the data processing device provided by the embodiment of the present application is as follows: Figure 6 As shown, the device provided by the embodiment can include:

[0122] The receiving module 601 is configured to receive a data reading request sent by a target consumer.

[0123] The processing module 602 is configured to determine target data from a data source according to the data reading request, and determine the state of the target data.

[0124] The processing module 602 is further configured to, if the state of the target data is the first state, perform flow limiting processing on the target consumer according to a flow limiting rule.

[0125] According to the above scheme, the data reading request sent by the target consumer can be received, the target data is determined from the data source according to the message reading request, and the state of the target data is determined. If the state of the target data is the first state, the target consumer is subjected to flow limiting processing according to the flow limiting rule. The control logic of determining the state of the data read by the consumer first, and then processing the data in different ways according to the state of the data, realizes the separation control of the first state data and the second state data in a software manner, improves the popularity of the separation control process of different state data, and when the state of the data read by the consumer is determined to be the first state, the consumer is subjected to flow limiting processing according to the flow limiting rule, so as to reduce the reading of the first state data as much as possible, and then improve the efficiency of data reading, and ensure the normal implementation of the transaction.

[0126] In addition, in another embodiment, the processing module 602 is further configured to:

[0127] obtain a data and data state mapping relationship.

[0128] determine the state of the target data according to the data and data state mapping relationship, wherein the state of the target data is a first state or a second state, the second state is a state corresponding to data stored in the cache, and the first state is a state corresponding to data in the data source other than the data stored in the cache.

[0129] Further, the processing module 602 is further configured to:

[0130] If the target data is abnormal data in the data source, the state of the target data is determined according to the data and data state mapping relationship, wherein the abnormal data in the data source is data stored in the data source for a duration longer than a first preset duration threshold.

[0131] In addition, in another embodiment, the processing module 602 is further configured to:

[0132] If the number of first state data obtained by the target consumer within a second preset duration exceeds a preset number threshold, the target consumer is limited to obtain new first state data within a third preset duration.

[0133] After the third preset duration, the number of first state data obtained by the target consumer within the second preset duration is re-accumulated.

[0134] In addition, the processing module 602 is further configured to:

[0135] If the target consumer does not obtain new first state data within a fourth preset duration, the flow limiting processing on the target consumer is released.

[0136] Further, in another embodiment, the processing module 602 is further configured to:

[0137] scan the data in the data source, and determine whether each scanned data has been cached.

[0138] For each target data, if the scanned target data has been cached, set the state of the target data as a second state, and if the scanned target data has not been cached, set the state of the target data as a first state, to obtain the data and data state mapping relationship.

[0139] Further, the processing module 602 is further configured to:

[0140] scan the data in the data source at a timing, and from each scanned data, select a data every preset number to determine whether the data has been cached.

[0141] Further, in another embodiment, the processing module 602 is further configured to:

[0142] determine whether the target consumer is a throttling consumer.

[0143] If not, determine target data from the data source according to the data read request.

[0144] The apparatus provided by the embodiments of the present application can implement the method of the embodiments as shown in Figure 2 The implementation principle and technical effects are similar, and will not be described here.

[0145] Figure 7 The hardware structure schematic diagram of the electronic device provided by the embodiments of the present application is shown in Figure 7 The device 700 provided by the embodiments includes a processor 701 and a memory in communication connection with the processor. The processor 701 and the memory 702 are connected through a bus 703.

[0146] In the specific implementation process, the processor 701 executes the computer execution instructions stored in the memory 702, so that the processor 701 executes the method in the above method embodiments.

[0147] The specific implementation process of the processor 701 can refer to the above method embodiments, and the implementation principle and technical effects are similar, which will not be described here.

[0148] In the above Figure 7In the illustrated embodiment, it is to be understood that the processor can be a central processing unit (CPU), but can also be other general purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the disclosed method can be directly embodied as hardware processor execution or a combination of hardware and software modules in the processor.

[0149] The memory can include a high-speed RAM memory and can also include a non-volatile storage NVM, such as at least one disk memory.

[0150] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0151] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, the data processing method of the above method embodiment is realized.

[0152] The embodiment of the present application also provides a computer program product, including a computer program, when the processor executes the computer program, the data processing method as described above is realized.

[0153] The above computer readable storage medium, the readable storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0154] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can exist as discrete components in the device.

[0155] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the foregoing storage medium includes: ROM, RAM, magnetic disk or optical disk and various storage medium that can store program codes.

[0156] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Receive data read requests sent by target consumers; determining target data from a data source according to the data read request, and determining a state of the target data; If the state of the target data is a first state, the target consumer is subjected to flow limiting processing according to the flow limiting rule, where the first state is a state corresponding to the data in the data source excluding the data stored in the cache; The flow limiting processing of the target consumer according to the flow limiting rule includes: if the amount of first state data acquired by the target consumer within the second preset time period exceeds a preset amount threshold, restricting the target consumer from acquiring new first state data within a third preset time period; After the third preset time period, the amount of first status data acquired by the target consumer within the second preset time period is accumulated again.

2. The method according to claim 1, characterized in that Determining the state of the target data includes: Get the mapping relationship between data and data status; The state of the target data is determined according to the mapping relationship between the data and the data state, wherein the state of the target data is a first state or a second state, and the second state is a state corresponding to the data stored in the cache.

3. The method according to claim 2, characterized in that The determining the state of the target data according to the mapping relationship between the data and the data state includes: If the target data is abnormal data in the data source, the state of the target data is determined according to a mapping relationship between data and data state, wherein the abnormal data in the data source is data whose storage time in the data source exceeds a first preset time threshold.

4. The method according to claim 3, characterized in that The method further comprises: If the target consumer does not obtain new data in the first state within a fourth preset time period, the flow limiting process for the target consumer is released.

5. The method according to claim 2, characterized in that Before determining the state of the target data according to the mapping relationship between the data and the data state, the method further includes: Scan the data in the data source and determine whether the scanned data has been cached; For each target data, if the scanned target data has been cached, the state of the target data is set to the second state; if the scanned target data has not been cached, the state of the target data is set to the first state, and a mapping relationship between the data and the data state is obtained.

6. The method according to claim 5, characterized in that Scanning the data in the data source and determining whether each scanned data has been cached includes: The data in the data source is scanned periodically, and a data item is selected every preset number of items from the scanned data to determine whether it has been cached.

7. The method according to any one of claims 1 to 3, characterized in that Determining target data from a data source according to the data read request includes: Determining whether the target consumer is a flow-limited consumer; If not, the target data is determined from the data source according to the data read request.

8. A data processing device, characterized in that: include: The receiving module is used to receive data reading requests sent by target consumers; a processing module, configured to determine target data from a data source according to the data read request, and determine a state of the target data; The processing module is further configured to perform flow limiting processing on the target consumer according to a flow limiting rule if the state of the target data is a first state, wherein the first state is a state corresponding to data in the data source excluding data stored in the cache; The processing module is further specifically configured to: if the amount of first status data acquired by the target consumer within the second preset time period exceeds a preset threshold, restrict the target consumer from acquiring new first status data within a third preset time period; After the third preset time period, the amount of first status data acquired by the target consumer within the second preset time period is accumulated again.

9. An electronic device, characterized in that: include: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the data processing method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

Images