Data Processing Method, Apparatus, Device, Medium and Program Product

Abstract

The present application provides a data processing method, apparatus, device, medium and program product. By obtaining the capacity information and data usage information of a first buffer area, where the first buffer area is used to store data generated by a data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of data by each usage node in a data usage end; according to the capacity information and the data usage information, data that meets a preset transfer condition is transferred from the first buffer area to a historical data storage area. This solves the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the data usage node with the weakest data processing ability, resulting in a relatively low overall data processing ability of the system. At the same time, data can be called from the historical data storage area, making it more convenient to add new data usage nodes and also improving the data security of each data usage node.

Description

Technical Field

[0001] The present application relates to the field of computer data processing, and in particular, to a data processing method, apparatus, device, medium, and program product. Background Art

[0002] When processing computer data, since the data usage end and the data generation end are often not synchronized in time, and the processing speeds of the two ends are also inconsistent, a buffer is required at this time to buffer the data and achieve asynchronous data transfer processing.

[0003] With the continuous expansion of the network of the data usage end, there will be multiple usage nodes that need to use the data in the buffer, but the processing capabilities of each usage node are different. This makes the data processing efficiency of the entire system subject to the usage node with the lowest processing rate, or simply referred to as the bottleneck problem.

[0004] In the case where the buffer capacity is limited, enabling the data generation end and the data usage end not to be restricted by the node with the weakest data processing ability in the overall data transmission efficiency has become an urgent technical problem to be solved. Summary of the Invention

[0005] The present application provides a data processing method, apparatus, device, medium, and program product to solve the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the data usage node with the weakest data processing ability, resulting in a low overall data processing ability of the system.

[0006] In a first aspect, the present application provides a data processing method, including:

[0007] Obtain the capacity information and data usage information of a first buffer, where the first buffer is used to store data generated by a data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end;

[0008] According to the capacity information and the data usage information, transfer the data that meets the preset transfer condition from the first buffer to the historical data storage area.

[0009] In a possible design, the preset transfer condition includes a first transfer condition, and the first transfer condition is used to meet the data production efficiency requirement of the data generation end. Correspondingly, according to the capacity information and the data usage information, transferring the data that meets the preset transfer condition from the first buffer to the historical data storage area includes:

[0010] Determine the remaining capacity information according to the capacity information, where the remaining capacity information includes: remaining capacity, and / or, the change rate of the remaining capacity;

[0011] If the remaining capacity is less than or equal to a preset capacity threshold, and / or the change rate is greater than or equal to a preset change threshold, then according to the data usage information, transfer the data to be transferred from the first buffer to the historical data storage area at a preset period.

[0012] In a possible design, the preset transfer condition includes a second transfer condition, which is used to meet the data usage requirements of high-speed data usage nodes, and the data usage rate of the high-speed data usage nodes is greater than or equal to a preset rate. Correspondingly, according to the capacity information and the data usage information, transferring the data that meets the preset transfer condition from the first buffer to the historical data storage area includes:

[0013] Determine the proportion of the used data according to the data usage information, where the used data is the data used by at least one of the nodes;

[0014] When the proportion is greater than or equal to a preset proportion, if the usage times of the used data are greater than or equal to a preset usage times, then transfer the data to be transferred from the first buffer to the historical data storage area according to the data usage information.

[0015] Optionally, the data to be transferred includes used data, where the used data is the data used by at least one of the nodes.

[0016] Optionally, the data to be transferred includes all the data in the first buffer.

[0017] In a possible design, the preset transfer condition includes a third transfer condition, which is used to clean the first buffer in time after the data is used by all the usage nodes. Correspondingly, according to the capacity information and the data usage information, transferring the data that meets the preset transfer condition from the first buffer to the historical data storage area further includes:

[0018] If all the data in the first buffer has been used up by all the data usage nodes, then transfer all the data to the historical data storage area.

[0019] In a possible design, the data processing method further includes:

[0020] Obtain the historical data usage requests of the data usage nodes in the data usage terminal;

[0021] Determine target historical data in the historical data storage area according to the historical data usage request;

[0022] Send the target historical data to the data usage node.

[0023] In a possible design, the sending the target historical data to the data usage node includes:

[0024] If the remaining capacity of the first buffer meets the storage requirement of the target historical data, store the target historical data in the first buffer for use by the data usage node.

[0025] Optionally, the sending the target historical data to the data usage node includes:

[0026] If the remaining capacity of the first buffer does not meet the storage requirement of the target historical data, store the target historical data in the second buffer for use by the data usage node.

[0027] In a possible design, the sending the target historical data to the data usage node includes:

[0028] Store the target historical data in the second buffer for use by the data usage node.

[0029] In a second aspect, the present application provides a data processing device, including:

[0030] An acquisition module, configured to acquire capacity information and data usage information of a first buffer, where the first buffer is used to store data generated by a data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end;

[0031] A processing module, configured to transfer data that meets a preset transfer condition from the first buffer to the historical data storage area according to the capacity information and the data usage information.

[0032] In a possible design, the preset transfer condition includes a first transfer condition, and the first transfer condition is used to meet the data production efficiency requirement of the data generation end. Correspondingly, the processing module is specifically configured to:

[0033] Determine remaining capacity information according to the capacity information, where the remaining capacity information includes: remaining capacity, and / or, the change rate of the remaining capacity;

[0034] If the remaining capacity is less than or equal to a preset capacity threshold, and / or the change rate is greater than or equal to a preset change threshold, then according to the data usage information, transfer the data to be transferred from the first buffer to the historical data storage area at a preset period.

[0035] In a possible design, the preset transfer condition includes a second transfer condition, which is used to meet the data usage requirements of high-speed data usage nodes, and the data usage rate of the high-speed data usage nodes is greater than or equal to a preset rate. Correspondingly, the processing module is specifically configured to:

[0036] According to the data usage information, determine the proportion of the used data, where the used data is the data used by at least one of the nodes;

[0037] When the proportion is greater than or equal to a preset proportion, if the usage times of the used data are greater than or equal to a preset usage times, then according to the data usage information, transfer the data to be transferred from the first buffer to the historical data storage area.

[0038] Optionally, the data to be transferred includes used data, where the used data is the data used by at least one of the nodes.

[0039] Optionally, the data to be transferred includes all the data in the first buffer.

[0040] In a possible design, the preset transfer condition includes a third transfer condition, which is used to clean the first buffer in time after the data is used by all the usage nodes. Correspondingly, the processing module is specifically configured to:

[0041] If all the data in the first buffer has been used up by all the data usage nodes, then transfer all the data to the historical data storage area.

[0042] In a possible design, the obtaining module is further configured to obtain the historical data usage requests of the data usage nodes in the data usage end;

[0043] The processing module is further configured to determine target historical data in the historical data storage area according to the historical data usage request;

[0044] The processing module is further configured to send the target historical data to the data usage node.

[0045] In a possible design, the processing module is further configured to:

[0046] If the remaining capacity of the first buffer meets the storage requirement of the target historical data, store the target historical data in the first buffer for use by the data usage node.

[0047] Optionally, the processing module is further configured to send the target historical data to the data usage node, specifically including:

[0048] If the remaining capacity of the first buffer does not meet the storage requirement of the target historical data, store the target historical data in the second buffer for use by the data usage node.

[0049] In a possible design, the processing module is further configured to send the target historical data to the data usage node, specifically including:

[0050] Store the target historical data in the second buffer for use by the data usage node.

[0051] In a third aspect, the present application provides a data processing system, including: a data generation end, a data caching end, and a data usage end; wherein,

[0052] The data generation end is configured to generate data to be used and store the data to be used in the data caching end;

[0053] The data caching end includes a first buffer and a historical data storage area. The historical data storage area is configured to store the data in the first buffer that meets a preset transfer condition. Optionally, the data caching end further includes a second buffer. The second buffer is configured to store the target historical data from the historical data storage area to the second buffer for use by the data usage node when the data usage node requests to use historical data;

[0054] The data usage end includes at least one data usage node, and the data usage node is configured to call the data in the first buffer or the historical data storage area;

[0055] The data generation end or the data caching end is further configured to implement any possible data processing method provided in the first aspect.

[0056] In a fourth aspect, the present application provides an electronic device, including:

[0057] A memory, configured to store program instructions;

[0058] A processor, configured to call and execute the program instructions in the memory and execute any possible data processing method provided in the first aspect.

[0059] Fifth aspect, the present application provides a storage medium, in which a computer program is stored, and the computer program is used to execute any possible data processing method provided in the first aspect.

[0060] Sixth aspect, the present application further provides a computer program product, including a computer program, which when executed by a processor implements any possible data processing method provided in the first aspect.

[0061] The present application provides a data processing method, device, equipment, medium and program product. By obtaining the capacity information and data usage information of the first buffer, the first buffer is used to store the data generated by the data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end; according to the capacity information and the data usage information, the data that meets the preset transfer condition is transferred from the first buffer to the historical data storage area. It solves the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the data usage node with the weakest data processing ability, resulting in a low overall data processing ability of the system. At the same time, data can be called from the historical data storage area, making it more convenient to add new data usage nodes and improving the data security of each data usage node. Description of the Drawings

[0062] The drawings here are incorporated into the specification and constitute a part of this specification, showing the embodiments that conform to the present application, and are used together with the specification to explain the principles of the present application.

[0063] Figure 1 It is a schematic diagram of the application scenario of the data processing method provided by the embodiment of the present application;

[0064] Figure 2 It is a schematic flowchart of a data processing method provided by the present application;

[0065] Figure 3 It is a schematic flowchart of another data processing method provided by the embodiment of the present application;

[0066] Figure 4 It is a schematic flowchart of yet another data processing method provided by the embodiment of the present application;

[0067] Figure 5 It is a schematic flowchart of still another data processing method provided by the embodiment of the present application;

[0068] Figure 6 It is a schematic flowchart of yet another data processing method provided by the embodiment of the present application;

[0069] Figure 7Structural schematic diagram of a data processing device provided by an embodiment of the present application;

[0070] Figure 8 Structural schematic diagram of an electronic device provided by the present application.

[0071] Through the above-mentioned drawings, specific embodiments of the present application have been shown, and more detailed descriptions will be given in the following text. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0072] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts, including but not limited to combinations of multiple embodiments, fall within the scope of protection of the present application.

[0073] The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims, and above-mentioned drawings of the present application are used to distinguish similar objects and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order different from those illustrated or described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0074] With the continuous development of computer processing technology, the processing of data is no longer completed by only one processing terminal, but a multi-terminal distributed processing structure has been formed. Even in an electronic device, the processing of its data is often completed by multiple chips. However, no matter how complex the structure is, it can ultimately be divided into two categories, or rather two ends, namely the data generation end and the data usage end.

[0075] In order to fully utilize the best performance of the electronic devices at both ends, the data generation end and the data usage end operate independently. This results in the lack of synchronous operation between the data generation end and the data usage end. Moreover, either end may include multiple different nodes. For example, the data usage end can include multiple data usage nodes. There are also differences in data transmission and processing capabilities among different data usage nodes, which leads to varying data processing capabilities, some stronger and some weaker, among different data usage nodes.

[0076] To address the issue of asynchrony in the time dimension between the data generation end and the data usage end, a buffer or cache area is introduced to temporarily store the data generated by the data generation end, as the data generation end generally has relatively fast data processing capabilities. For example, in a computer, the data processing capability of the central processing unit (CPU) far exceeds that of other devices, so memory is introduced for buffering.

[0077] However, since the capacity of the buffer is limited, if the data usage end fails to use the data in the buffer in a timely manner, the buffer will become full, and at this time, the data generation end has to stop generating data.

[0078] That is to say, although the buffer can adjust the asynchrony problem in the time dimension between the data generation end and the data usage end, due to its limited capacity, the buffer cannot be made infinitely large. However, the difference in processing speeds between the data generation end and the data usage end is indeed showing an increasing trend. Therefore, a common approach is to increase the capacity of the buffer. For example, the buffer memory of a mobile phone is upgraded from 1G to 2G, 4G, 8G, etc. However, further increases may be restricted by the physical space or energy consumption of the device itself. Therefore, this is obviously not a long-term solution.

[0079] This has given rise to the existing technology to improve the processing capacity of the data usage end in turn to narrow the difference in data asynchrony between the two. However, for some application scenarios, there are inevitable asynchrony differences during their operation. For example, servers or databases distributed in various locations all need to use the data of the central server or central database, that is, there is only one node at the data generation end, but there are multiple data usage nodes at the data usage end. To ensure the consistency of data among all data usage nodes, the data generated by the data generation end must wait until all usage nodes have used the data before the data can be deleted from the buffer. This results in the data processing capacity or data processing efficiency of the entire system being restricted by the data usage node with the weakest processing capacity, which is simply referred to as the bottleneck problem.

[0080] Regarding the bottleneck problem, the common practice is to equip each data-using node with a data buffer, that is, a one-to-one exclusive relationship is formed between the data-using node and the data buffer. However, this configuration method has a high cost, and subsequently, if new data-using nodes are added, data buffers need to be added simultaneously. This configuration method is very complex, involves significant changes to the entire system, and has a high cost. Moreover, how the new nodes obtain the same historical data becomes a new problem to be faced.

[0081] Therefore, for the existing technology, there are two caching forms. One is the single buffer form, and the other is the multi-buffer form corresponding to the number of using nodes, such as a circular buffer generated according to the number of using nodes. Each unit in the circular buffer is used to store the address of the data queue corresponding to each data-using node. The data generation end sends the data to the data queue corresponding to each data-using node, and then the data-using node processes the data.

[0082] The above two caching forms have the following defects:

[0083] 1. The single buffer has the bottleneck problem of data synchronization between the data generation end and the data-using end, that is, the data-using node that uses or processes data the fastest will be restricted by the data-using node that uses or processes data the slowest. Eventually, the data processing efficiency of the entire system is consistent with the data usage or processing speed of the slowest data-using node. And the data generation speed of the data generation end is also determined by the slowest data-using node. Because when the slowest data-using node has not used or processed the data in the finite buffer, the data in the buffer will not be cleared, resulting in the data generation end being unable to continue generating or storing data in the finite buffer, and the data-using node that uses or processes data the fastest being unable to obtain new data. In short, the data processing capacity or data processing efficiency of the entire system is severely restricted.

[0084] 2. When the data of the data-using node is incorrect or accidentally deleted, the data in the buffer has been deleted, and the data-using node cannot directly obtain the data from the buffer and needs to reapply to the data generation end for generation, wasting the computing resources of the data generation end.

[0085] 3. When new data-using nodes are added, historical data synchronization cannot be performed, and reapplication to the data generation end for generation is required, wasting the computing resources of the data generation end.

[0086] Generally speaking, the existing caching forms have the technical problem of low data processing efficiency or data processing capacity of the entire system.

[0087] The data processing method provided by this application aims to solve the above technical problems of the existing technology.

[0088] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0089] Figure 1 It is a schematic diagram of the application scenario of the data processing method provided by the embodiment of the present application. As Figure 1 shown, when the MySQL database is performing unitized data synchronization, one source database corresponds to multiple target databases, that is, one data generation end 101 corresponds to multiple data usage nodes of the data usage end 102. Due to the network bandwidth and hardware and other conditions between different data usage nodes, when the databases between different data usage nodes are performing data synchronization, the ability of each database to process or use the source data is different, and the database with the lowest processing or using ability seriously affects the ability of the source database, that is, the data generation end 101, to generate data. For this reason, in the data cache end 103 of the embodiment of the present application, a first cache area and a historical data storage area are set. The first cache area can regularly store the data that meets the predetermined conditions in the historical data storage area. When the data requested by the data usage node with a lower processing or using ability is determined to be historical data, the target historical data required by the data usage node can be sent to the data usage node from the historical data storage area.

[0090] It should be noted that in this embodiment, the first cache area can be a storage unit for fast data reading, such as a flash memory, and the historical data storage area can be a storage area on a mechanical hard disk with a slightly slower read and write speed, or a storage area on a solid-state drive with a faster read and write speed, etc.

[0091] By means of timed cleaning, the data in the first buffer area is transferred to the historical data storage area, so that the valuable capacity of the first buffer area can be restored in time, so that the data generation end can maintain an efficient working state and continuously generate data without being forced to stop working because the capacity of the first buffer area is full. In addition, for data use nodes with strong processing or use capabilities, new data can also be quickly obtained without being restricted by data use nodes with weak processing or use capabilities, solving the shortest board problem. For data use nodes with weak processing or use capabilities, data that cannot be obtained from the first buffer area in time can still be obtained through the historical data storage area without occupying the data generation end to regenerate. Further, if a new data use node is added, or the data of the use node is wrong or deleted by mistake, then the historical data can also be directly obtained from the historical data storage area without occupying the data generation end to regenerate the corresponding data. The data processing method of the embodiment of the present application, while solving the shortest board problem, also solves the problem of high cost or complex operation when adding a new use node, and also improves the data security of each data use node, achieving a significant improvement effect relative to the prior art, and greatly improving the data processing capability or data processing efficiency of the entire data processing system.

[0092] The detailed steps of the data processing method provided by this application are introduced below in conjunction with the accompanying drawings.

[0093] Figure 2 A flowchart of a data processing method provided in this application. Figure 2 As shown, the specific steps of the data processing method include:

[0094] S201. Obtain capacity information and data usage information of a first cache area.

[0095] In this step, the first buffer area is used to store data generated by the data generation end, the capacity information is used to indicate the dynamic usage of the capacity, and the data usage information is used to indicate the usage of the data by each usage node in the data usage end.

[0096] Specifically, the data generation end or the data caching end monitors the dynamic usage of the capacity of the first buffer in real time, such as the used capacity, the remaining capacity, the dynamic change rate of the used capacity, the dynamic change rate of the remaining capacity, and so on. The capacity information can be used as a basis for the data generation end to determine whether to stop generating or storing new data. At the same time, it can also be used as a trigger signal for the timing cleaning mode of the first buffer. When the remaining capacity reaches the preset remaining threshold, or when the used capacity reaches the preset limit value, the timing cleaning mode is triggered. The timing cleaning mode is to transfer the data that meets the preset conditions in the first buffer, such as the data that has been used by at least one data usage node of the data usage end, to the historical data storage area.

[0097] The data usage information can be used to determine which data can be transferred and reflects the usage rate of the data by the data usage nodes. For example, the number of times the same data is called by different data usage nodes can be recorded as the data usage information. When the number of usage times exceeds half of the total number of data usage nodes, the data can be transferred to the historical data storage area.

[0098] S202. According to the capacity information and the data usage information, transfer the data that meets the preset transfer conditions from the first buffer to the historical data storage area.

[0099] In this step, the preset transfer conditions at least include: the first transfer condition, the second transfer condition, and the third transfer condition.

[0100] Specifically, the first transfer condition is used to meet the data production efficiency requirements of the data generation end. When the used capacity of the first buffer reaches the preset limit value, or when the remaining capacity reaches the preset remaining value, some or all of the data in the first buffer can be transferred to the historical data storage area.

[0101] In a possible design, it is also possible to combine the data generation rate of the data generation end to determine whether some or all of the data in the first buffer needs to be transferred to the historical data storage area.

[0102] Optionally, the data production efficiency of the data generation end can also be judged by the change rate of the used capacity or the remaining capacity. When the change rate is greater than the preset threshold, it can be determined that the data generation end is generating data in large quantities and quickly. Then, the timing cleaning mode of the first buffer can be enabled, and some or all of the data in the first buffer can be transferred to the historical data storage area according to the preset time period.

[0103] The second transfer condition is used to meet the data usage requirements of high-speed data usage nodes, and the data usage rate of the high-speed data usage nodes is greater than or equal to a preset rate. At this time, the main factor is the data usage information. For each data usage node, only the unused data in the first buffer is valid data. The proportion of valid data of high-speed data usage nodes is relatively small, or the proportion of used data is relatively high. When it reaches the preset ratio, the data processing efficiency of high-speed data usage nodes will be restricted. Therefore, by detecting the proportion of used data of each data usage node, the data that has been classified as used data in a preset number of data usage nodes can be transferred to the historical data storage area.

[0104] For example, if there are 3 data usage nodes at the data usage end and a certain data has been used by two of the data usage nodes, then this data can be transferred to the historical data storage area.

[0105] The third transfer condition is used to clean the first buffer in time after the data has been used by all usage nodes. That is, through the data usage information, when it is detected that any data in the first buffer has been used by all data usage nodes, this data can be transferred to the historical data storage area. Further, all data can also be transferred to the historical data storage area.

[0106] It should be noted that the data transferred to the historical data storage area can still be called by any one of the data usage nodes in the data usage end. Even for newly added data usage nodes, in order to enable the data usage nodes to use historical data, the historical data in the historical data storage area can be directly sent to the newly added data usage nodes.

[0107] Therefore, when using the data processing method provided in the embodiments of the present application, when adding a new data usage node, there is no need to make any changes to the data cache end and the data generation end. Compared with the prior art, it is simple, convenient and has low cost.

[0108] At the same time, since any data usage node can call the historical data in the historical data storage area, when the data of the data usage node is incorrect or accidentally deleted, it only needs to be read and called again from the historical data storage area, without the need for the data generation end to generate it again. While improving the data security of the data usage node, it also saves the computing resource consumption of the data generation end.

[0109] This embodiment provides a data processing method. By obtaining the capacity information and data usage information of the first buffer, where the first buffer is used to store data generated by a data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of data by each usage node in the data usage end; according to the capacity information and the data usage information, data that meets the preset transfer condition is transferred from the first buffer to the historical data storage area. This solves the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the data usage node with the weakest data processing ability, resulting in a low overall data processing ability of the system. At the same time, data can be called from the historical data storage area, making it more convenient to add new data usage nodes and improving the data security of each data usage node.

[0110] For ease of understanding, one of the embodiments of the first transfer condition in the preset transfer condition is given below for illustration.

[0111] Figure 3 It is a schematic flowchart of another data processing method provided by an embodiment of this application. As Figure 3 shown, the specific steps of this data processing method include:

[0112] S301. Obtain the capacity information and data usage information of the first buffer.

[0113] In this step, the first buffer is used to store data generated by a data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end.

[0114] S302. Determine the remaining capacity information according to the capacity information.

[0115] In this step, the remaining capacity information includes: the remaining capacity, and / or, the change rate of the remaining capacity.

[0116] Specific remaining capacity information can be represented in at least three ways:

[0117] The first is to represent it with the remaining capacity;

[0118] The second is to represent it with the change rate of the remaining capacity;

[0119] The third is to represent it jointly with the remaining capacity and the change rate of the remaining capacity.

[0120] S303. Determine whether the remaining capacity information meets the first transfer condition.

[0121] In this step, if it is satisfied, continue to execute S304; if it is not satisfied, return to repeat the execution of S301.

[0122] In this embodiment, if the remaining capacity information is represented by the remaining capacity, the first transfer condition is that the remaining capacity is less than or equal to a preset capacity threshold;

[0123] If the remaining capacity information is represented by the change rate of the remaining capacity, the first transfer condition is that the change rate is greater than or equal to a preset change threshold;

[0124] If the remaining capacity information is jointly represented by the remaining capacity and the change rate of the remaining capacity, the first transfer condition is that the remaining capacity is less than or equal to a preset capacity threshold and the change rate is greater than or equal to a preset change threshold.

[0125] S304. According to the data usage information, transfer the data to be transferred from the first buffer to the historical data storage area at a preset period.

[0126] In this step, the data usage information is used to judge or identify the data to be transferred.

[0127] In a possible implementation manner, the data to be transferred includes the used data, and the used data is the data used by at least one of the nodes.

[0128] In another possible implementation manner, the data to be transferred includes all the data in the first buffer.

[0129] In still another possible implementation manner, the data to be transferred includes the data whose usage times reach a preset usage times, for example, the data whose usage times are greater than or equal to half of the total number of data usage nodes.

[0130] Specifically, the preset period can be 30s, and the transfer is performed every 30 seconds to ensure that the first buffer has sufficient capacity, so that the data generation end can maintain high-efficient data production efficiency.

[0131] This embodiment provides a data processing method. By obtaining the capacity information and data usage information of the first buffer, the first buffer is used to store the data generated by the data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end; according to the capacity information and the data usage information, transfer the data that meets the preset transfer conditions from the first buffer to the historical data storage area. This solves the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the weakest data usage node in terms of data processing ability, resulting in a low overall data processing ability of the system. At the same time, the data can be called from the historical data storage area, making it more convenient to add new data usage nodes and improving the data security of each data usage node.

[0132] For ease of understanding, one embodiment of the second transfer condition in the preset transfer conditions is given below for illustration.

[0133] Figure 4 This is a schematic flowchart of another data processing method provided by an embodiment of the present application. As Figure 4 shown, the specific steps of this data processing method include:

[0134] S401. Obtain the capacity information and data usage information of the first buffer.

[0135] In this step, the first buffer is used to store the data generated by the data generation end. The capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end.

[0136] S402. Determine the proportion of the used data according to the data usage information.

[0137] In this step, the used data is the data used by at least one of the nodes. The proportion of the used data is the ratio of the total amount of used data to the total amount of data in all the first buffers.

[0138] S403. Judge whether the proportion is greater than or equal to a preset proportion.

[0139] In this step, if so, execute S404; if not, return to execute S401 again.

[0140] Specifically, the preset proportion can take any proportion between 0 and 1, and those skilled in the art can select it according to actual needs. The selection principle can be determined according to the processing rate of the data usage node with the fastest data processing rate among all the data usage nodes, or according to the average data processing rate of all the data usage nodes.

[0141] S404. Judge whether the usage times of the used data are greater than or equal to a preset usage times.

[0142] In this step, if so, execute S405; if not, return to execute S401 again.

[0143] Specifically, the preset usage times can be half of the total number of data usage nodes. It can be understood that those skilled in the art can set it according to the actual application scenario. The preset usage times can also be set as a dynamic value, which changes correspondingly with the average data processing rate of all the data usage nodes or the processing rate of the data usage node with the fastest data processing rate.

[0144] S405. Transfer the data to be transferred from the first buffer to the historical data storage area according to the data usage information.

[0145] In this step, the data usage information is used to determine or identify the data to be transferred.

[0146] In a possible implementation, the data to be transferred includes used data, where the used data is data that has been used by at least one of the nodes.

[0147] In another possible implementation, the data to be transferred includes the data in all the first buffer areas.

[0148] In yet another possible implementation, the data to be transferred includes data whose usage times reach a preset number of usage times, such as data whose usage times are greater than or equal to half of the total number of data usage nodes.

[0149] This embodiment provides a data processing method. By obtaining the capacity information of the first buffer area and the data usage information, the first buffer area is used to store the data generated by the data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end; according to the capacity information and the data usage information, the data that meets the preset transfer conditions is transferred from the first buffer area to the historical data storage area. This solves the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the data usage node with the weakest data processing ability, resulting in a low overall data processing ability of the system. At the same time, data can be called from the historical data storage area, making it more convenient to add new data usage nodes and improving the data security of each data usage node.

[0150] For ease of understanding, one of the embodiments of the third transfer condition in the preset transfer conditions is given below for illustration.

[0151] Figure 5 It is a schematic flowchart of yet another data processing method provided by the embodiments of the present application. As Figure 5 shown, the specific steps of this data processing method include:

[0152] S501. Obtain the capacity information of the first buffer area and the data usage information.

[0153] In this step, the first buffer area is used to store the data generated by the data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end.

[0154] S502. Determine whether all the data in the first buffer area has been used up by all the data usage nodes.

[0155] In this step, if yes, then execute S503; if no, then return to S501 to re-loop the determination.

[0156] S503. Transfer all data to the historical data storage area.

[0157] It should be noted that in this embodiment, when the processing capabilities of each data usage node are not very different and the data generation end does not generate a large amount of data, after all the data in the first buffer area has been used, instead of directly deleting it as in the prior art, it is transferred to the historical data storage area, which is convenient for each data usage node to call historical data and for newly added data usage nodes to call historical data, without occupying the computing resources of the data generation end to regenerate.

[0158] This embodiment provides a data processing method. By obtaining the capacity information and data usage information of the first buffer area, the first buffer area is used to store the data generated by the data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of data by each usage node in the data usage end; according to the capacity information and the data usage information, the data that meets the preset transfer conditions is transferred from the first buffer area to the historical data storage area. It solves the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the data usage node with the weakest data processing ability, resulting in a low overall data processing ability of the system. At the same time, data can be called from the historical data storage area, making it more convenient for newly added data usage nodes and improving the data security of each data usage node.

[0159] It should be noted that Figures 3 to 5 The three embodiments shown can be used as three parallel implementation schemes, or parallel threads, to implement the data processing method of the present application.

[0160] Based on the above-mentioned various embodiments, in order to further illustrate how the data usage node calls the data in the historical data storage area, the following uses Figure 6 the embodiment shown for explanation.

[0161] Figure 6 It is a schematic flow chart of another data processing method provided by an embodiment of the present application. As Figure 6 shown, the specific steps of this data processing method include:

[0162] S601. Obtain the historical data usage request of the data usage node in the data usage end.

[0163] S602. Determine the target historical data in the historical data storage area according to the historical data usage request.

[0164] S603. Send the target historical data to the data usage node.

[0165] In this step, there are various implementation manners for sending the target historical data to the data usage node.

[0166] In a possible implementation manner, if the remaining capacity of the first buffer meets the storage requirement of the target historical data, then store the target historical data in the first buffer for the data usage node to use.

[0167] In another possible implementation manner, if the remaining capacity of the first buffer does not meet the storage requirement of the target historical data, then store the target historical data in the second buffer for the data usage node to use.

[0168] In still another possible implementation manner, store the target historical data in the second buffer for the data usage node to use.

[0169] It should be noted that, in this embodiment, the second buffer is a storage device of the same type as the first buffer that can perform fast read and write operations, such as a memory or a flash memory, and can also be a solid-state drive, etc.

[0170] It should also be noted that, in this embodiment, any data usage node in the data usage end can execute the request for using historical data. Because during the data usage process, data may be incorrect or accidentally deleted due to various reasons. At this time, as long as the data processing method of this embodiment is executed to retrieve the historical data from the historical data storage area, the data can be restored, which improves the data security of the data usage node. At the same time, it avoids the data generation end from recalculating to generate historical data, saves the computing resources of the data generation end, and improves the data processing efficiency and ability of the system.

[0171] In addition, for newly added data usage nodes, the historical data can also be obtained according to Figure 6 the shown data processing method, so that each data usage node can obtain the same data and there is no need for the data generation end to regenerate it.

[0172] This embodiment provides a data processing method. By obtaining the capacity information and data usage information of the first buffer, where the first buffer is used to store data generated by a data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of data by each usage node in the data usage end; according to the capacity information and the data usage information, transfer the data that meets the preset transfer conditions from the first buffer to the historical data storage area; then obtain the historical data usage requests of the data usage nodes in the data usage end, and determine the target historical data in the historical data storage area according to the historical data usage requests, and finally send the target historical data to the data usage nodes. This solves the technical problem in the prior art that the data transmission between the data generation end and the data usage end is restricted by the data usage node with the weakest data processing ability, resulting in a low overall data processing ability of the system. At the same time, data can be called from the historical data storage area, making it more convenient to add new data usage nodes and improving the data security of each data usage node.

[0173] Figure 7 This is a schematic structural diagram of a data processing device provided by an embodiment of the present application. The data processing device 700 can be implemented through software, hardware, or a combination of both.

[0174] As Figure 7 shown, the data processing device 700 includes:

[0175] An acquisition module 701, configured to acquire the capacity information and data usage information of the first buffer, where the first buffer is used to store data generated by a data generation end, the capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of data by each usage node in the data usage end;

[0176] A processing module 702, configured to transfer the data that meets the preset transfer conditions from the first buffer to the historical data storage area according to the capacity information and the data usage information.

[0177] In a possible design, the preset transfer conditions include a first transfer condition, and the first transfer condition is used to meet the data production efficiency requirements of the data generation end. Correspondingly, the processing module 702 is specifically configured to:

[0178] Determine the remaining capacity information according to the capacity information, where the remaining capacity information includes: remaining capacity, and / or, the change rate of the remaining capacity;

[0179] If the remaining capacity is less than or equal to a preset capacity threshold, and / or the change rate is greater than or equal to a preset change threshold, then transfer the data to be transferred from the first buffer to the historical data storage area according to the data usage information at a preset period.

[0180] In a possible design, the preset transfer condition includes a second transfer condition, and the second transfer condition is used to meet the data usage requirements of a high-speed data usage node, where the data usage rate of the high-speed data usage node is greater than or equal to a preset rate. Correspondingly, the processing module 702 is specifically configured to:

[0181] Determine the proportion of the used data according to the data usage information, where the used data is the data used by at least one of the nodes;

[0182] When the proportion is greater than or equal to a preset proportion, if the usage times of the used data are greater than or equal to a preset usage times, transfer the data to be transferred from the first buffer to the historical data storage area.

[0183] Optionally, the data to be transferred includes used data, and the used data is the data used by at least one of the nodes.

[0184] Optionally, the data to be transferred includes all the data in the first buffer.

[0185] In a possible design, the preset transfer condition includes a third transfer condition, and the third transfer condition is used to clean the first buffer in time after the data is used by all the usage nodes. Correspondingly, the processing module 702 is specifically configured to:

[0186] If all the data in the first buffer has been used up by all the data usage nodes, transfer all the data to the historical data storage area.

[0187] In a possible design, the obtaining module 701 is further configured to obtain the historical data usage requests of the data usage nodes in the data usage end;

[0188] The processing module 702 is further configured to determine target historical data in the historical data storage area according to the historical data usage request;

[0189] The processing module 702 is further configured to send the target historical data to the data usage node.

[0190] In a possible design, the processing module 702 is further configured to:

[0191] If the remaining capacity of the first buffer meets the storage requirements of the target historical data, store the target historical data in the first buffer for the data usage node to use.

[0192] Optionally, the processing module 702 is further configured to send the target historical data to the data usage node, specifically including:

[0193] If the remaining capacity of the first buffer does not meet the storage requirement of the target historical data, the target historical data is stored in the second buffer for use by the data usage node.

[0194] In a possible design, the processing module 702 is further configured to send the target historical data to the data usage node, specifically including:

[0195] The target historical data is stored in the second buffer for use by the data usage node.

[0196] It should be noted that Figure 7 The data processing device provided in the illustrated embodiment can execute the methods provided in any of the above method embodiments. The specific implementation principles, technical features, explanations of professional terms, and technical effects are similar and will not be elaborated here.

[0197] Figure 8 FIG. is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 8 shown, the electronic device 800 may include: at least one processor 801 and a memory 802. Figure 8 The illustrated shows an electronic device with one processor as an example.

[0198] The memory 802 is used to store a program. Specifically, the program may include program code, and the program code includes computer operation instructions.

[0199] The memory 802 may include a high-speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

[0200] The processor 801 is configured to execute the computer execution instructions stored in the memory 802 to implement the methods described in the above method embodiments.

[0201] Among them, the processor 801 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0202] Optionally, the memory 802 can be either independent or integrated with the processor 801. When the memory 802 is a device independent of the processor 801, the electronic device 800 may further include:

[0203] A bus 803 for connecting the processor 801 and the memory 802. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0204] Optionally, in specific implementation, if the memory 802 and the processor 801 are integrated on a single chip, the memory 802 and the processor 801 can communicate through an internal interface.

[0205] The embodiment of the present application also provides a computer-readable storage medium, which may include various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc. Specifically, the computer-readable storage medium stores program instructions, and the program instructions are used for the methods in the above method embodiments.

[0206] The embodiment of the present application also provides a computer program product, including a computer program, which implements the methods in the above method embodiments when executed by a processor.

[0207] The embodiment of the present application also provides a data processing system, including: a data generation end, a data caching end, and a data usage end; wherein,

[0208] The data generation end is used to generate data to be used and store the data to be used in the data caching end;

[0209] The data caching end includes a first buffer area and a historical data storage area. The historical data storage area is used to store the data in the first buffer area that meets the preset transfer condition. Optionally, the data caching end further includes a second buffer area, and the second buffer area is used to store the target historical data from the historical data storage area to the second buffer area for the data usage node to use when the data usage node requests to use the historical data;

[0210] The data usage end includes at least one data usage node, and the data usage node is used to call the data in the first buffer area or the historical data storage area;

[0211] The data generation end or the data caching end is further used to implement any possible data processing method provided by the above method embodiments.

[0212] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the claims of the present application.

[0213] It should be understood that the present application is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

[0214] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that, Including: Obtain the capacity information and data usage information of the first buffer. The first buffer is used to store data generated by a data generation end. The capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in a data usage end. According to the capacity information and the data usage information, transfer the data that meets the preset transfer condition from the first buffer to a historical data storage area. Obtain a historical data usage request of any one data usage node in the data usage end. According to the historical data usage request, determine target historical data in the historical data storage area. Send the target historical data to the data usage node.

2. The data processing method according to claim 1, wherein The step of transferring the data that meets the preset transfer condition from the first buffer to the historical data storage area according to the capacity information and the data usage information includes: Determine remaining capacity information according to the capacity information. The remaining capacity information includes: remaining capacity, and / or, the change rate of the remaining capacity. If the remaining capacity is less than or equal to a preset capacity threshold, and / or the change rate is greater than or equal to a preset change threshold, then according to the data usage information, transfer the data to be transferred from the first buffer to the historical data storage area at a preset period.

3. The data processing method according to claim 1, wherein The step of transferring the data that meets the preset transfer condition from the first buffer to the historical data storage area according to the capacity information and the data usage information includes: Determine the proportion of the used data according to the data usage information. The used data is the data that has been used by at least one of the nodes. When the proportion is greater than or equal to a preset proportion, if the usage times of the used data are greater than or equal to a preset usage times, then according to the data usage information, transfer the data to be transferred from the first buffer to the historical data storage area.

4. The data processing method according to claim 2 or 3, characterized in that The data to be transferred includes the used data, and the used data is the data that has been used by at least one of the nodes.

5. The data processing method according to claim 2 or 3, characterized in that The data to be transferred includes all the data in the first buffer.

6. The data processing method according to claim 2 or 3, characterized in that The step of transferring the data that meets the preset transfer condition from the first buffer to the historical data storage area according to the capacity information and the data usage information further includes: If all the data in the first buffer has been used up by all the data usage nodes, then transfer all the data to the historical data storage area.

7. The data processing method according to claim 1, wherein The step of sending the target historical data to the data usage node includes: If the remaining capacity of the first buffer meets the storage requirement of the target historical data, then store the target historical data in the first buffer for the data usage node to use.

8. The data processing method according to claim 1 or 7, characterized in that, The step of sending the target historical data to the data usage node includes: If the remaining capacity of the first buffer does not meet the storage requirement of the target historical data, then store the target historical data in a second buffer for the data usage node to use.

9. The data processing method according to claim 1, wherein The step of sending the target historical data to the data usage node includes: Store the target historical data in a second buffer for the data usage node to use.

10. A data processing device, characterized in that, Including: An acquisition module, configured to acquire the capacity information and data usage information of a first buffer. The first buffer is used to store data generated by a data generation end. The capacity information is used to represent the dynamic usage of the capacity, and the data usage information is used to represent the usage of the data by each usage node in the data usage end. A processing module, configured to transfer data that meets a preset transfer condition from the first buffer to a historical data storage area according to the capacity information and the data usage information. The acquisition module is further configured to acquire a historical data usage request of any one data usage node in the data usage end. The processing module is further configured to determine target historical data in the historical data storage area according to the historical data usage request, and send the target historical data to the data usage node.

11. A data processing system, characterized in that, Comprising: A data generation end, a data buffer end, and a data usage end. Among them, The data generation end is configured to generate data to be used and store the data to be used in the data buffer end. The data buffer end includes a first buffer and a historical data storage area. The historical data storage area is used to store data in the first buffer that meets the preset transfer condition. The data usage end includes at least one data usage node. The data usage node is configured to call data in the first buffer or the historical data storage area. The data generation end or the data buffer end is further configured to implement the data processing method described in any one of claims 1-9.

12. An electronic device, characterized in that, Comprising: A processor; And, A memory, configured to store the computer program of the processor. Wherein, the processor is configured to execute the data processing method described in any one of claims 1 to 9 by executing the computer program.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data processing method described in any one of claims 1 to 9.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data processing method described in any one of claims 1 to 9.

Images