Data processing method and device, computer device and storage medium

Abstract

The application relates to a data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: in response to a data addition request of a disk data table, obtaining a request time of the data addition request, determining a time identifier corresponding to the data addition request according to the request time; in the case that the time identifier does not exist in the disk data table, locking the disk data table, and establishing the time identifier in the disk data table; establishing an association relationship between the time identifier corresponding to the data addition request and the time identifier in the disk data table, and adding data corresponding to the data addition request to a storage area corresponding to the newly established time identifier. The method can improve the data processing efficiency and ensure the integrity of effective data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a data processing method and device, computer equipment, a storage medium and a computer program product. BACKGROUND

[0002] In the field of network technology, resource refreshing and configuration distribution and other business scenarios often need to perform task distribution, especially when facing daily performance requirements of billions of levels. The existing technology usually copies disk data to memory to utilize the efficient read-write capability of memory to process upstream issued task requests and downstream obtained requests.

[0003] In this process, the system first modifies the data in the memory through the task request of the upstream, and then transmits the modified data to the corresponding IP address through the downstream request. However, when writing the data in the memory data table to the disk data table, the write lock stealing phenomenon often occurs. This phenomenon may cause data loss, thus failing to meet the actual business requirements. SUMMARY

[0004] Therefore, it is necessary to provide a data processing method and device, computer equipment, a computer readable storage medium and a computer program product in view of the above technical problems.

[0005] In a first aspect, the present application provides a data processing method. The method comprises:

[0006] In response to a data addition request for a disk data table, obtaining a request time of the data addition request, and determining a time identifier corresponding to the data addition request according to the request time;

[0007] In the case where the time identifier does not exist in the disk data table, locking the disk data table, and establishing the time identifier in the disk data table;

[0008] Establishing an association between the time identifier corresponding to the data addition request and the time identifiers in the disk data table, and adding the data corresponding to the data addition request to the storage area corresponding to the newly established time identifier.

[0009] In one of the embodiments, the method further comprises:

[0010] In response to an interaction request for target data, obtaining the time identifiers of each data in the disk data table;

[0011] Using the time identifiers in the disk data table, selecting a preset number of data from the disk data table and storing them in the memory data table; wherein the preset number is the maximum data storage capacity of the memory data table.

[0012] Performing corresponding interactive operation on the memory data table by using the interactive request.

[0013] In one of the embodiments, the disk data includes one or more data files; and selecting a preset number of data from the disk into the memory data table by using the time identifier in the disk data table comprises:

[0014] Determining the number of target data files by using the maximum data storage capacity of the memory data table and the data storage capacity of the data files;

[0015] Determining the target data files by using the number of target data files and the time identifiers of the data files;

[0016] Selecting target data from the target data files into the memory data table in sequence by using the time identifier of the target data files and the time identifiers of the data in the data files.

[0017] In one of the embodiments, the method further comprises:

[0018] In response to the acquisition request for the target data, determining whether the time identifier is included in the acquisition request;

[0019] In the case that the time identifier is included in the acquisition request, acquiring the data corresponding to the time identifier from the memory data table and acquiring the data corresponding to the next time identifier in sequence;

[0020] In the case that the time identifier is not included in the acquisition request, acquiring the data corresponding to the first time identifier from the memory data table and acquiring the data corresponding to the next time identifier in sequence.

[0021] In one of the embodiments, the memory data table further comprises the association relationship between the account and the data identifier; and the step of determining whether the time identifier is included in the acquisition request in response to the acquisition request for the target data comprises:

[0022] In response to the acquisition request for the target data by the account, acquiring the data identifier from the interactive request and determining whether the association relationship exists between the account and the data identifier by using the memory data table;

[0023] In the case that the association relationship exists between the account and the data identifier, determining whether the time identifier is included in the acquisition request.

[0024] In one of the embodiments, the method further comprises:

[0025] In response to the interactive request for the target data, adding the target data in the disk data table into the memory data table;

[0026] In a case where a preset time elapses, the memory data table is used to perform an interaction operation corresponding to the interaction request.

[0027] In a second aspect, the present application further provides a data processing apparatus. The apparatus comprises:

[0028] A data acquisition module is configured to, in response to a data addition request for a disk data table, acquire a request time of the data addition request, and determine a time identifier corresponding to the data addition request according to the request time.

[0029] An identifier establishing module is configured to, in a case where the time identifier does not exist in the disk data table, lock the disk data table, and establish the time identifier in the disk data table.

[0030] A data interaction module is configured to establish an association between the time identifier corresponding to the data addition request and the time identifiers in the disk data table, and add data corresponding to the data addition request to a storage area corresponding to the newly established time identifier.

[0031] In one of the embodiments, the apparatus further comprises:

[0032] An identifier acquisition module is configured to, in response to an interaction request for target data, acquire time identifiers of data in a disk data table.

[0033] A data storage module is configured to select a preset number of data from the disk data table by using the time identifiers in the disk data table and store the data in a memory data table; wherein the preset number is a maximum data storage capacity of the memory data table.

[0034] The data interaction module is further configured to perform a corresponding interaction operation on the memory data table by using the interaction request.

[0035] In one of the embodiments, the disk data comprises a single data file or multiple data files; and the data storage module comprises:

[0036] A number determination sub-module is configured to determine a number of target data files by using the maximum data storage capacity of the memory data table and a data storage capacity of the data files.

[0037] A file determination sub-module is configured to determine the target data files by using the number of the target data files and the time identifiers of the data files.

[0038] A data storage sub-module is configured to sequentially select target data from the target data files and store the target data in the memory data table by using the time identifiers of the target data files and the time identifiers of the data in the data files.

[0039] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the data processing method as described in any one of the embodiments of this disclosure.

[0040] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the data processing method as described in any one of the embodiments of this disclosure.

[0041] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the data processing method as described in any of the embodiments of this disclosure.

[0042] The aforementioned data processing methods, apparatus, computer equipment, storage media, and computer program products determine the timestamp corresponding to a data addition request based on the request time. If the timestamp does not exist in the disk data table, it is created in the disk data table, and its association with other timestamps in the disk data table is established. The data corresponding to the data addition request is then added to the storage area corresponding to the timestamp. By obtaining the request time and generating a timestamp when responding to a data addition request, data updates in the disk data table can be effectively managed. In the absence of a timestamp, locking the disk data table and creating a new timestamp ensures data consistency and integrity. Simultaneously, this mechanism reduces lock contention during write operations, allowing the data transfer unit to quickly deliver new data to the management unit, ensuring timely storage of valid data and preventing data loss. This method ensures stable system operation in high-concurrency environments, improving data processing efficiency and guaranteeing the integrity of valid data. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating a data processing method in one embodiment;

[0044] Figure 2 This is a schematic diagram of the data interaction process in one embodiment;

[0045] Figure 3 This is a schematic diagram illustrating the process of data storage into memory in one embodiment;

[0046] Figure 4 This is a schematic diagram of the data acquisition process in one embodiment;

[0047] Figure 5 This is a flowchart illustrating the process of determining association relationships in one embodiment;

[0048] Figure 6 A flowchart of a data storage and interaction process in an embodiment;

[0049] Figure 7 A schematic diagram of a data table in an embodiment;

[0050] Figure 8 A flowchart of an initial unit in an embodiment;

[0051] Figure 9 A flowchart of data landing in an embodiment;

[0052] Figure 10 A flowchart of data rejection in an embodiment;

[0053] Figure 11 A flowchart of downstream task acquisition in an embodiment;

[0054] Figure 12 A flowchart of responding to an upstream request in an embodiment;

[0055] Figure 13 A flowchart of responding to a downstream request in an embodiment;

[0056] Figure 14 A flowchart of a transfer unit in an embodiment;

[0057] Figure 15 A block diagram of a data processing apparatus in an embodiment;

[0058] Figure 16 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0059] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be given to the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0060] In an embodiment, as shown in Figure 1 , a data processing method is provided, and the embodiment is exemplified by applying the method to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and can be realized through the interaction of the terminal and the server. In the embodiment, the method includes the following steps:

[0061] In step S100, in response to a data addition request for a disk data table, a request time of the data addition request is acquired, and a time identifier corresponding to the data addition request is determined according to the request time.

[0062] In an example embodiment, the disk data table can include time identifiers and data, and specifically, can include a plurality of data groups corresponding to each time identifier, each data group including a plurality of data. In actual use, the data group corresponding to each time identifier can be obtained by the same task, for example, the data groups include data group A and data group B. Data group A includes data of task a and data of task b. In different time identifiers, the data group A and the data group B can be included, that is, the data of task a and the data of task b are included. However, the corresponding data of the task is different in different time identifiers.

[0063] In an example embodiment, the data adding to the disk data table can include adding data in the memory data table to the disk data table, and the like. The memory data table can be used for interaction between the IP and the data. The request time can include a time when the data is requested to be added to the memory data table. It can be understood that the data in the memory data table can be finally stored in the disk data table, so that the data is added to the memory data table, that is, the data is added to the disk data table, and the like.

[0064] In an example embodiment, the time identifier determined by the request time can include setting a preset time period, and performing a quotient operation on the request time and the time period to obtain the corresponding time identifier. It can be understood that the quotient operation on the request time and the time period obtains the number of periods corresponding to the request. The number of periods can represent a time identifier, and the like. The data in one period is stored in the storage area corresponding to one time identifier, and the like.

[0065] Step S200, in the case where the time identifier does not exist in the disk data table, the disk data table is locked, and the time identifier is established in the disk data table.

[0066] In an example embodiment, the time identifier does not exist in the disk data table, that is, the data of the time identifier is not stored in the disk data table. At this time, in order to avoid read-write conflict and write lock stealing of the disk data table, the disk data is locked, and the like.

[0067] In an example embodiment, the time identifier does not exist in the disk data table, so the time identifier needs to be established in the disk data table, and the like.

[0068] Step S300, the association relationship between the time identifier corresponding to the data adding request and the time identifier in the disk data table is established, and the data corresponding to the data adding request is added to the storage area corresponding to the newly established time identifier.

[0069] In one exemplary embodiment, there are correlations among the time identifiers in the disk data table. Specifically, the correlation among the time identifiers in the disk data table may include sorting the time identifiers by request time, with each time identifier pointing to the next time identifier, i.e., the previous request time points to the next request time. It is understood that no time identifier points to the first time identifier (i.e., the earliest request time), and the last time identifier does not point to any time identifier (i.e., the latest request time), etc. In one exemplary embodiment, establishing the correlation between the time identifier corresponding to the data addition request and the time identifiers in the disk data table may include using the last time identifier in the disk data table (i.e., the time identifier corresponding to the latest request time) to point to the newly created time identifier, etc.

[0070] In one exemplary embodiment, after establishing the association relationship for the newly created time identifier, the data addition request can be added to the storage area of ​​the disk data table corresponding to the newly created time identifier, etc.

[0071] In the aforementioned data processing method, the request time of a data addition request is used to determine the corresponding timestamp. If this timestamp does not exist in the disk data table, it is created in the disk data table, and its association with other timestamps in the disk data table is established. The data corresponding to the data addition request is then added to the storage area corresponding to the timestamp. By obtaining the request time and generating a timestamp when responding to a data addition request, data updates in the disk data table can be effectively managed. In the absence of a timestamp, locking the disk data table and creating a new timestamp ensures data consistency and integrity. Simultaneously, this mechanism reduces lock contention during write operations, allowing the data transfer unit to quickly deliver new data to the management unit, ensuring timely storage of valid data and preventing data loss. This method ensures stable system operation in high-concurrency environments, improving data processing efficiency and guaranteeing the integrity of valid data.

[0072] In one embodiment, such as Figure 2 As shown, the method further includes:

[0073] Step S400: In response to the interaction request for the target data, obtain the timestamp of each data in the disk data table.

[0074] In one exemplary embodiment, the interaction request may include listening to the target data to see if there is a possibility of interaction with the target data. In actual use, the disk data table may be electrically connected to a program and interact with the disk data table through the program. In this case, the interaction request may include opening the program, that is, listening to whether the program is opened, and performing subsequent steps if the program is opened.

[0075] In step S500, a preset number of data is selected from the disk data table according to the time identifier and stored in the memory data table; wherein the preset number is the maximum data storage of the memory data table.

[0076] In an exemplary embodiment, storing data in the memory data table according to the time identifier can include selecting data with the latest time identifier (i.e. the data corresponding to the time identifier closest to the current time) from the disk data table and storing the selected data in the memory data table.

[0077] In step S600, the memory data table is subjected to corresponding interactive operations according to the interactive request.

[0078] In an exemplary embodiment, after the data is stored in the memory data table, the memory data table can be used to perform corresponding interactive operations.

[0079] In this embodiment, the data in the disk data table is stored in the memory data table, and the memory data table is used to perform corresponding interactive operations. The data interaction is completed by storing the data of the disk data table in the memory data table. Since the data access speed of the memory is much faster than that of the disk, the efficiency of data processing and interactive operation can be significantly improved. At the same time, using the memory data table can greatly reduce the interaction burden with the disk data table and improve the efficiency of obtaining data from the disk data table. At the same time, by presetting the number and the time identifier, the use of the memory data table can be effectively managed, which not only avoids the problem of memory overflow, but also ensures the timeliness and relevance of the memory in the memory data table, and improves the accuracy of data processing.

[0080] In one embodiment, as shown in Figure 3 the disk data includes a single or multiple data files; and the step of selecting a preset number of data from the disk according to the time identifier in the disk data table and storing the data in the memory data table includes:

[0081] In step S501, the number of target data files is determined according to the maximum data storage of the memory data table and the data storage of the data file.

[0082] In an example embodiment, the maximum data storage of the memory data table can include the memory size of the memory data table, etc.; the data storage of the data file, i.e. the number of data stored in each data file, etc.; it can be understood that the memory size of the memory data table is usually greater than the number of data of the data file, and therefore it is usually necessary to store the data of multiple data files into the memory, etc. Specifically, the maximum data storage of the memory and the data storage of the data file can be used to perform a division process, i.e. the number of target data files, etc.

[0083] In step S502, the number of target data files and the time identifier of each data file are used to determine the target data file.

[0084] In an example embodiment, the data file can include the time identifier of the data file, etc.; the time identifier of the data file can be determined by the time identifier of the data stored in the data file, etc. For example, the time identifier of the data file can be the earliest time identifier or the latest time identifier of the data stored in the data file, etc.

[0085] In an example embodiment, the data file with the earliest time identifier can be selected from the data file, etc.

[0086] In step S503, the time identifier of the target data file and the time identifier of each data in the data file are used to select the target data from the target data file in turn and store it into the memory data table.

[0087] In an example embodiment, the data file with the latest time identifier can be selected from the target data file, and the data of the other target data files can be stored into the memory data table, and the data with the latest time identifier can be selected from the target data file with the latest time identifier and added to the memory data table in turn until the memory of the memory data table is full, etc. It can be understood that the number of target data files represents that in the absence of a target data file, the memory data table cannot store enough data, and therefore one of the target data files can be selected, and the data of the other target data files can be stored into the memory data table, and the target data can be selected from the selected target data file and stored into the memory data table, etc.

[0088] In this embodiment, by utilizing the maximum storage capacity of the memory data table and the storage capacity of the data file, the required number of target data files can be effectively determined. This process not only optimizes the management of data storage, but also ensures the efficient use of memory. Then, by the time identifier of the target data file, the required data file can be accurately selected, thereby improving the efficiency of data retrieval. Finally, storing the target data into the memory data table helps to quickly access and process data, providing a good foundation for subsequent data analysis and decision-making. This method enhances the flexibility and response speed of the system, helping to improve overall performance.

[0089] In one embodiment, as shown in Figure 4 the method further comprises:

[0090] Step S401, in response to the acquisition request of the target data, judging whether the time identifier is included in the acquisition request.

[0091] In an exemplary embodiment, the target data acquisition request can include data identifier and time identifier, etc.; specifically, the data corresponding to the data identifier can include identifiers of different time points, i.e. target data corresponding to different time identifiers. Therefore, it is necessary to determine whether the time identifier corresponding to the target data identifier is included in the acquisition request.

[0092] Step S402, in the case that the time identifier is included in the acquisition request, acquiring the data corresponding to the time identifier from the memory data table, and sequentially acquiring the data corresponding to the next time identifier.

[0093] In an exemplary embodiment, the acquisition request can include a time identifier, and the data corresponding to the time identifier under the data identifier is acquired; and the data corresponding to the time identifier after the data identifier is sequentially acquired, etc. It can be understood that one time identifier in the memory data table points to the next time identifier, so the data corresponding to different time identifiers can be sequentially acquired, etc.

[0094] Step S403, in the case that the time identifier is not included in the acquisition request, acquiring the data corresponding to the first time identifier from the memory data table, and sequentially acquiring the data corresponding to the next time identifier.

[0095] In an exemplary embodiment, in the case that the acquisition request does not include a time identifier, the data corresponding to the first time identifier in the memory data table is acquired; wherein the first time identifier can include the earliest time identifier in the time identifier, i.e. the time identifier farthest from the current time, etc.; and according to the time identifier pointed to by the first time identifier, the data corresponding to the next time identifier is sequentially acquired, etc.

[0096] In this embodiment, by judging whether the time identifier is included in the acquisition request, different data access requirements are flexibly adapted. At the same time, the efficiency of data acquisition is improved by avoiding sequential acquisition from the first time point in the memory data table each time. When the time identifier is included in the request, the system can directly acquire the data at the specific time point, ensuring the accuracy and relevance of the data; and in the case where there is no time identifier, the system can sequentially acquire data from the first time point, ensuring the integrity of the data and the stability of the system. This design not only improves the efficiency of data acquisition, but also optimizes the data processing process.

[0097] In one embodiment, as shown in Figure 5 The memory data table further includes an association relationship between the account and the data identifier; and in response to the acquisition request for the target data, it is judged whether the time identifier is included in the acquisition request, comprising:

[0098] Step S411, in response to the acquisition request for the target data by the account, acquiring the data identifier from the interaction request, and determining whether the account and the data identifier have an association relationship by using the memory data table.

[0099] Step S412, in the case where the account and the data identifier have an association relationship, judging whether the time identifier is included in the acquisition request.

[0100] In an exemplary embodiment, the association relationship between the account and the data identifier can represent which target data the account follows, i.e., which data the account can acquire.

[0101] In an exemplary embodiment, if there is an association relationship between the account and the data identifier in the memory data table, it means that the account can acquire the data of the data identifier, and if there is no association relationship between the account and the data identifier in the memory data table, it means that the account does not follow the data of the data identifier, so it cannot acquire the data.

[0102] In this embodiment, by confirming the association relationship between the account and the data identifier, it is ensured that only authorized users can access specific data, improving the security and data protection of the system. After confirming the existence of the association relationship, it is further judged whether the time identifier is included in the acquisition request. This process not only enhances the flexibility of data access, but also provides accurate data acquisition methods according to specific requirements. This method makes data management more efficient and reliable, while reducing the risk of misoperation.

[0103] In one embodiment, as shown in Figure 6 The method further comprises:

[0104] Step S700, in response to the interaction request for the target data, adding the target data in the disk data table to the memory data table.

[0105] Step S800, if the preset time is reached, the interaction operation corresponding to the interaction request is performed using the memory data table.

[0106] In an exemplary embodiment, after the preset time is reached, the interaction using the memory data table can avoid interacting with the memory data table when the data in the memory data table is not completely written, thereby avoiding read-write conflicts and the like.

[0107] In this embodiment, by adding the target data in the disk data table to the memory data table, it is ensured that the data in the memory is the latest and relevant, thereby improving the efficiency of the interaction operation. After the preset time is reached, the interaction operation can be performed, which can effectively avoid accessing the data when it is not completely written into the memory, thereby reducing the risk of read-write conflicts. This not only improves the stability and data consistency of the system, but also optimizes the utilization of resources, so that a faster and more reliable response can be obtained when interacting. This mechanism enhances the overall performance of the system.

[0108] In an exemplary embodiment, the disk data table and the memory data table can be as shown in Figure 7 The data table can include a tasks format and an iptaskGroups format, wherein the tasks store time identifiers (T1, T2,..., Tn); the time identifiers correspond to a plurality of task data groups (taskGroupA, taskGroupB, etc.) and a next time identifier, i.e., _nextT, and each task data group stores a plurality of task data (task1, task2, etc.); the iptaskGroups format includes a plurality of accounts (ipA, ipB, etc.), each of which has an associated task group, ipA is associated with taskGroupA and taskGroupB, i.e., ipA follows the data of taskGroupA and taskGroupB, etc. In an exemplary embodiment, the data in the disk can be stored as a plurality of data files, i.e., each data file has a plurality of time identifiers. It can be understood that in this way, the number of data files can be reduced (avoiding storing one data file for one time identifier, etc.).

[0109] In an exemplary embodiment, the data processing method can be applied to four units, specifically including an initial unit, a management unit, a transceiving unit, and a transfer unit; wherein the initial unit can be implemented as shown in the schematic diagram Figure 8 Specifically, it includes:

[0110] Step S901, it is determined whether there is a data file in the disk; if yes, step S902 is performed; if no, step S906 is performed.

[0111] Step S902, reading from the latest data file of the disk data table, accumulating a preset number of files to load into the memory data table; it can be understood that the data file can be named by time identification, starting from the largest to read line by line, and reaching the preset number; the preset number is the amount of data stored in the memory data table; for example, there are data3, data2, data1, which are from the nearest to the farthest from the current time, and each stores ten data; and the memory data table can store fifteen data, at this time, ten data from data3 and five data from data2 are needed to be stored in the memory data table.

[0112] Step S903, updating the farthest time identification and the nearest time identification in the memory data; wherein the farthest time identification is the time identification farthest from the current time, and the nearest time identification is the time identification farthest from the current time.

[0113] Step S904, judging whether there is an ipTaskGroups file in the disk data table, if not, performing step S906, if yes, performing step S905;

[0114] Step S905, loading the memory in iptaskGroups into the memory data table;

[0115] Step S906, sleeping for a preset time.

[0116] The initial unit completes the transmission of data in the disk data table to the memory data table and the like through the above manner. The management unit can include data landing and proposing data, wherein the data logic can be implemented as shown in the flowchart, and specifically includes: Figure 9

[0117] Step S911, receiving a preset signal;

[0118] Step S912, judging whether the signal has an exit signal; if yes, performing the landing logic and continuing to step S912 to judge whether the received signal is an exit signal; if no, performing step S913;

[0119] Step S913, judging whether there is a landing signal in the signal, if yes, performing the landing logic and continuing to judge whether there is a landing signal in the signal; if no, performing step S914;

[0120] ​Step S914, obtaining data in the temporary variable from the memory data table and performing landing logic; wherein the data in the temporary variable is data stored in the memory data table after the account interacts with the memory data table. The landing logic can include storing data in the memory data table to the disk data table, and specifically can include obtaining a time identifier and task data, calculating the identifier of a data file through the time identifier of the data, and storing the data to the file corresponding to the identifier, etc. The data to be removed can be implemented as shown in the flowchart of Figure 10 , and specifically includes:

[0121] Step S921, determining whether the preset time is met; if yes, performing step S922; if not, returning to continue waiting until the preset time is met; wherein the preset time can include sequentially removing data after the preset time, etc.

[0122] Step S922, calculating the number of data in the memory data table.

[0123] Step S923, determining whether the number of data in the memory data table is greater than the memory of the memory data table; if yes, performing step S924; if not, performing step S927.

[0124] Step S924, determining the number of redundant data in the memory data table, and selecting data of the number of data farthest from the current time.

[0125] Step S925, releasing all the selected data.

[0126] Step S926, determining the time identifier farthest from the present time in the memory data table as the farthest time identifier, etc.

[0127] Step S927, calculating the number of data files.

[0128] Step S928, determining whether the number of data files is greater than the maximum capacity of the disk data table; if yes, performing step S929; if not, ending the running.

[0129] Step S929, deleting the data file corresponding to the time identifier farthest from the current time from the disk data table until the number of data in the disk data table meets the maximum capacity, etc.

[0130] The transceiver unit can include downstream acquisition request, responding to upstream request and responding to downstream request, and specifically the flowchart of the downstream acquisition request is as shown in Figure 11 , and specifically includes:

[0131] Step S931, whether the account corresponding to the acquisition request and the task data group followed by the account are included in the memory data table; if yes, step S932 is performed; if no, step S923 is performed;

[0132] Step S932, determining a preset variable as 0, and ending the acquisition request; wherein the preset variable of 0 indicates that the account does not exist in the memory data table;

[0133] Step S933, returning the task group information followed by the account, etc.

[0134] The response to the upstream request can be as shown in the flowchart, specifically including: Figure 12

[0135] Step S941, storing the upstream data into the memory data table;

[0136] Step S942, determining the time identifier of the data by using the output storage time;

[0137] Step S943, waiting for the acquisition write lock;

[0138] Step S944, judging whether the time identifier of the added data is stored in the memory data table; if yes, step S945 is performed; if no, step S948 is performed;

[0139] Step S945, judging whether the task data group identifier exists in the memory data table; if yes, step S946 is performed; if no, step S950 is performed;

[0140] Step S946, adding the data in the memory data table into the disk data table;

[0141] Step S947, releasing the write lock;

[0142] Step S948, sending the added data, the time identifier and the nearest time identifier to the current time to the carrying unit;

[0143] Step S949, replacing the nearest time identifier in the disk data table with the time identifier of the added data; and initializing the temporary memory in the memory data table (wherein the temporary memory stores the added data, etc.), and performing step S945;

[0144] Step S950, creating the corresponding task data group identifier in the disk data table.

[0145] The response to the downstream request can be as shown in the flowchart, specifically including: Figure 13

[0146] ​​Step S951, judging whether the time identifier exists in the acquisition request, if not, determining the time identifier of the acquisition request as the time identifier farthest from the current time in the memory data table;

[0147] Step S952, judging whether the time identifier in the acquisition request exists in the memory data table, if not, determining the time identifier of the acquisition request as the time identifier farthest from the current time in the memory data table;

[0148] Step S953, waiting for acquiring the read lock;

[0149] Step S954, judging whether the time identifier of the acquisition request points to the next time identifier, if yes, proceeding to step S955, if not, proceeding to step S957;

[0150] Step S955, obtaining all data corresponding to the time identifier of the acquisition request from the memory data table, and determining the time identifier of the acquisition request as the next time identifier; and determining the data state as "not read completely";

[0151] Step S956, releasing the read lock; and returning all task data, data state and time identifier to the requestor;

[0152] Step S957, judging whether the data state is "read completely", if yes, proceeding to step S958, if not, proceeding to step S959;

[0153] Step S958, obtaining all data corresponding to the time identifier of the acquisition request from the memory data table, and determining the data state as "read completely"; and proceeding to step S956;

[0154] Step S959, obtaining the data state "read completely", and proceeding to step S959.

[0155] The carrying unit can be as shown in the flow chart, specifically comprising: Figure 14

[0156] Step S961, obtaining the time identifier closest to the current time in the disk data table, the data to be added and the corresponding time identifier;

[0157] Step S962, waiting for acquiring the write lock;

[0158] Step S963, adding the data to be added and the corresponding time identifier to the disk data table;

[0159] Step S964, pointing the time identifier closest to the current time in the disk data table to the newly added time identifier;

[0160] ​Step S965, releasing the read lock, executing the landing logic, and returning to step S962.

[0161] It should be understood that, although each step in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or stages or steps or stages in other steps.

[0162] Based on the same inventive concept, the embodiments of the present application also provide a data processing apparatus for implementing the above-mentioned data processing method. The implementation scheme for solving the problem provided by the apparatus is similar to the implementation scheme described in the above method, so the specific limitations in one or more data processing apparatus embodiments provided below can refer to the limitations of the data processing method described above, which will not be repeated here.

[0163] In one embodiment, as shown in Figure 15 A data processing apparatus 100 is provided, comprising a data acquisition module 101, an identifier establishing module 102, and a data interaction module 103, wherein:

[0164] The data acquisition module is configured to, in response to a data addition request for a disk data table, acquire a request time of the data addition request, and determine a time identifier corresponding to the data addition request according to the request time;

[0165] The identifier establishing module is configured to, in a case where the time identifier does not exist in the disk data table, lock the disk data table, and establish the time identifier in the disk data table;

[0166] The data interaction module is configured to establish an association between the time identifier corresponding to the data addition request and the time identifiers in the disk data table, and add the data corresponding to the data addition request to a storage area corresponding to the newly established time identifier.

[0167] In one embodiment, the apparatus further comprises:

[0168] The identifier acquisition module is configured to, in response to an interaction request for target data, acquire time identifiers of each data in a disk data table;

[0169] The data storage module is configured to select a preset number of data from the disk data table into a memory data table by using the time identifiers in the disk data table, wherein the preset number is the maximum data storage capacity of the memory data table.

[0170] The data interaction module is further configured to perform corresponding interaction operations on the memory data table by using the interaction request.

[0171] In one of the embodiments, the disk data includes one or more data files; and the data storage module includes:

[0172] The number determination sub-module is configured to determine the number of target data files by using the maximum data storage capacity of the memory data table and the data storage capacity of the data files.

[0173] The file determination sub-module is configured to determine the target data files by using the number of target data files and the time identifiers of the data files.

[0174] The data storage sub-module is configured to select target data from the target data files into the memory data table in sequence by using the time identifiers of the target data files and the time identifiers of the data in the data files.

[0175] In one of the embodiments, the device further includes:

[0176] The request determination sub-module is configured to determine whether the time identifier is included in the acquisition request in response to the acquisition request for the target data.

[0177] The data acquisition sub-module is configured to acquire the data corresponding to the time identifier from the memory data table and acquire the data corresponding to the next time identifier in sequence in the case that the time identifier is included in the acquisition request.

[0178] The data acquisition sub-module is further configured to acquire the data corresponding to the first time identifier from the memory data table and acquire the data corresponding to the next time identifier in sequence in the case that the time identifier is not included in the acquisition request.

[0179] In one of the embodiments, the memory data table further includes an association relationship between the account and the data identifier; and the request determination sub-module includes:

[0180] The association relationship determination unit is configured to acquire the data identifier from the interaction request and determine whether the association relationship exists between the account and the data identifier by using the memory data table in response to the acquisition request for the target data by the account.

[0181] The request judging unit is configured to judge whether the time identifier is included in the acquisition request when the account and the data identifier have an association relationship.

[0182] In one of the embodiments, the apparatus further comprises:

[0183] The data adding module is configured to add the target data in the disk data table into the memory data table in response to the interaction request for the target data.

[0184] The data interaction module is configured to perform the interaction operation corresponding to the interaction request by using the memory data table when a preset time elapses.

[0185] The modules in the data processing apparatus can be realized by software, hardware and combinations thereof in whole or in part. The modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0186] In one of the embodiments, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in Figure 16 The computer device includes a processor, a memory, an input / output interface and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store task data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a data processing method.

[0187] Those skilled in the art can understand that Figure 16 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0188] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0189] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of each method can be included. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0190] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0191] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: In response to a data addition request to a disk data table, the request time of the data addition request is obtained, and a time identifier corresponding to the data addition request is determined based on the request time; wherein, determining the time identifier using the request time includes setting a preset time period, performing a quotient operation on the request time and the time period to obtain the corresponding time identifier; data within one period is stored in a storage area corresponding to a time identifier; If the time identifier does not exist in the disk data table, the disk data table is locked, and the time identifier is created in the disk data table. Establish the association between the time identifier corresponding to the data addition request and the time identifier in the disk data table, and add the data corresponding to the data addition request to the storage area corresponding to the newly created time identifier; the association between the time identifiers in the disk data table includes each time identifier pointing to the next time identifier, that is, the previous request time points to the next request time, and the time identifier corresponding to the last time identifier in the disk data table, that is, the latest request time, points to the newly created time identifier.

2. The method according to claim 1, characterized in that, The method further includes: In response to an interaction request for target data, retrieve the timestamp of each data item in the disk data table; Using the time identifier in the disk data table, a preset number of data items are selected from the disk data table and stored in the memory data table; wherein, the preset number is the maximum data storage capacity of the memory data table; The interaction request is used to perform corresponding interactive operations on the memory data table.

3. The method according to claim 2, characterized in that, The disk data includes one or more data files; the step of selecting a preset number of data files from the disk and storing them in a memory data table using the time identifier in the disk data table includes: The number of target data files is determined by using the maximum data storage capacity of the memory data table and the data storage capacity of the data file. The target data files are determined using the number of target data files and the timestamps of each data file; Using the timestamp of the target data file and the timestamp of each data item in the data file, target data is selected and stored sequentially in the memory data table.

4. The method according to claim 2, characterized in that, The method further includes: In response to a request to acquire target data, determine whether the acquisition request includes a time identifier; If the acquisition request includes a time identifier, the data corresponding to the time identifier is obtained from the memory data table, and the data corresponding to the next time identifier is obtained sequentially. If the time identifier is not included in the acquisition request, the data corresponding to the first time identifier is obtained from the memory data table, and the data corresponding to the next time identifier is obtained in sequence.

5. The method according to claim 4, characterized in that, The memory data table also includes the association between accounts and data identifiers; the step of responding to a request to obtain target data, determining whether the request includes a timestamp, includes: In response to an account's request to retrieve target data, a data identifier is obtained from the interaction request, and an in-memory data table is used to determine whether there is an association between the account and the data identifier; If the account and the data identifier are associated, determine whether the acquisition request includes a time identifier.

6. The method according to claim 2, characterized in that, The method further includes: In response to an interaction request for target data, the target data in the disk data table is added to the memory data table; After a preset time has elapsed, the interactive operation corresponding to the interactive request is performed using the memory data table.

7. A data processing apparatus, characterized in that, The device includes: The data acquisition module is used to respond to a data addition request to a disk data table, acquire the request time of the data addition request, and determine the time identifier corresponding to the data addition request based on the request time; wherein, determining the time identifier using the request time includes setting a preset time period, performing a quotient process on the request time and the time period to obtain the corresponding time identifier; data within one period is stored in a storage area corresponding to a time identifier; The identifier establishment module is used to lock the disk data table and establish the time identifier in the disk data table when the time identifier does not exist in the disk data table. The data interaction module is used to establish the association between the time identifier corresponding to the data addition request and the time identifier in the disk data table, and to add the data corresponding to the data addition request to the storage area corresponding to the newly created time identifier; the association between the time identifiers in the disk data table includes each time identifier pointing to the next time identifier, that is, the previous request time points to the next request time, and the time identifier corresponding to the last time identifier in the disk data table, that is, the latest request time, points to the newly created time identifier.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images