Method, device, storage medium and electronic device for data processing

Abstract

The application discloses a data processing method and device, a storage medium and an electronic device, and relates to the field of big data. The method comprises the following steps: acquiring access data of a transaction system by different access devices in a preset time period, and obtaining a plurality of access data; grouping the plurality of access data according to device IDs, and obtaining a plurality of data sets; respectively traversing the access data of each data set, generating an identifier, updating the identifier according to the time of the currently traversed access data in sequence, and removing repeated access data in the data set in the process of updating the identifier each time until the traversal is completed, and obtaining a plurality of data sets after the removal of repetition. Through the application, the problem that the stability and accuracy of the data removal operation are low in the related art in the scene with a large amount of data storage or the scene with disordered data time is solved.

Description

Technical Field

[0001] This application relates to the field of big data, and more specifically, to a data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Flink is a distributed real-time data processing engine that can process real-time data in a pipelined manner, supporting high throughput, low latency, and high-performance stateful computation. In real-time computing scenarios with large transaction volumes and long aggregation windows based on the Flink framework, the current industry approach is to group the data in the long, ultra-large window, split the large amount of data processed in the long pipeline into small, short windows for deduplication, and then perform deduplication on the data from the multiple small windows. This process yields a large deduplication result for the large amount of data, thereby improving the stability of data deduplication and reducing the amount of deduplicated data in each window, thus reducing the amount of data stored during deduplication and avoiding storage unit anomalies.

[0003] However, when deduplicating data based on the time dimension, since the data is determined sequentially in a pipeline manner to determine whether it is time-repeated, there are two methods to deduplicate data when the data volume is large.

[0004] Option 1 involves saving the data corresponding to the first occurrence of the data. When new data is obtained, it is checked whether the time of the new data is the same as the time of the already stored data. If they are the same, it indicates that the data is duplicated and needs to be removed; otherwise, it is stored, thus obtaining data with no duplicate times. However, Option 1 is only suitable for scenarios with small data volumes or short time spans. When the data volume is large or the time span between data points is wide, the amount of data to be stored will be very large, which may lead to data storage anomalies and reduce Flink's stability.

[0005] Option two involves storing data chronologically. If the time of new data is greater than the time of stored data, the new data replaces the stored data. If the time of new data is less than or equal to the time of stored data, the data is considered a duplicate and deduplicated. Option two significantly reduces the amount of data stored; however, it is prone to data loss when the data is out of order, leading to inaccurate deduplication results.

[0006] There is currently no effective solution to the problem of low stability and accuracy of data deduplication operations in scenarios with large data storage volumes or out-of-order data. Summary of the Invention

[0007] This application provides a data processing method, apparatus, storage medium, and electronic device to solve the problem of low stability and accuracy of data deduplication operations in scenarios with large data storage volume or out-of-order data time.

[0008] According to one aspect of this application, a data processing method is provided. The method includes: acquiring access data of different access devices to a transaction system within a preset time period, obtaining multiple access data sets, wherein each access data set includes at least the following information: device ID of the access device and access time; grouping the multiple access data sets according to the device ID, obtaining multiple data sets, wherein the device ID of the access data in each data set is the same; traversing the access data in each data set, generating an identifier during the traversal, updating the identifier sequentially according to the access time of the currently traversed access data, and deduplicating duplicate access data in each identifier update process, until the traversal is complete, obtaining multiple deduplicated data sets, wherein the identifier is used to record the access time of each traversed access data set, and duplicate access data refers to access data with the same device ID and access time.

[0009] Optionally, generating an identifier during the traversal includes: obtaining the maximum and minimum access times of the accessed data in the data set; calculating the time difference between the maximum and minimum access times to obtain a first difference value, and generating an initial identifier with the number of flag bits equal to the first difference value; setting the parameter of each flag bit in the initial identifier to a first preset value to obtain the identifier.

[0010] Optionally, the identifier is updated sequentially based on the access time of the currently traversed access data, and duplicate access data in the data set is deduplicated during each identifier update until the traversal is complete, resulting in multiple deduplicated data sets. This includes: reading any access data in the data set to obtain the first access data, and obtaining the access time of the first access data to obtain the first access time; generating a timestamp for the data set based on the first access time, moving the first access data to the data pool corresponding to the data set, and updating the data set, wherein the timestamp represents the maximum access time of the currently traversed access data, and the data pool is used to store the deduplicated access data; reading any access data in the updated data set to obtain the second access data, and obtaining the access time of the second access data to obtain the second access time; determining whether the second access data is duplicated access data based on the second access time, timestamp, and identifier, obtaining the determination result, updating the timestamp, identifier, and data set based on the determination result, until the data set is empty, and determining the deduplicated data set based on the multiple access data in the data pool.

[0011] Optionally, determining whether the second access data is duplicate access data using the second access time, timestamp, and identifier, and updating the timestamp, identifier, and data set based on the determination result includes: comparing the second access time with the timestamp; if the second access time is greater than the timestamp, changing the identifier based on the second access time and timestamp to obtain an updated identifier, changing the timestamp to the second access time to obtain an updated timestamp, and moving the second access data to the data pool corresponding to the data set; if the second access time is equal to the timestamp, deleting the second access data; if the second access time is less than the timestamp, determining whether the second access data is duplicate data using the second access time, identifier, and timestamp.

[0012] Optionally, the identifier includes multiple flag bits and parameters for each flag bit. If the second access time is greater than the timestamp, the identifier is changed according to the second access time and the timestamp to obtain the updated identifier. This includes: determining the number of bits in the flag bits whose parameters are second preset values ​​to obtain multiple first digits, wherein the second preset value is different from the first preset value; subtracting the timestamp from each first digit in turn to obtain multiple access times that have occurred, and determining the multiple access times and timestamps that have occurred as multiple access times that have occurred; determining the time difference between the second access time and each access time that has occurred to obtain multiple third time differences; updating the parameters of the flag bits corresponding to each third time difference in the identifier to the second preset value, and setting the parameters of the remaining flag bits in the identifier to the first preset value to obtain the updated identifier.

[0013] Optionally, the identifier includes multiple flag bits and parameters for each flag bit. When the second access time is less than the timestamp, determining whether the second access data is duplicated data based on the second access time, the identifier, and the timestamp includes: calculating the difference between the second access time and the timestamp to obtain a second difference; obtaining the flag bit in the identifier that corresponds to the second difference to obtain a target flag bit; determining whether the parameter of the target flag bit is a first preset value; if the parameter is the first preset value, determining that the second access time is not duplicated, changing the identifier according to the second access time and timestamp to obtain an updated identifier, and moving the second access data to the data pool corresponding to the data set; if the parameter is the second preset value, deleting the second access data, where the second preset value is different from the first preset value.

[0014] Optionally, when the parameter is a first preset value, if it is determined that the second access time has not occurred repeatedly, the identifier is changed according to the second access time and timestamp to obtain the updated identifier, which includes: determining the number of bits in the flag bit of the identifier where the parameter is the second preset value, to obtain multiple second bits; subtracting the timestamp from each second bit in turn to obtain multiple access times that have occurred, and determining the multiple access times and the second access time as multiple access times that have occurred; determining the time difference between the timestamp and each access time that has occurred, to obtain multiple fourth time differences; setting the parameter of the flag bit corresponding to each fourth time difference in the identifier to the second preset value, and setting the parameter of the remaining flag bits in the identifier to the first preset value, to obtain the updated identifier.

[0015] According to another aspect of this application, a data processing apparatus is provided. The apparatus includes: an acquisition unit, configured to acquire access data of a transaction system from different access devices within a preset time period, obtaining multiple access data sets, wherein each access data set includes at least the following information: device ID of the accessing device and access time; a grouping unit, configured to group the multiple access data sets according to the device ID, obtaining multiple data sets, wherein the device ID of the access data in each data set is the same; and a deduplication unit, configured to traverse the access data in each data set, generate an identifier during the traversal, update the identifier sequentially according to the access time of the currently traversed access data, and deduplicate the access data that appears repeatedly in the data set during each identifier update, until the traversal is complete, obtaining multiple deduplicated data sets, wherein the identifier is used to record the access time of each traversed access data set, and the repeatedly appearing access data refers to access data with the same device ID and access time.

[0016] According to another aspect of the present invention, a computer storage medium is also provided for storing a program, wherein the program, when running, controls the device where the computer storage medium is located to perform a data processing method.

[0017] According to another aspect of the present invention, an electronic device is also provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed, perform a data processing method.

[0018] This application employs the following steps: First, it acquires access data from different devices accessing the transaction system within a preset time period, resulting in multiple access data sets. Each access data set includes at least the following information: the device ID of the accessing device and the access time. Second, it groups the multiple access data sets according to the device ID, resulting in multiple data collections. Each data collection contains access data with the same device ID. Third, iterates through the access data in each data collection, generating an identifier during the iteration. The identifier is updated sequentially based on the access time of the currently iterated access data, and duplicate access data is deduplicated during each identifier update until all iterations are complete, resulting in multiple deduplicated data collections. The identifier records the access time of each iterated access data set, and duplicate access data refers to access data with the same device ID and access time. This solves the problem of low stability and accuracy in data deduplication operations in related technologies, particularly in scenarios with large data storage volumes or out-of-order data. By setting an identifier corresponding to each device ID and setting an initial identifier, and then checking whether the access time of each piece of data is duplicated, and changing the identifier according to the check result, the access time that has occurred and the access time that has not occurred can be determined by the identifier and the current access time. Thus, the data storage volume can be reduced by using the identifier, and the access time can be prevented from being missed. In this way, the data storage volume can be reduced while preventing data loss due to data time disorder. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a flowchart of a data processing method provided according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of an optional process for deduplicating accessed data according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram of a data processing apparatus provided according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent information from the aforementioned user or organization.

[0028] It should be noted that the data processing methods, apparatus, storage media, and electronic devices defined in this disclosure can be used in the field of big data, or in any field other than big data. The application fields of the data processing methods, apparatus, storage media, and electronic devices defined in this disclosure are not limited.

[0029] According to embodiments of this application, a data processing method is provided.

[0030] Figure 1 This is a flowchart of a data processing method provided according to an embodiment of this application. For example... Figure 1 As shown, the method includes the following steps:

[0031] Step S101: Obtain access data of different access devices to the transaction system within a preset time period, and obtain multiple access data. Each access data includes at least the following information: device ID of the access device and access time.

[0032] Specifically, the Flink processing engine can obtain access data within the time period that needs to be deduplicated as the data source, and determine the device ID and access time of each data access device in sequence through a pipeline.

[0033] It should be noted that in order to determine a user's daily active user information, or the usage trend and time of a user's use of a certain application, or to analyze the user's usage characteristics, it is possible to obtain the access information of each user in a certain time period. For example, it is possible to obtain the number of times each user accesses the application in a certain day. However, since multiple accesses may be triggered within a minute, it is necessary to deduplicate the second-level data and keep only the minute-level data. At this time, all access data of the application for the day can be deduplicated, and only the access data of each device at the minute level can be kept, thereby greatly reducing the amount of data to be analyzed.

[0034] Step S102: Group multiple access data according to device ID to obtain multiple data sets, wherein the access data in each data set has the same device ID.

[0035] Specifically, after acquiring a large amount of access data, multiple access data can be grouped according to the device ID of each data to obtain multiple data sets, so that the device IDs of all access data in each data set are the same. This ensures that each data set obtained after deduplication includes access requests with no duplicate access times corresponding to each device.

[0036] Step S103: Iterate through the access data in each data set, generate an identifier during the iteration, update the identifier according to the access time of the currently iterated access data, and remove duplicate access data in the data set during each identifier update until the iteration is complete, resulting in multiple deduplicated data sets. The identifier is used to record the access time of each iterated access data, and duplicate access data refers to access data with the same device ID and access time.

[0037] Specifically, after obtaining multiple data sets, the access time of each accessed data can be traversed sequentially in each data set to identify accessed data with duplicate access times from multiple accessed data, and then the accessed data with duplicate access times can be deleted, thereby completing the data deduplication.

[0038] It should be noted that when determining the access time of each piece of data, the access time of the data can be determined based on the identifier in each data set to determine whether the access time of the data is a repeated time. After determination, the identifier is changed accordingly, so that the identifier can change accordingly as the number of data with repeated times increases. In this way, the time of subsequent access data can be judged and deduplicated only based on the identifier information and the current time information.

[0039] Furthermore, after obtaining the deduplicated dataset, multiple deduplicated datasets can be used as updated access data. The frequency of occurrence of access data for each device is then determined from the updated access data. Based on the frequency of occurrence, the user's interest level for each device is determined, where interest level represents the user's level of interest in the accessed object indicated by the access data; the more times the access data appears, the higher the user's interest level. Using this dataset to analyze user behavior improves the convenience of user behavior analysis compared to using the undeduplicated data.

[0040] For example, the data to be deduplicated could be browsing data obtained from each user's browsing of a video in an application using their electronic device. To obtain users' browsing records at the minute level, browsing data with the same minute information can be identified as duplicates and deduplicated, resulting in deduplicated browsing information where the browsing time for each device is unique at the minute level. For instance, if the original browsing data is 12:10:10, 12:10:15, 12:10:20, and 12:11:10, then after deduplication at the minute level, it becomes 12:10:10 and 12:11:10. After obtaining the deduplicated browsing information, it can be determined that the user only browsed the video for two minutes, therefore, the user's interest in the video is low.

[0041] The data processing method provided in this application embodiment obtains access data of a transaction system from different access devices within a preset time period, resulting in multiple access data sets. Each access data set includes at least the following information: the device ID of the accessing device and the access time. The multiple access data sets are grouped according to the device ID to obtain multiple data sets, where the device IDs of the access data in each data set are the same. The access data in each data set is traversed, and an identifier is generated during the traversal. The identifier is updated sequentially according to the access time of the currently traversed access data, and duplicate access data is deduplicated during each identifier update until the traversal is complete, resulting in multiple deduplicated data sets. The identifier records the access time of each traversed access data set, and duplicate access data refers to access data with the same device ID and access time. This method solves the problem of low stability and accuracy of data deduplication operations in related technologies, particularly in scenarios with large data storage volumes or out-of-order data. By setting an identifier corresponding to each device ID and setting an initial identifier, and then checking whether the access time of each piece of data is duplicated, and changing the identifier according to the check result, the access time that has occurred and the access time that has not occurred can be determined by the identifier and the current access time. Thus, the data storage volume can be reduced by using the identifier, and the access time can be prevented from being missed. In this way, the data storage volume can be reduced while preventing data loss due to data time disorder.

[0042] Optionally, in the data processing method provided in this application embodiment, generating an identifier during the traversal process includes: obtaining the maximum access time and minimum access time of the accessed data in the data set; calculating the time difference between the maximum access time and the minimum access time to obtain a first difference value, and generating an initial identifier with the number of flag bits equal to the first difference value; setting the parameter of each flag bit in the initial identifier to a first preset value to obtain the identifier.

[0043] It should be noted that the identifier consists of multiple flag bits and parameters for each flag bit. Figure 1 As an optional identifier, as shown in Table 1, the identifier includes 9 flag bits, and the parameter of each flag bit is 0.

[0044] Table 1

[0045] Flag Position 9 Position 8 Position 7 Position 6 Position 5 Position 4 Position 3 Position 2 Position 1 parameter 0 0 0 0 0 0 0 0 0

[0046] Specifically, in order to determine whether an access time is a repeated access time through the identifier, the parameters in the corresponding flag bits can be changed. Therefore, it is necessary to determine the number of bits in the flag bits of each access set based on the maximum and minimum access times of the accessed data in each data set, so as to ensure that all access times can be represented based on the identifier.

[0047] After obtaining the maximum and minimum access times of the accessed data in the dataset, the difference needs to be determined according to the time level. For example, if the maximum access time is 19:30 and the minimum access time is 19:20, then the flag bit of the identifier needs to be set to 11 bits.

[0048] After determining the number of bits in the flag array, each flag bit needs to be initialized to 0. This allows the user to check the time of the first accessed data in the dataset and update the flag accordingly. This achieves the goal of registering each time point using the flag and then determining whether the access time is a repeating time.

[0049] Optionally, in the data processing method provided in this application embodiment, updating the identifier sequentially according to the access time of the currently traversed access data, and deduplicating the access data that appears repeatedly in the data set during each identifier update process, until the traversal is completed, and obtaining multiple deduplicated data sets includes: reading any access data in the data set to obtain first access data, and obtaining the access time of the first access data to obtain the first access time; generating a timestamp of the data set according to the first access time, moving the first access data to the data pool corresponding to the data set, and updating the data set, wherein the timestamp represents the maximum access time of the currently traversed access data, and the data pool is used to store the deduplicated access data; reading any access data in the updated data set to obtain second access data, and obtaining the access time of the second access data to obtain the second access time; determining whether the second access data is duplicated access data through the second access time, timestamp, and identifier, obtaining a determination result, updating the timestamp, identifier, and data set according to the determination result, until the data set is an empty set, and determining the deduplicated data set according to the multiple access data in the data pool.

[0050] Specifically, after generating the identifier, any access data can be read from the dataset as the first access data to be judged, i.e., the first access data. Since the first access data will definitely not have a duplicate access time, the access time of this access data, i.e., the first access time, can be used as the timestamp of the dataset, and the first access data is sent to the data pool storing the filtered access data. At this time, the identifier corresponding to the dataset remains unchanged, the corresponding timestamp is the first access time, and the dataset is reduced by one access data.

[0051] Furthermore, when reading any subsequent access data, it is necessary to compare the access time of the access data, i.e. the second access time, with the timestamp, and determine whether the access data is accessed repeatedly based on the relationship between the second access time and the timestamp. Based on the determination result, the timestamp, identifier, and data set are updated.

[0052] The process described above iterates through each accessed data in the data set, makes a judgment based on the relationship between the access time and the timestamp, and updates the timestamp, identifier, and data set until there is no data in the data set, which means the deduplication of the data set is completed, and the deduplicated data result of the data set is obtained from the data pool.

[0053] Optionally, in the data processing method provided in this application embodiment, determining whether the second access data is duplicate access data by using the second access time, timestamp, and identifier to obtain a determination result, and updating the timestamp, identifier, and data set according to the determination result includes: comparing the second access time with the timestamp; if the second access time is greater than the timestamp, changing the identifier according to the second access time and timestamp to obtain an updated identifier, changing the timestamp to the second access time to obtain an updated timestamp, and moving the second access data to the data pool corresponding to the data set; if the second access time is equal to the timestamp, deleting the second access data; if the second access time is less than the timestamp, determining whether the second access data is duplicate data by using the second access time, identifier, and timestamp.

[0054] Specifically, after comparing the second access time with the timestamp, if the second access time is greater than the timestamp, it indicates that the access data corresponding to the second access time definitely does not have duplicate data in the currently judged data. Therefore, the identifier can be changed based on the second access time and the timestamp, thus recording the existence of the second access time in the identifier, ensuring that subsequent access data with the second access time can be detected as duplicate data. After changing the identifier, the timestamp needs to be changed to the second access time, thus changing the timestamp to the largest time node among the confirmed time nodes. After changing the timestamp and identifier, the second access data can be sent to the storage pool to update the data in the dataset.

[0055] Furthermore, if the second access time equals the timestamp, it indicates that the second access time must be a repeated access time, so the second access data corresponding to the second access time can be directly deleted.

[0056] Furthermore, if the second access time is less than the timestamp, since the method of changing the identifier and timestamp cannot be directly determined, it is necessary to further judge the second access time using the second access time, identifier, and timestamp to ensure the accuracy of the judgment of the second access time.

[0057] Optionally, in the data processing method provided in this application embodiment, the identifier includes multiple flag bits and parameters for each flag bit. When the second access time is greater than the timestamp, the identifier is changed according to the second access time and the timestamp to obtain the updated identifier. This includes: determining the number of bits in the flag bits whose parameters are second preset values ​​in the identifier to obtain multiple first digits, wherein the second preset value is different from the first preset value; subtracting the timestamp from each first digit in turn to obtain multiple access times that have occurred, and determining the multiple access times and timestamps that have occurred as multiple access times that have occurred; determining the time difference between the second access time and each access time that has occurred to obtain multiple third time differences; updating the parameters of the flag bits corresponding to each third time difference in the identifier to the second preset value, and setting the parameters of the remaining flag bits in the identifier to the first preset value to obtain the updated identifier.

[0058] Specifically, when changing an identifier, the first step is to determine the position of the second preset value in the current identifier. This second preset value indicates that the access time corresponding to the flag bit of the second preset value has been recorded. After determining the flag bit of each second preset value, the time corresponding to each flag bit needs to be determined based on the current timestamp, thus obtaining multiple access times that have occurred.

[0059] For example, the first preset value can be 0, the second preset value can be 1, the current timestamp is 19:30, and the current identifier is shown in Table 2. Then, the parameters of position 2 and position 4 can be determined to be the second preset value. At this time, by subtracting 2 minutes and 4 minutes from 19:30, we can obtain the access times that have occurred as 19:28 and 19:26, thus obtaining the access times that have occurred as 19:30, 19:28, and 19:26.

[0060] Table 2

[0061] Flag Position 5 Position 4 Position 3 Position 2 Position 1 parameter 0 1 0 1 0

[0062] After obtaining the access times that have occurred, the difference between each access time and the second access time can be determined, and the parameters of the corresponding flag bits in the identifier can be changed according to the difference.

[0063] For example, the second access time can be 19:31. If the access times that have already occurred are 19:30, 19:28, and 19:26, subtract the second access time from each of the already occurred access times to get a difference of 1, 3, and 5. Then, change the corresponding flag bits in the identifier, namely bits 1, 3, and 5, to the second preset value, which is 1, and change the parameters of all other flag bits to the first preset value, which is 0. The changed identifier is shown in Table 3.

[0064] Table 3

[0065] Flag Position 5 Position 4 Position 3 Position 2 Position 1 parameter 1 0 1 0 1

[0066] At this point, the access time of subsequent access data can be determined based on the updated identifier and the second access time, which is also the changed timestamp. This allows the identifier to replace a large amount of stored data in determining whether the data time is duplicated, greatly reducing the amount of data stored and improving the stability of the entire deduplication system.

[0067] Optionally, in the data processing method provided in this application embodiment, the identifier includes multiple flag bits and parameters for each flag bit. When the second access time is less than the timestamp, determining whether the second access data is duplicated data by using the second access time, the identifier, and the timestamp includes: calculating the difference between the second access time and the timestamp to obtain a second difference; obtaining the flag bit in the identifier that is the second difference to obtain a target flag bit; determining whether the parameter of the target flag bit is a first preset value; when the parameter is the first preset value, determining that the second access time is not duplicated, changing the identifier according to the second access time and the timestamp to obtain an updated identifier, and moving the second access data to the data pool corresponding to the data set; when the parameter is the second preset value, deleting the second access data, wherein the second preset value is different from the first preset value.

[0068] Specifically, when the second access time equals the timestamp, the second access time may be a duplicate or it may never have appeared. Therefore, it is necessary to determine whether the second access data is a duplicate based on the identifier and the timestamp.

[0069] First, calculate the time difference between the second access time and the timestamp to obtain the difference value, and determine whether the parameter of the flag bit corresponding to the difference value in the identifier is the first preset value. If it is the first preset value, the second access time is not repeated. If it is the second preset value, the second access time is a repeated time, and the second access data corresponding to the second access time needs to be deleted.

[0070] For example, the identifier can be as shown in Table 4. If the timestamp is 20:00 and the second access time is 19:58, then the difference is 2, and the corresponding 2nd bit is 1, which is the second preset value. Therefore, the second access time is a repeated time, and the second access data corresponding to the second access time needs to be deleted.

[0071] Table 4

[0072] Flag Position 5 Position 4 Position 3 Position 2 Position 1 parameter 0 0 1 1 1

[0073] Optionally, in the data processing method provided in this application embodiment, when the parameter is a first preset value, determining that the second access time has not occurred repeatedly, and changing the identifier according to the second access time and timestamp to obtain the updated identifier includes: determining the number of bits in the flag bit of the identifier where the parameter is the second preset value, to obtain multiple second bits; sequentially subtracting the timestamp from each second bit to obtain multiple access times that have occurred, and determining the multiple access times and the second access time as multiple access times that have occurred; determining the time difference between the timestamp and each access time that has occurred, to obtain multiple fourth time differences; setting the parameter of the flag bit corresponding to each fourth time difference in the identifier to the second preset value, and setting the parameter of the remaining flag bits in the identifier to the first preset value, to obtain the updated identifier.

[0074] Specifically, if the parameter of the flag corresponding to the second access time is the first preset value, the identifier needs to be changed to record the second access time. When changing the identifier, the position of the second preset value in the current identifier must first be determined. The second preset value indicates that the access time corresponding to the flag corresponding to the second preset value has been recorded. After determining the flag for each second preset value, the time corresponding to each flag needs to be determined based on the current timestamp, thus obtaining multiple access times that have occurred.

[0075] For example, the first preset value can be 0, the second preset value can be 1, the current timestamp is 19:30, the second access time can be 19:29, and the current identifier is shown in Table 5. Then, the parameters in positions 2 and 4 can be determined to be the second preset value. At this time, by subtracting 2 minutes and 4 minutes from 19:30, we can obtain the access times that have occurred as 19:26 and 19:28. These can then be combined with the second access time to obtain the access times that have occurred as 19:29, 19:28, and 19:26.

[0076] Table 5

[0077] Flag Position 5 Position 4 Position 3 Position 2 Position 1 parameter 0 1 0 1 0

[0078] After obtaining the access times of 19:29, 19:28 and 19:26, we can calculate the time difference between each access time and the timestamp, and obtain time differences of 1, 2 and 4. The updated identifiers are shown in Table 6.

[0079] Table 6

[0080] Flag Position 5 Position 4 Position 3 Position 2 Position 1 parameter 0 1 0 1 1

[0081] By using the above method, it is possible to ensure that no access data at any point in time is lost, even when the time order is out of order.

[0082] Figure 2 This is a schematic diagram of an optional process for deduplicating accessed data according to an embodiment of this application, such as... Figure 2 As shown, it should be noted that since the flags are too long to be easily displayed, only the changes in the parameters of the four flags, flag 1 to flag 4, are shown to complete the deduplication of part of the data. The parameters from right to left are the parameters of flag 1 to flag 4.

[0083] Specifically, during the process of traversing the data in the dataset, the access time of the first access data 1 read is 00:01. At this time, the timestamp is set to 0.01, and the identifier is not changed. The parameter in the identifier is four zeros. Then, access data 1 is output to the storage pool.

[0084] Furthermore, the access time of the next access data 2 is 00:02. At this time, it is determined that 00:02 is greater than the timestamp 00:01. Therefore, the flag is changed. Since there is a difference of 1 between 00:02 and 00:01, the flag bit 1 can be changed from 0 to 1. At this time, the parameter in the flag is 0001, and the timestamp is changed to 00:02. At the same time, access data 2 is output to the storage pool.

[0085] Furthermore, the access time of the next access data 3 is 00:04. At this time, it is determined that 00:04 is greater than the timestamp 00:02. Therefore, the identifier is changed. Since there is a difference of 2 between 00:04 and 00:02 and a difference of 3 between 00:01 and 00:04, flag bits 2 and 3 can be changed from 0 to 1, and flag bit 1 can be changed from 1 to 0. At this time, the parameter in the identifier is 0110, and the timestamp is changed to 00:04. At the same time, access data 3 is output to the storage pool.

[0086] Furthermore, the access time of the next access data 4 is 00:01. At this time, it is determined that 00:01 is less than the timestamp 00:04. Since there is a difference of 3 between 00:04 and 00:01, it is necessary to determine whether the parameter of flag bit 3 in the identifier is 1. Since the identifier is 0110 at this time, the parameter of flag bit 3 is 1. Therefore, the access data 4 is determined to be duplicate data and is deleted. At this time, the timestamp and the identifier remain unchanged.

[0087] Furthermore, the access time of the next access data 5 is 00:03. At this time, it is determined that 00:03 is less than the timestamp 00:04. Since there is a difference of 1 between 00:04 and 00:03, it is necessary to determine whether the parameter of flag bit 1 in the identifier is 1. Since the identifier is 0110 at this time, the parameter of flag bit 1 is not 1. Therefore, it is determined that the access data 5 is not duplicate data, and the parameter of flag bit 1 is changed from 0 to 1 according to the difference of 1. At this time, the identifier becomes 0111, but the timestamp does not change.

[0088] By traversing all the data in the dataset using the method described above, a deduplicated dataset can be obtained from the data pool, thus achieving the technical effect of accurately deduplicating accessed data without occupying a large amount of data storage space.

[0089] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0090] This application also provides a data processing apparatus. It should be noted that the data processing apparatus of this application can be used to execute the data processing method provided in this application. The data processing apparatus provided in this application will be described below.

[0091] Figure 3 This is a schematic diagram of a data processing apparatus provided according to an embodiment of this application. Figure 3As shown, the device includes: an acquisition unit 31, a grouping unit 32, and a deduplication unit 33.

[0092] The acquisition unit 31 is used to acquire access data of different access devices to the transaction system within a preset time period, and obtain multiple access data. Each access data includes at least the following information: device ID of the access device and access time.

[0093] Grouping unit 32 is used to group multiple access data according to device ID to obtain multiple data sets, wherein the access data in each data set have the same device ID.

[0094] The deduplication unit 33 is used to traverse the access data in each data set, generate an identifier during the traversal, update the identifier according to the access time of the currently traversed access data, and deduplicate the access data that appears repeatedly in the data set during each identifier update until the traversal is completed, resulting in multiple deduplicated data sets. The identifier is used to record the access time of each traversed access data, and the repeated access data refers to access data with the same device ID and access time.

[0095] The data processing apparatus provided in this application embodiment acquires access data of different accessing devices to a transaction system within a preset time period through an acquisition unit 31, obtaining multiple access data sets. Each access data set includes at least the following information: the device ID of the accessing device and the access time. A grouping unit 32 groups the multiple access data sets according to the device ID, obtaining multiple data sets. The device IDs of the access data in each data set are the same. A deduplication unit 33 traverses the access data in each data set, generating an identifier during the traversal process. The identifier is updated sequentially according to the access time of the currently traversed access data, and duplicate access data is deduplicated in each identifier update process until the traversal is complete, resulting in multiple deduplicated data sets. The identifier is used to record the access time of each traversed access data set, and duplicate access data refers to access data with the same device ID and access time. This solves the problem of low stability and accuracy of data deduplication operations in related technologies in scenarios with large data storage volumes or out-of-order data times. By setting an identifier corresponding to each device ID and setting an initial identifier, and then checking whether the access time of each piece of data is duplicated, and changing the identifier according to the check result, the access time that has occurred and the access time that has not occurred can be determined by the identifier and the current access time. Thus, the data storage volume can be reduced by using the identifier, and the access time can be prevented from being missed. In this way, the data storage volume can be reduced while preventing data loss due to data time disorder.

[0096] Optionally, in the data processing apparatus provided in this application embodiment, the deduplication unit 33 includes: a first acquisition subunit, used to acquire the maximum access time and minimum access time of accessed data in the data set; a calculation subunit, used to calculate the time difference between the maximum access time and the minimum access time, obtain a first difference value, and generate an initial identifier with the number of flag bits equal to the first difference value; and a setting subunit, used to set the parameter of each flag bit in the initial identifier to a first preset value, thereby obtaining an identifier.

[0097] Optionally, in the data processing apparatus provided in this application embodiment, the deduplication unit 33 includes: a second acquisition subunit, configured to read any access data in the data set to obtain first access data, and acquire the access time of the first access data to obtain the first access time; an update subunit, configured to generate a timestamp of the data set according to the first access time, move the first access data to the data pool corresponding to the data set, and update the data set, wherein the timestamp represents the maximum access time of the currently traversed access data, and the data pool is used to store the deduplicated access data; a third acquisition subunit, configured to read any access data in the updated data set to obtain second access data, and acquire the access time of the second access data to obtain the second access time; and a judgment subunit, configured to judge whether the second access data is duplicated access data through the second access time, timestamp, and identifier, obtain a judgment result, update the timestamp, identifier, and data set according to the judgment result until the data set is an empty set, and determine the deduplicated data set according to the multiple access data in the data pool.

[0098] Optionally, in the data processing apparatus provided in this application embodiment, the determination subunit includes: a comparison module, used to compare the second access time with the timestamp; an update module, used to, when the second access time is greater than the timestamp, change the identifier according to the second access time and the timestamp to obtain an updated identifier, change the timestamp to the second access time to obtain an updated timestamp, and move the second access data to the data pool corresponding to the data set; a deletion module, used to delete the second access data when the second access time is equal to the timestamp; and a determination module, used to, when the second access time is less than the timestamp, determine whether the second access data is duplicate data based on the second access time, the identifier, and the timestamp.

[0099] Optionally, in the data processing apparatus provided in this application embodiment, the identifier includes multiple flag bits and parameters for each flag bit. The update module includes: a first determining submodule, used to determine the number of bits in the flag bits whose parameters are second preset values ​​in the identifier, to obtain multiple first digits, wherein the second preset value is different from the first preset value; a first calculating submodule, used to subtract each first digit from the timestamp sequentially to obtain multiple access times that have occurred, and to determine the multiple access times and timestamps as multiple access times that have occurred; a second determining submodule, used to determine the time difference between the second access time and each access time that has occurred, to obtain multiple third time differences; and a first updating submodule, used to update the parameters of the flag bits corresponding to each third time difference in the identifier to the second preset value, and to set the parameters of the remaining flag bits in the identifier to the first preset value, to obtain the updated identifier.

[0100] Optionally, in the data processing apparatus provided in this application embodiment, the identifier includes multiple flag bits and parameters for each flag bit. The judgment module includes: a second calculation submodule, used to calculate the difference between the second access time and the timestamp to obtain a second difference; an acquisition submodule, used to acquire the flag bit in the identifier that is the second difference to obtain a target flag bit; a judgment submodule, used to determine whether the parameter of the target flag bit is a first preset value; a third determination submodule, used to determine that the second access time has not repeated when the parameter is the first preset value, change the identifier according to the second access time and timestamp to obtain an updated identifier, and move the second access data to the data pool corresponding to the data set; and a deletion submodule, used to delete the second access data when the parameter is the second preset value, wherein the second preset value is different from the first preset value.

[0101] Optionally, in the data processing apparatus provided in this application embodiment, the third determining submodule includes: a fourth determining submodule, used to determine the number of bits in the flag bits of the identifier whose parameters are second preset values, to obtain a plurality of second bits; a third calculating submodule, used to sequentially subtract each second bit from the timestamp to obtain a plurality of access times that have occurred, and to determine the plurality of access times and the second access times as a plurality of access times that have occurred; a fifth determining submodule, used to determine the time difference between the timestamp and each access time that has occurred, to obtain a plurality of fourth time differences; and a second updating submodule, used to set the parameter of the flag bit corresponding to each fourth time difference in the identifier to the second preset value, and to set the parameter of the remaining flag bits in the identifier to the first preset value, to obtain an updated identifier.

[0102] The aforementioned data processing device includes a processor and a memory. The aforementioned acquisition unit 31, grouping unit 32, deduplication unit 33, etc., are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0103] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the low stability and accuracy issues in data deduplication operations, particularly in scenarios with large data storage volumes or out-of-order data.

[0104] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0105] This invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the data processing method.

[0106] This invention provides a processor for running a program, wherein the program executes the data processing method during runtime.

[0107] like Figure 4 As shown, this embodiment of the invention provides an electronic device 40, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: acquiring access data from different accessing devices to a transaction system within a preset time period, resulting in multiple access data sets. Each access data set includes at least the following information: the device ID of the accessing device and the access time. The multiple access data sets are grouped according to the device IDs to obtain multiple data sets, where the device IDs of the access data in each data set are the same. The access data in each data set is traversed, and an identifier is generated during the traversal. The identifier is updated sequentially according to the access time of the currently traversed access data, and duplicate access data is deduplicated during each identifier update until the traversal is complete, resulting in multiple deduplicated data sets. The identifier records the access time of each traversed access data set, and duplicate access data refers to access data with the same device ID and access time. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0108] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: acquiring access data of different access devices to the transaction system within a preset time period, obtaining multiple access data, wherein each access data includes at least the following information: device ID of the access device and access time; grouping the multiple access data according to the device ID to obtain multiple data sets, wherein the device ID of the access data in each data set is the same; traversing the access data in each data set respectively, generating an identifier during the traversal process, updating the identifier sequentially according to the access time of the currently traversed access data, and deduplicating the access data that appears repeatedly in the data set during each identifier update process, until the traversal is completed, obtaining multiple deduplicated data sets, wherein the identifier is used to record the access time of each traversed access data, and the repeated access data refers to access data with the same device ID and access time.

[0109] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0114] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0115] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0116] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0117] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data processing method, characterized in that, include: Obtain access data of the transaction system from different access devices within a preset time period to obtain multiple access data. Each access data shall include at least the following information: device ID of the accessing device and access time. The multiple access data are grouped according to the device ID to obtain multiple data sets, wherein the access data in each data set have the same device ID; The access data in each data set is traversed one by one. During the traversal, an identifier is generated. The identifier is updated according to the access time of the access data being traversed. In each update of the identifier, duplicate access data in the data set is deduplicated until the traversal is completed, resulting in multiple deduplicated data sets. The identifier is used to record the access time of each traversed access data. Duplicate access data refers to access data with the same device ID and access time. The process of generating an identifier during traversal includes: obtaining the maximum and minimum access times of the accessed data in the data set; calculating the time difference between the maximum and minimum access times to obtain a first difference value, and generating an initial identifier with the number of flag bits equal to the first difference value; and setting the parameter of each flag bit in the initial identifier to a first preset value to obtain the identifier.

2. The method according to claim 1, characterized in that, The identifier is updated sequentially based on the access time of the currently traversed access data. During each identifier update, duplicate access data in the data set is removed until the traversal is complete, resulting in multiple deduplicated data sets, including: Read any access data from the data set to obtain the first access data, and obtain the access time of the first access data to obtain the first access time; The timestamp of the data set is generated based on the first access time, the first access data is moved to the data pool corresponding to the data set, and the data set is updated. The timestamp represents the maximum access time of the currently traversed access data, and the data pool is used to store the deduplicated access data. Read any access data from the updated dataset to obtain the second access data, and obtain the access time of the second access data to obtain the second access time; The system determines whether the second access data is duplicate access data by using the second access time, the timestamp, and the identifier, obtains a determination result, updates the timestamp, the identifier, and the data set according to the determination result, until the data set is an empty set, and determines the deduplicated data set based on multiple access data in the data pool.

3. The method according to claim 2, characterized in that, Determining whether the second access data is duplicate access data by using the second access time, the timestamp, and the identifier, obtaining a determination result, and updating the timestamp, the identifier, and the data set according to the determination result includes: The second access time is compared with the timestamp; If the second access time is greater than the timestamp, the identifier is changed according to the second access time and the timestamp to obtain an updated identifier, the timestamp is changed to the second access time to obtain an updated timestamp, and the second access data is moved to the data pool corresponding to the data set; If the second access time is equal to the timestamp, delete the second access data; If the second access time is less than the timestamp, the second access data is determined to be duplicate data by using the second access time, the identifier, and the timestamp.

4. The method according to claim 3, characterized in that, The identifier includes multiple flag bits and parameters for each flag bit. If the second access time is greater than the timestamp, the identifier is changed based on the second access time and the timestamp to obtain the updated identifier, which includes: The number of bits in the flag bit of the identifier whose parameter is a second preset value is determined to obtain multiple first digits, wherein the second preset value is different from the first preset value; The timestamp is subtracted from each of the first digits in sequence to obtain multiple access times that have occurred, and the multiple access times that have occurred and the timestamp are determined as multiple access times that have occurred; Determine the time difference between the second access time and each access time that has occurred to obtain multiple third time differences; In the identifier, the parameter of the flag bit corresponding to each third time difference is updated to the second preset value, and the parameters of the remaining flag bits in the identifier are set to the first preset value to obtain the updated identifier.

5. The method according to claim 3, characterized in that, The identifier includes multiple flag bits and parameters for each flag bit. When the second access time is less than the timestamp, determining whether the second access data is duplicate data based on the second access time, the identifier, and the timestamp includes: Calculate the difference between the second access time and the timestamp to obtain the second difference; Obtain the flag bit in the identifier whose number of bits is the second difference, and obtain the target flag bit; Determine whether the parameter of the target flag bit is a first preset value; If the parameter is the first preset value, it is determined that the second access time does not occur repeatedly. The identifier is changed according to the second access time and the timestamp to obtain the updated identifier, and the second access data is moved to the data pool corresponding to the data set. If the parameter is a second preset value, delete the second access data, wherein the second preset value is different from the first preset value.

6. The method according to claim 5, characterized in that, When the parameter is the first preset value, it is determined that the second access time has not occurred repeatedly. The identifier is changed according to the second access time and the timestamp to obtain the updated identifier, which includes: Determine the number of flag bits in the identifier whose parameters are the second preset values ​​to obtain multiple second digits; The timestamp is subtracted from each of the second digits in sequence to obtain multiple access times that have occurred, and the multiple access times that have occurred and the second access time are determined as multiple access times that have occurred; Determine the time difference between the timestamp and each access time that has occurred to obtain multiple fourth time differences; In the identifier, the parameter of the flag bit corresponding to each fourth time difference is set to the second preset value, and the parameter of the remaining flag bits in the identifier is set to the first preset value, so as to obtain the updated identifier.

7. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire access data of different access devices to the transaction system within a preset time period, and obtain multiple access data. Each access data includes at least the following information: device ID of the access device and access time. A grouping unit is used to group the multiple access data according to the device ID to obtain multiple data sets, wherein the access data in each data set have the same device ID; The deduplication unit is used to traverse the access data in each data set, generate an identifier during the traversal, update the identifier according to the access time of the currently traversed access data, and deduplicate the access data that appears repeatedly in the data set during each update of the identifier, until the traversal is completed, resulting in multiple deduplicated data sets. The identifier is used to record the access time of each traversed access data, and the repeated access data refers to access data with the same device ID and access time. The deduplication unit includes: a first acquisition subunit, used to acquire the maximum access time and minimum access time of accessed data in the data set; a calculation subunit, used to calculate the time difference between the maximum access time and the minimum access time to obtain a first difference value, and generate an initial identifier with the number of flag bits equal to the first difference value; and a setting subunit, used to set the parameter of each flag bit in the initial identifier to a first preset value to obtain the identifier.

8. A computer storage medium, characterized in that, The computer storage medium is used to store a program, wherein the program, when running, controls the device where the computer storage medium is located to perform the data processing method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the data processing method according to any one of claims 1 to 6.

Images