A data processing method, device, apparatus, and storage medium

Abstract

Embodiments of the present application disclose a data processing method, device and equipment and a storage medium. The method comprises: obtaining current interaction data of a current object, at least one first historical interaction data and a historical repeated occurrence number of the first historical interaction data; when it is detected that the current interaction data contains target parameter information corresponding to a preset statistical dimension parameter, determining second historical interaction data containing the target parameter information from the first historical interaction data; and updating historical statistical information corresponding to the target parameter information based on a preset statistical condition, according to the historical repeated occurrence number of the second historical interaction data and the current interaction data, to obtain current statistical information. The technical scheme of the embodiments of the present application can solve the problem of the limitation of the application scene in the prior art, realize real-time deduplication statistics, improve the application scope of the application scene, and meet the personalized needs of users.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to computer technology, and particularly relate to a data processing method, device, equipment and storage medium. BACKGROUND

[0002] With the rapid development of technologies such as artificial intelligence, Internet and Internet of Things, real-time computing is becoming more and more important. Real-time computing is a real-time analysis of massive streaming data at a second level, so as to support instant management and decision-making based on the analysis results. In real-time computing, it is usually necessary to perform deduplication statistics, such as statistics of the number of users who purchase an item, or statistics of the number of independent visitors (Unique Visitor, UV) of a website, etc.

[0003] At present, the existing deduplication statistics method is to use a Bloom filter to perform deduplication statistics. Specifically, a series of random mapping functions, such as Hash functions, are used to calculate the hash value corresponding to each element that needs to be deduplicated, and the hash value is stored in a bit binary array as an index, and 1 is stored at the corresponding position in the binary array. When a new element is received, whether the position corresponding to the hash value of the new element in the binary array is 1 can be detected, if yes, it means that the new element already exists and does not need to be counted, if not, the new element is counted.

[0004] However, in the process of implementing the present application, the inventors found that at least the following problems exist in the prior art:

[0005] The existing deduplication statistics method can only determine whether an element exists or not, so that the element is counted once when it appears for the first time, and is not counted again when it appears subsequently. However, this deduplication statistics method cannot be applied to a scenario where counting is performed only when an element appears twice or more times, such as a scenario of counting the number of valid UVs of a webpage, i.e., a scenario where a webpage is considered as a valid UV only when the number of clicks on the webpage is greater than or equal to two, thereby making the existing deduplication statistics method have limitations in application scenarios and unable to meet the personalized needs of users. SUMMARY

[0006] Embodiments of the present application provide a data processing method, device, equipment and storage medium to solve the problem of limitations in application scenarios in the prior art, realize real-time deduplication statistics, improve the application scope of scenarios, and meet the personalized needs of users.

[0007] In a first aspect, embodiments of the present application provide a data processing method, characterized in that it comprises:

[0008] obtaining current interaction data of a current object, at least one first historical interaction data, and a historical repeat occurrence number of the first historical interaction data which is statistically obtained in advance, wherein the current interaction data comprises current parameter information corresponding to at least one preset interaction parameter, and the first historical interaction data comprises historical parameter information corresponding to each preset interaction parameter;

[0009] when it is detected that the current interaction data comprises target parameter information corresponding to a preset statistical dimension parameter, determining second historical interaction data comprising the target parameter information from the first historical interaction data, wherein the preset statistical dimension parameter is a parameter selected in advance from the preset interaction parameter;

[0010] updating historical statistical information corresponding to the target parameter information based on a preset statistical condition, the current interaction data, and the historical repeat occurrence number of the second historical interaction data, to obtain current statistical information.

[0011] In a second aspect, an embodiment of the present application further provides a data processing apparatus, comprising:

[0012] an interaction data obtaining module, configured to obtain current interaction data of a current object, at least one first historical interaction data, and a historical repeat occurrence number of the first historical interaction data which is statistically obtained in advance, wherein the current interaction data comprises current parameter information corresponding to at least one preset interaction parameter, and the first historical interaction data comprises historical parameter information corresponding to each preset interaction parameter;

[0013] a second historical interaction data determining module, configured to, when it is detected that the current interaction data comprises target parameter information corresponding to a preset statistical dimension parameter, determine second historical interaction data comprising the target parameter information from the first historical interaction data, wherein the preset statistical dimension parameter is a parameter selected in advance from the preset interaction parameter;

[0014] a historical statistical information updating module, configured to update historical statistical information corresponding to the target parameter information based on a preset statistical condition, the current interaction data, and the historical repeat occurrence number of the second historical interaction data, to obtain current statistical information.

[0015] In a third aspect, an embodiment of the present application further provides a device, comprising:

[0016] one or more processors;

[0017] a memory, configured to store one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any of the embodiments of the present application.

[0019] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data processing method according to any of the embodiments of the present application.

[0020] The embodiments of the above application have the following advantages or beneficial effects:

[0021] By pre-statistically counting the historical repeated occurrence times of each first historical interaction data of each object, the historical repeated occurrence times corresponding to each element that needs to be deduplicated can be obtained. When the statistical information corresponding to the target parameter information of the preset statistical dimension parameter is statistically counted in real time, such as the number of objects corresponding to the target parameter information, when it is detected that the current interaction data of the current object contains the target parameter information, the second historical interaction data containing the target parameter information can be determined from the various first historical interaction data of the current object, and based on the historical repeated occurrence times of the second historical interaction data and the current interaction data, the historical statistical information corresponding to the target parameter information can be updated based on the preset statistical condition to obtain the current statistical information, thereby realizing real-time deduplication statistics. The preset statistical condition can be set according to user needs, such as setting the preset statistical condition to count only once when the repeated occurrence times of the target parameter information is greater than or equal to one, or setting the preset statistical condition to count only once when the repeated occurrence times of the target parameter information is greater than or equal to two, so that it can be applied to the scene of counting only once when it occurs once or more than once, and it can also be applied to the scene of counting only once when it occurs twice or more than twice, greatly improving the application scope of the application scene and meeting the personalized needs of users. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a data processing method provided by the first embodiment of the present application;

[0023] Figure 2 is a flowchart of a data processing method provided by the second embodiment of the present application;

[0024] Figure 3 is an example of a historical byte array related to the second embodiment of the present application;

[0025] Figure 4 is a flowchart of a data processing method provided by the third embodiment of the present application;

[0026] Figure 5 is a structural schematic diagram of a data processing device provided by the fourth embodiment of the present application;

[0027] Figure 6 is a structural schematic diagram of a data processing device provided by an embodiment five of the present application. DETAILED DESCRIPTION

[0028] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures.

[0029] Embodiment one

[0030] Figure 1 A flow chart of a data processing method provided by an embodiment one of the present application, the embodiment can be applicable to the case of real-time deduplication statistics of data, especially can be used in the real-time deduplication statistics scene in the e-commerce platform or website. The method can be executed by a data processing device, which can be realized by software and / or hardware, integrated in a device with data processing function, such as desktop computer, notebook computer, etc. As shown in the figure, the method specifically includes the following steps: Figure 1

[0031] S110, obtaining current interaction data of a current object, at least one first historical interaction data and a pre-statistical number of historical repeated occurrences of the first historical interaction data.

[0032] Among them, the object can refer to the object of real-time deduplication statistics. For example, the object can refer to but not limited to user or independent visitor UV. When the number of users who buy a certain item is counted, the object can refer to the user. When the number of independent visitors UV of the website is counted, the object can refer to UV. The current object can refer to the object that generates the current interaction data at the current time. The current interaction data can refer to the interaction data generated by the current object at the current time. The first historical interaction data can refer to the interaction data generated by the current object at the historical time. Exemplarily, when the number of independent visitors UV of the website is counted, the current interaction data can refer to the log data of the current UV accessing the website at the current time. When the number of users who buy the item is counted, the current interaction data can refer to the order information of the current user buying the item at the current time.

[0033] ​The current interaction data can include current parameter information corresponding to at least one preset interaction parameter, and the first historical interaction data can include historical parameter information corresponding to each preset interaction parameter. The preset interaction parameter can be pre-set to represent information of each dimension in the interaction data generated by the current object, so as to store each source data generated by the current object. The current parameter information can refer to a parameter value corresponding to the preset interaction parameter at the current time. The historical parameter information can refer to a parameter value corresponding to the preset interaction parameter at the historical time. The historical repeated occurrence number of the first historical interaction data can be used to represent the repeated occurrence number of each historical parameter information in the first historical interaction data. For example, when the historical repeated occurrence number of the first historical interaction data is two, it indicates that the historical repeated occurrence of each historical parameter information in the first historical interaction data is two.

[0034] Exemplarily, when performing real-time deduplication statistics in an e-commerce platform, the current interaction data of the current object can refer to current item acquisition task data of a current user. The current item acquisition task data can refer to current order data. The preset interaction parameters in the current item acquisition task data can include, but are not limited to, at least one of current item acquisition platform information, current item acquisition parent task identifier, current item acquisition sub-task identifier, current item identifier, current item ownership identifier, and current item acquisition time. The current item acquisition platform information can refer to order placement platform information, which can include, but is not limited to, APP mobile terminal identifier and PC computer terminal identifier. The current item acquisition parent task identifier can refer to the parent order identifier of an order. The current item acquisition sub-task identifier can refer to the sub-order identifier of an order. For example, if the current user places an order for two items of different logistics parties, a nested parent order will be generated, and a sub-order will be generated for each item in the parent order. If the current user places an order for an item or multiple items belonging to the same logistics party, the current item acquisition parent task identifier and the current item acquisition sub-task identifier are the same. The current item identifier can refer to the identifier of the item purchased by the current user. The current item ownership identifier can refer to the store identifier where the item purchased by the current user is located. The current item acquisition time can refer to the order placement time of the current user.

[0035] Similarly, the first historical interaction data of the current object can refer to historical item acquisition task data of the current user. The historical item acquisition task data can refer to historical order data. The historical item acquisition task data can include, but is not limited to, at least one of historical item acquisition platform information, historical item acquisition parent task identifier, historical item acquisition sub-task identifier, historical item identifier, historical item ownership identifier, and historical item acquisition time.

[0036] Specifically, a cache unit can be allocated to each object in advance for storing the first historical interaction data generated by each object and the historical repetition number of the first historical interaction data pre-statistically. When the current interaction data is acquired, the first historical interaction data of the current object and the historical repetition number of the first historical interaction data pre-statistically can be acquired from the cache unit corresponding to the current object according to the current object generating the current interaction data.

[0037] It should be noted that the current interaction data and the first historical interaction data can each be source data without data processing, so that real-time deduplication statistics can be performed on different preset statistical dimension parameters based on the same interaction data, thereby avoiding the case that a different Bloom filter needs to be started for deduplication statistics on different dimensions in the prior art, which causes excessive performance overhead, and greatly saves device resources.

[0038] S120, when it is detected that the current interaction data contains target parameter information corresponding to the preset statistical dimension parameter, determining second historical interaction data containing the target parameter information from the first historical interaction data.

[0039] The preset statistical dimension parameter can be a parameter preselected from the plurality of preset interaction parameters. The target parameter information can be a parameter value of the preset statistical dimension parameter referred to in the deduplication statistics process. For example, if the number of users purchasing item A is to be counted, the preset statistical dimension parameter can be set as the item identifier, and the corresponding target parameter information can be set as the specific identifier of item A. It should be noted that the preset statistical dimension parameter can be one selected parameter or a plurality of selected parameters. For example, if the number of users purchasing item A on March 23 is to be counted, the preset statistical dimension parameter can be set as the item identifier and the item acquisition time, and the corresponding target parameter information can be set as the specific identifier of item A and March 23. The second historical interaction data can be the first historical interaction data containing the target parameter information selected from the first historical interaction data. The number of second historical interaction data can be one or more, and the specific number is determined by the selection result.

[0040] Specifically, the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data can be matched with the target parameter information. If the matching is successful, it is determined that the target parameter information is included in the current interaction data. When it is detected that the target parameter information is included in the current interaction data, it indicates that the current interaction data can affect the historical statistical information corresponding to the target parameter information, that is, the historical statistical information corresponding to the target parameter information needs to be updated. At this time, the historical parameter information corresponding to the preset statistical dimension parameter in each first historical interaction data of the current object can be matched with the target parameter information. If the matching is successful, it is determined that the target parameter information is included in the first historical interaction data, and thus each matched first historical interaction data, that is, each second historical interaction data, can be obtained.

[0041] In S130, based on a preset statistical condition, the historical statistical information corresponding to the target parameter information is updated according to the historical repeated occurrence number of the second historical interaction data and the current interaction data, and current statistical information is obtained.

[0042] The preset statistical condition can be a statistical condition set in advance based on user demand for deduplication statistics. For example, the preset statistical condition can be set to count only once when the repeated occurrence number of the target parameter information is greater than or equal to one, or can be set to count only once when the repeated occurrence number of the target parameter information is greater than or equal to two, so as to be applied to a scenario of counting only once when one or more times occur, or to a scenario of counting only once when two or more times occur, greatly improving the application scope of the scenario and meeting the personalized needs of users.

[0043] The historical statistical information corresponding to the target parameter information can refer to an object statistical result corresponding to the target parameter information obtained before the current time. For example, when the number of users who purchase an item A is counted, the target parameter information is set to the specific identifier of the item A, and the historical statistical information corresponding to the target parameter information can refer to the number of users who purchase the item A counted before the current time.

[0044] Specifically, the embodiment can determine that the number of repeated occurrences of the target parameter information in the current interaction data is a preset number of times, such as 1 time, when the target parameter information is included in the current interaction data. The historical number of repeated occurrences of the second historical interaction data is the number of repeated occurrences of the target parameter information in the second historical interaction data. According to the number of repeated occurrences of the target parameter information in each second historical interaction data and the number of repeated occurrences of the target parameter information in the current interaction data, the historical statistical information corresponding to the target parameter information can be updated based on a preset statistical condition to obtain the current statistical information, thereby realizing real-time deduplication statistics. For example, when the preset statistical condition is set to count only once when the number of repeated occurrences of the target parameter information is greater than or equal to two times, if the total number of historical repeated occurrences of the target parameter information in each second historical interaction data is 1 time, the number of repeated occurrences of the target parameter information in the current interaction data is 1 time, and the historical statistical information corresponding to the target parameter information is that the number of objects is 2, then since the total number of historical repeated occurrences of 1 time does not meet the preset statistical condition, the current object is not counted in the historical statistical information, that is, the 2 objects counted do not include the current object. However, since the sum of the total number of historical repeated occurrences and the number of repeated occurrences of the target parameter information in the current interaction data is 2 times, which meets the preset statistical condition, the historical statistical information needs to be updated, that is, the number of objects is updated to 3. If the total number of historical repeated occurrences of each second historical interaction data is 2 times, it indicates that the total number of historical repeated occurrences of 2 times meets the preset statistical condition, so that the current object has been counted in the historical statistical information, that is, the 2 objects counted include the current object. At this time, even if the sum of the total number of historical repeated occurrences and the number of repeated occurrences of the target parameter information in the current interaction data is 3 times, which meets the preset statistical condition, there is no need to update the historical statistical information, and the historical statistical information can be directly determined as the current statistical information, thereby realizing real-time deduplication statistics and being suitable for various deduplication statistical application scenarios, and solving the problem of scene adaptation limitation of the existing deduplication statistical manner.

[0045] It should be noted that when each first historical interaction data does not include the target parameter information, that is, there is no second historical interaction data, the total number of historical repeated occurrences can be determined as 0.

[0046] It should be noted that when it is detected that the current interaction data does not include the target parameter information corresponding to the preset statistical dimension parameter, it indicates that the historical statistical information corresponding to the target parameter information does not need to be updated, and at this time the historical statistical information can be directly determined as the current statistical information.

[0047] The technical scheme of the embodiment can obtain the historical repeated occurrence number of each element that needs to be deduplicated. When real-time statistics of the statistical information corresponding to the target parameter information of the preset statistical dimension parameter is performed, for example, when the number of objects corresponding to the target parameter information is counted, when it is detected that the current interaction data of the current object contains the target parameter information, the second historical interaction data containing the target parameter information can be determined from the first historical interaction data of the current object, and the historical repeated occurrence number of the second historical interaction data and the current interaction data can be used to update the historical statistical information corresponding to the target parameter information based on the preset statistical condition, to obtain the current statistical information, thereby realizing real-time deduplication statistics. The preset statistical condition can be set based on user demand, for example, the preset statistical condition can be set to count only once when the repeated occurrence number of the target parameter information is greater than or equal to one, or the preset statistical condition can be set to count only once when the repeated occurrence number of the target parameter information is greater than or equal to two, thereby being applicable to a scenario in which counting is performed only once when the occurrence is one or more times, or being applicable to a scenario in which counting is performed only once when the occurrence is two or more times, greatly improving the application scope of the application scenario and meeting the personalized needs of users.

[0048] On the basis of the above technical scheme, after S130, the method can further include: matching each current parameter information in the current interaction data with corresponding historical parameter information in each first historical interaction data; if there is third historical interaction data that matches each current parameter information in the current interaction data successfully, updating the historical repeated occurrence number of the third historical interaction data; and if there is no third historical interaction data that matches each current parameter information in the current interaction data successfully, storing the current interaction data in a current cache unit corresponding to the current object, wherein the current cache unit can be used to store the first historical interaction data of the current object.

[0049] Specifically, after obtaining the current statistical information, it is necessary to store the current interaction data into the current cache unit corresponding to the current object, so as to store each interaction data generated by the current object in the current cache unit in real time. In this embodiment, each current parameter information corresponding to each preset interaction parameter in the current interaction data can be matched with the historical parameter information corresponding to the same preset interaction parameter in each first historical interaction data. If each current parameter information and the corresponding historical parameter information in a certain first historical interaction data are matched successfully, it indicates that the current interaction data and the first historical interaction data are the same data. At this time, it can be determined that the first historical interaction data is the third historical interaction data, and the current interaction data is stored by updating the historical repeated occurrence number of the third historical interaction data, such as adding 1 to the historical repeated occurrence number of the third historical interaction data. If the current interaction data does not match each first historical interaction data, that is, there is no third historical interaction data, the current interaction data can be stored as a new first historical interaction data in the current cache unit, and the historical repeated occurrence number of the current interaction data is stored, such as 1, so as to ensure that each first historical interaction data and the historical repeated occurrence number generated by the current object can be stored in the current cache unit in real time, and the subsequent real-time deduplication statistics can be accurately performed.

[0050] Exemplarily, the current interaction data can further include an interaction mode identifier of the current object. The interaction mode identifier can include an addition identifier and a cancellation identifier. The addition identifier can be used to represent that the current interaction data is data to be added to the current cache unit. The cancellation identifier can be used to represent that the current interaction data is data to be cancelled, that is, data to be deleted in the current cache unit. For example, if a user places an order to purchase item A, the interaction mode identifier in the current interaction data of the user is the addition identifier. If the user cancels the order after placing an order to purchase item A, the interaction mode identifier in the current interaction data of the user is the cancellation identifier.

[0051] Exemplarily, updating the historical repeated occurrence number of the third historical interaction data can include: if the interaction mode identifier in the current interaction data is the addition identifier, updating the historical repeated occurrence number of the third historical interaction data by adding a preset value; if the interaction mode identifier in the current interaction data is the cancellation identifier, updating the historical repeated occurrence number of the third historical interaction data by subtracting the preset value. The preset value can be the processing number corresponding to the addition identifier or the cancellation identifier. For example, the preset value can be set to 1.

[0052] Specifically, the historical repeated occurrence number of the third historical interaction data can be updated based on the interaction manner identifier in the current interaction data. When the interaction manner identifier is an addition identifier, the historical repeated occurrence number of the third historical interaction data can be added by a preset value, such as adding 1, to add the current interaction data. When the interaction manner identifier is a cancellation identifier, the historical repeated occurrence number of the third historical interaction data can be subtracted by a preset value, such as subtracting 1, to delete the current interaction data, thereby supporting the data deletion operation. In the prior art, when an element needs to be deleted in the de-duplication statistics using a set of binary bits BitMap, the binary bit corresponding to the element is updated from 1 to 0 to delete the element. However, this de-duplication statistics method can only reflect whether the current operation is an addition operation or a deletion operation, and cannot real-time statistics the current number after addition and deletion. For example, when the same element is added twice in succession and then deleted once, the binary bit corresponding to the element in the BitMap is updated to 0, which indicates that the element does not exist at present, and cannot reflect that the element is equivalent to only one addition. However, the embodiment can update the historical repeated occurrence number in real time based on the interaction manner identifier, thereby accurately storing the repeated occurrence number of the interaction data, such as updating the corresponding historical repeated occurrence number to 1 when the same element is added twice in succession and then deleted once, thereby being applicable to the application scenario where the data deletion operation exists, and further improving the application scope of the application scenario.

[0053] It should be noted that the embodiment can use the positive and negative signs and the preset value to represent the addition identifier and the cancellation identifier. For example, if the preset value is set to 1, the addition identifier can be +1 and the cancellation identifier can be -1, thereby directly updating the historical repeated occurrence number of the third historical interaction data in the manner of adding the interaction manner identifier, to further simplify the updating operation and improve the processing efficiency.

[0054] Exemplarily, due to network delay and other reasons, the current interaction data for cancellation can be obtained first, and the current interaction data for addition can be obtained later, thereby for the case where the third historical interaction data does not exist, the historical repeated occurrence number corresponding to the current interaction data can be determined based on the interaction manner identifier when the current interaction data is stored to the current cache unit. For example, when the interaction manner identifier is an addition identifier, a positive preset value such as +1 can be used as the historical repeated occurrence number corresponding to the current interaction data. When the interaction manner identifier is a cancellation identifier, a negative preset value such as -1 can be used as the historical repeated occurrence number corresponding to the current interaction data, thereby ensuring the accuracy of data storage.

[0055] On the basis of the above technical solutions, S130 can include: determining a total number of historical repeated occurrences according to the number of historical repeated occurrences of each second historical interaction data; determining a total number of current repeated occurrences according to the current interaction data and the total number of historical repeated occurrences; determining whether the total number of historical repeated occurrences and the total number of current repeated occurrences meet a preset statistical condition, and updating the historical statistical information corresponding to the target parameter information according to the determination result to obtain the current statistical information.

[0056] Specifically, the number of historical repeated occurrences of each second historical interaction data can be added, and the addition result is determined as the total number of historical repeated occurrences of the target parameter information. If the current interaction data is only used to represent the interaction data to be added, the result of adding 1 to the total number of historical repeated occurrences can be directly determined as the total number of current repeated occurrences of the target parameter information. Whether the total number of historical repeated occurrences meets the preset statistical condition can determine whether the current object is counted in the historical statistical information. Whether the total number of current repeated occurrences meets the preset statistical condition can determine whether the current object can be counted at the current time. According to the above two determination results, the historical statistical information corresponding to the target parameter information can be updated to obtain the current statistical information, thereby realizing real-time deduplication statistics.

[0057] Exemplarily, determining whether the total number of historical repeated occurrences and the total number of current repeated occurrences meet the preset statistical condition, and updating the historical statistical information corresponding to the target parameter information according to the determination result to obtain the current statistical information can include:

[0058] When the total number of historical repeated occurrences meets the preset statistical condition and the total number of current repeated occurrences does not meet the preset statistical condition, the result of subtracting a preset number from the historical statistical information corresponding to the target parameter information is determined as the current statistical information; when the total number of historical repeated occurrences does not meet the preset statistical condition and the total number of current repeated occurrences meets the preset statistical condition, the result of adding a preset number to the historical statistical information corresponding to the target parameter information is determined as the current statistical information; when the total number of historical repeated occurrences and the total number of current repeated occurrences both meet the preset statistical condition, or the total number of historical repeated occurrences and the total number of current repeated occurrences both do not meet the preset statistical condition, the historical statistical information corresponding to the target parameter information is determined as the current statistical information.

[0059] The preset number can be a statistical value corresponding to the current object. For example, the preset value can be set to 1. Specifically, when the total number of historical repeated occurrences meets the preset statistical condition and the total number of current repeated occurrences does not meet the preset statistical condition, it indicates that the current object has been counted in the historical statistical information, and the current time cannot continue to count the current object, so the result of subtracting the preset number from the historical statistical information corresponding to the target parameter information is determined as the current statistical information, such as the result of subtracting 1 from the number of objects counted in the history is determined as the number of objects counted in the current time. When the total number of historical repeated occurrences does not meet the preset statistical condition and the total number of current repeated occurrences meets the preset statistical condition, it indicates that the current object has not been counted in the historical statistical information, and the current time can count the current object, so the result of adding the preset number to the historical statistical information corresponding to the target parameter information is determined as the current statistical information, such as the result of adding 1 to the number of objects counted in the history is determined as the number of objects counted in the current time. When the total number of historical repeated occurrences and the total number of current repeated occurrences both meet the preset statistical condition, it indicates that the current object has been counted in the historical statistical information, and the current time can continue to count the current object, so the historical statistical information does not need to be modified, and can be directly determined as the current statistical information. When the total number of historical repeated occurrences and the total number of current repeated occurrences both do not meet the preset statistical condition, it indicates that the current object has not been counted in the historical statistical information, and the current time also cannot count the current object, so the historical statistical information does not need to be modified, and can be directly determined as the current statistical information.

[0060] Exemplarily, when the current interaction data further includes an interaction mode identifier of the current object, determining the total number of current repeated occurrences according to the current interaction data and the total number of historical repeated occurrences can include: if the interaction mode identifier in the current interaction data is an addition identifier, the result of adding a preset value to the total number of historical repeated occurrences is determined as the total number of current repeated occurrences; if the interaction mode identifier in the current interaction data is a cancellation identifier, the result of subtracting the preset value from the total number of historical repeated occurrences is determined as the total number of current repeated occurrences.

[0061] Specifically, the total number of current repeated occurrences can be determined according to the interaction mode identifier in the current interaction data and the total number of historical repeated occurrences. When the interaction mode identifier is an addition identifier, it indicates that the current interaction data is addition data, at this time the result of adding a preset value, such as 1, to the total number of historical repeated occurrences is determined as the total number of current repeated occurrences. When the interaction mode identifier is a cancellation identifier, it indicates that the current interaction data is deletion data, at this time the result of subtracting a preset value, such as 1, from the total number of historical repeated occurrences is determined as the total number of current repeated occurrences, so that the data deletion operation can be supported, which is applicable to application scenarios with data deletion operation, and further improves the application scope of the application scenario.

[0062] It should be noted that in the embodiment, when the addition identifier and the cancellation identifier in the interaction mode identifier are represented by using the sign and the preset value, the result of accumulating the total number of repeated occurrences of the history can be directly determined as the current total number of repeated occurrences, so that the de-duplication statistical operation can be further simplified, and the statistical efficiency can be improved.

[0063] Embodiment Two

[0064] Figure 2 A flowchart of a data processing method provided by Embodiment Two of the present application is provided, and the storage manner of the current interaction data and each first historical interaction data is described in detail in the embodiment. The explanations of the same or corresponding terms in the above embodiments are not repeated here.

[0065] Referring to Figure 2 The data processing method provided by the embodiment specifically includes the following steps:

[0066] S210, converting the historical parameter information in the first historical interaction data of the current object into byte type historical parameter information, and generating a historical byte array corresponding to the first historical interaction data according to a preset arrangement order, the historical parameter information of each byte type, and the historical repeated occurrence number of the first historical interaction data which is statistically obtained in advance.

[0067] The byte type (Byte) can refer to a data type containing 8 binary bits (bit). The preset arrangement order can be a preset arrangement order of each preset interaction parameter.

[0068] Specifically, the data type (such as int, short, long, etc.) of the historical parameter information corresponding to each preset interaction parameter can be converted into the byte type, so as to obtain the corresponding byte data, so that the parameter information can be uniformly stored in the form of the byte array. Based on the maximum length of the parameter information corresponding to each preset interaction parameter, a corresponding number of bytes is allocated to each preset interaction parameter, and the corresponding parameter information is stored in the allocated bytes. The embodiment can determine the position of the historical parameter information of each byte type in the historical byte array based on the preset arrangement order, and store the historical repeated occurrence number of the first historical interaction data which is statistically obtained in advance at the preset position in the historical byte array. For example, the historical repeated occurrence number of the first historical interaction data which is statistically obtained in advance can be stored on the first byte or the last byte of the historical byte array, so as to obtain the historical byte array corresponding to each first historical interaction data.

[0069] Exemplarily, Figure 3 An example of a historical byte array is given. As Figure 3As shown, when performing real-time deduplication statistics in the e-commerce platform, the first historical interaction data of the current object can refer to the historical item acquisition task data of the current user. Each historical item acquisition task data can include: historical item acquisition platform information, historical item acquisition parent task identifier, historical item acquisition sub-task identifier, historical item item identifier, historical item ownership identifier, historical item acquisition date, hour number and minute number. Among them, the historical item acquisition date can refer to the date of purchasing the item, accurate to the day, such as March 24, 2020. The hour number can refer to the number of hours from the current time to 0 o'clock of the historical item acquisition date. The minute number can refer to the number of minutes from the current time to 0 o'clock 0 minutes of the historical item acquisition date. For example, if the current time is 1:05 on March 24, 2020, the hour number is 1 hour, and the minute number is 65 minutes. For example, Figure 3 As shown, the historical parameter information in each historical item acquisition task data and the historical repetition number of the historical item acquisition task data can be stored in the corresponding historical byte array based on the arrangement order in Figure 3 Figure 3 The data structure of each first historical interaction data and the historical repetition number stored in the corresponding cache unit of a user is given, and the user identifier can be stored as the primary key in the external data source.

[0070] Exemplarily, in S210, "converting the historical parameter information in each first historical interaction data of the current object into byte type historical parameter information" can include: if the target parameter information corresponding to the preset statistical dimension parameter contains at least two statistical parameter information, then based on the preset encoding manner, the target parameter information and each statistical parameter information are binary encoded to obtain the binary encoding information corresponding to the target parameter information and the binary encoding information corresponding to each statistical parameter information; according to the binary encoding information corresponding to each statistical parameter information, the historical binary encoding information corresponding to the historical parameter information of the preset statistical dimension parameter in each first historical interaction data of the current object is determined as the byte type historical parameter information.

[0071] ​The statistical parameter information can be used to represent a dimension value of a preset statistical dimension parameter. The target parameter information corresponding to the preset statistical dimension parameter can include one statistical parameter information, or two or more statistical parameter information, and the specific number can be pre-set based on business requirements. For example, if the number of users who purchase an item in an APP platform is counted, the item acquisition platform information is used as the preset statistical dimension parameter, the target parameter information corresponding to the preset statistical dimension parameter is APP platform, and the target parameter information includes only one statistical parameter information, i.e., APP platform. For another example, if the number of users who purchase an item in a total platform (i.e., APP platform and / or PC platform) is counted, the item acquisition platform information is used as the preset statistical dimension parameter, the target parameter information corresponding to the preset statistical dimension parameter is ALL platform, and the target parameter information includes two statistical parameter information, i.e., APP platform and PC platform.

[0072] The preset encoding mode can be pre-set, and when the target parameter information includes at least two statistical parameter information, the binary encoding mode of the target parameter information and each statistical parameter information having a “containment and being contained” relationship is based on binary bit operation, so as to ensure that each statistical parameter information can be contained in the target parameter information. For example, Table 1 gives an example of binary encoding information corresponding to platform information. As shown in Table 1, the binary encoding information 0b0001 corresponding to the target parameter information ALL and the binary encoding information 0b0011 corresponding to APP are operated to obtain the result of the binary encoding information corresponding to ALL, i.e., APP&ALL==ALL; and the binary encoding information 0b0001 corresponding to the target parameter information ALL and the binary encoding information 0b0101 corresponding to PC are operated to obtain the result of the binary encoding information corresponding to ALL, i.e., PC&ALL==ALL, so that based on binary bit operation, it can be ensured that APP and PC are contained in ALL, so as to ensure the accuracy of the deduplication statistics based on the target parameter information ALL in the subsequent deduplication statistics.

[0073] Table 1: Binary encoding value corresponding to platform information

[0074] Platform information ALL APP PC Binary coded information 0b0001 0b0011 0b0101

[0075] Specifically, the historical parameter information corresponding to each preset statistical dimension parameter in each first historical interaction data can be matched with the statistical parameter information based on the determined binary coding information corresponding to each statistical parameter information, the binary coding information corresponding to the statistical parameter information that passes the matching can be determined as the historical binary coding information corresponding to the historical parameter information, and used as the historical parameter information of the byte type. For example, if the preset statistical dimension parameter is platform information, and the historical platform information in a certain first historical interaction data is APP, the historical binary coding information corresponding to the historical platform information can be determined as 0b0011 based on Table 1, so that each historical parameter information corresponding to the target parameter information having a “containment and being contained” relationship can be binary coded, so that subsequent information matching can be performed in a manner of bit operation, and the accuracy of deduplication statistics can be improved.

[0076] S220, converting each current parameter information in the current interaction data of the current object into current parameter information of the byte type, and generating a current byte array corresponding to the current interaction data according to a preset arrangement order and each current parameter information of the byte type.

[0077] Specifically, the data type (such as int, short, long, etc.) of each current parameter information corresponding to each preset interaction parameter can be converted into the byte type, so as to obtain corresponding byte data, so that each parameter information can be uniformly stored in the form of a byte array. Based on the preset arrangement order in the historical byte array, each current parameter information of the byte type is stored in a corresponding position in the current byte array, so that the byte storage positions in the current byte array and the historical byte array are the same, facilitating the acquisition of parameter information corresponding to the same preset interaction parameter and information matching.

[0078] For example, when the current interaction data further includes the interaction mode identifier of the current object, the interaction mode identifier of the current object can be stored at a preset position of the current byte array, such as the first byte or the last byte of the current byte array.

[0079] For example, in S220, “converting each current parameter information in the current interaction data of the current object into current parameter information of the byte type” can include: when the target parameter information corresponding to the preset statistical dimension parameter contains at least two statistical parameter information, determining the current binary coding information corresponding to the current parameter information of the preset statistical dimension parameter in the current interaction data of the current object as the current parameter information of the byte type according to the binary coding information corresponding to each statistical parameter information.

[0080] Specifically, if the preset statistical dimension parameter is platform information, and the current platform information in the current interaction data is PC, it can be determined according to Table 1 above that the current binary coded information corresponding to the current platform information is 0b0101, so that the current parameter information corresponding to the target parameter information having a “containment and being contained” relationship can be binary coded, so that subsequent information matching can be performed in a manner of bit operation, thereby improving the accuracy of deduplication statistics.

[0081] S230, obtaining a current byte array corresponding to the current interaction data of the current object and a historical byte array corresponding to each first historical interaction data.

[0082] S240, when it is detected that the current interaction data contains the target parameter information corresponding to the preset statistical dimension parameter, determining, from the first historical interaction data, second historical interaction data containing the target parameter information.

[0083] Specifically, if the target parameter information corresponding to the preset statistical dimension parameter is non-binary coded information, it can be directly determined by character matching whether the current interaction data contains the target parameter information, for example, whether the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data is the same as the target parameter information. If yes, it is determined that the current interaction data contains the target parameter information. When it is detected that the current interaction data contains the target parameter information, the second historical interaction data containing the target parameter information can also be determined from the first historical interaction data based on the above character matching manner. For example, “determining, from the first historical interaction data, second historical interaction data containing the target parameter information” can include: detecting whether the historical parameter information corresponding to the preset statistical dimension parameter in the first historical interaction data is the same as the target parameter information, and if yes, determining that the first historical interaction data contains the target parameter information, that is, determining the first historical interaction data as the second historical interaction data.

[0084] Exemplarily, if the target parameter information corresponding to the preset statistical dimension parameter is binary coded information, whether the target parameter information is included in the current interaction data can be determined through binary bit operation, for example, performing bit operation on the current binary coded information corresponding to the preset statistical dimension parameter in the current interaction data and the target parameter information, and detecting whether the bit operation result is same as the target parameter information, if yes, it is determined that the target parameter information is included in the current interaction data. When it is detected that the target parameter information is included in the current interaction data, the second historical interaction data including the target parameter information can also be determined from the first historical interaction data based on the above-mentioned binary bit operation. For example, "determining the second historical interaction data including the target parameter information from the first historical interaction data" can include: performing bit operation on the historical binary coded information corresponding to the preset statistical dimension parameter in the first historical interaction data and the target parameter information, and detecting whether the bit operation result is same as the target parameter information, if yes, it is determined that the first historical interaction data includes the target parameter information, that is, the first historical interaction data is determined as the second historical interaction data.

[0085] It should be noted that, through the binary bit operation, whether the target parameter information is included in the current interaction data and the first historical interaction data can be more accurately and conveniently determined, and the deduplication statistical efficiency is further improved.

[0086] S250, updating the historical statistical information corresponding to the target parameter information based on the preset statistical condition, according to the historical repeated occurrence number of the second historical interaction data and the current interaction data, to obtain current statistical information.

[0087] The technical scheme of the embodiment converts the historical parameter information in each first historical interaction data into historical parameter information of byte type, generates a historical byte array corresponding to each first historical interaction data, converts each current parameter information in the current interaction data into current parameter information of byte type, and generates a current byte array corresponding to the current interaction data, so that byte data in the byte array can be used to more conveniently perform information matching, and binary bit operation can also be used to perform information matching, and the deduplication statistical efficiency is further improved.

[0088] On the basis of the above technical scheme, the method further includes: generating a historical hash value corresponding to the historical interaction data according to each historical parameter information in the first historical interaction data, and adding the historical hash value to the historical byte array; generating a current hash value corresponding to the current interaction data according to each current parameter information in the current interaction data, and adding the current hash value to the current byte array.

[0089] Specifically, based on the random mapping function, a hash calculation can be performed on each historical parameter information in the first historical interaction data to determine the historical hash value, and a hash calculation can be performed on each current parameter information in the current interaction data to determine the current hash value. The historical hash value can be stored at a preset position in the historical byte array, and the current hash value can be stored at a preset position in the current byte array. Figure 3 As shown, the historical hash value can be stored at the first byte in the historical byte array. For example, after S250, by comparing whether the current hash value is the same as each historical hash value, it can be determined from each first historical interaction data whether there is a third historical interaction data that successfully matches each current parameter information in the current interaction data. If the historical hash value corresponding to a certain first historical interaction data is the same as the current hash value, it indicates that the first historical interaction data is likely to be the third historical interaction data. At this time, it can continue to detect whether each historical parameter information in the first historical interaction data matches the corresponding current historical parameter. If so, it is determined that the first historical interaction data is determined to be the third historical interaction data. If the historical hash value corresponding to each first historical interaction data is different from the current hash value, it can be directly determined that there is no third historical interaction data, so that based on the hash value, it can be quickly determined whether there is a third historical interaction data that matches the current interaction data, thereby improving the storage efficiency of the current interaction data.

[0090] Example 3

[0091] Figure 4 This is a flowchart of a data processing method provided in Example 3 of the present invention. This example, based on the previous examples, optimizes the method for "when detecting that the current interaction data contains target parameter information corresponding to a preset statistical dimension parameter, determining second historical interaction data containing the target parameter information from the first historical interaction data." Explanations of terms that are identical or corresponding to those in the previous examples are omitted here.

[0092] See also Figure 4 The data processing method provided in this embodiment specifically includes the following steps:

[0093] S310: Obtain current interaction data of a current object, at least one first historical interaction data, and a pre-counted number of historical recurrences of the first historical interaction data.

[0094] S320: Determine, from each candidate parameter information corresponding to the preset statistical dimension parameter, target parameter information that matches the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data.

[0095] The candidate parameter information can refer to a parameter value corresponding to the preset statistical dimension parameter and available for statistics. For example, if the preset statistical dimension parameter is an item identifier, the specific identifier of each item in the item database can be used as the candidate parameter information, and / or the specific identifiers of two or more items can be used as the candidate parameter information (in this case, the binary code information corresponding to the specific identifiers of the two or more items can be determined as the candidate parameter information). For another example, if the preset statistical dimension parameter is an item acquisition platform information, the binary code information corresponding to the APP platform, the binary code information corresponding to the PC platform, and the binary code information corresponding to the total ALL platform can all be used as the candidate parameter information, so that the de-duplication statistics can be performed on each candidate parameter information.

[0096] Specifically, each candidate parameter information corresponding to the preset statistical dimension parameter can be matched with the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data, to determine a target parameter information matched with the current parameter information. For example, if the candidate parameter information is non-binary code information, whether the candidate parameter information is the same as the current parameter information can be detected by directly using a character matching method, and if yes, the candidate parameter information is determined as the target parameter information. If the candidate parameter information is binary code information, the candidate parameter information can be operated by bit operation with the current parameter information, and whether the operation result is the same as the candidate parameter information can be detected, and if yes, the candidate parameter information is the target parameter information.

[0097] It should be noted that the target parameter information matched with the current parameter information can be one or more. For each target parameter information, the historical statistical information corresponding to each target parameter information can be updated by the following steps S330-S350, so that the de-duplication statistics of each candidate parameter information of the preset statistical dimension parameter can be performed simultaneously based on the same interaction data, and the performance overhead caused by the need to start a different Bloom filter for the de-duplication statistics of different candidate parameter information in the prior art is avoided, the device resources are greatly saved, and the de-duplication statistics based on the dimension is realized, rather than the de-duplication statistics based on a specific dimension value.

[0098] Exemplarily, if the preset statistical dimension parameter is the item obtaining platform information, corresponding each candidate parameter information is: the binary coding information 0b0011 corresponding to the APP platform, the binary coding information 0b0101 corresponding to the PC platform and the binary coding information 0b0001 corresponding to the total ALL platform. Table 2 gives an example of the current interaction data of the current object. As shown in Table 2, the current platform information in the current interaction data is 0b0011 (i.e. the APP platform), so that the bit operation can be performed based on the binary coding information corresponding to each candidate parameter information, i.e. 0b0011&0b0011==0b0011, indicating matching, so that the candidate parameter information 0b0011 can be determined as the target parameter information; 0b0101&0b0011==0b0001, indicating non-matching, so that the candidate parameter information 0b0101 can be determined as the target parameter information; 0b0001&0b0011==0b0001, indicating matching, so that the candidate parameter information 0b0001 can be determined as the target parameter information, so that two target parameter information 0b0011 (i.e. the APP platform) and 0b0001 (i.e. the ALL platform) can be determined, and the deduplication statistics can be performed for each target parameter information through the following steps S330-S350.

[0099] Table 2 current interaction data of the current object

[0100]

[0101] It should be noted that the embodiment can perform the deduplication statistics based on one preset statistical dimension parameter, or perform the deduplication statistics based on multiple preset statistical dimension parameters. For example, when the preset statistical dimension parameters include the item obtaining platform information and the item identifier, the target parameter information matched with the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data can be determined from each candidate parameter information corresponding to each preset statistical dimension parameter, such as the determined target parameter information being the platform information 0b0011 (i.e. the APP platform) and the item identifier 21, so that the number of users purchasing the item identifier 21 in the APP platform can be counted, the multi-dimensional deduplication statistics is realized, and the situation of excessive performance consumption caused by the separate statistics of each dimension is avoided, further saving the device resources.

[0102] S330, determining the second historical interaction data containing the target parameter information from the first historical interaction data.

[0103] Exemplarily, if the preset statistical dimension parameter is the item identifier, the target parameter information determined based on Table 2 can be: the item identifier 21. Since the item identifier is non-binary coded information, the historical item identifier in each first historical interaction data and the target parameter information can be directly character matched. Table 3 gives an example of each first historical interaction data of a current object. In Table 3, two second historical interaction data can be determined from the three first historical interaction data, which are the first historical interaction data in the first row and the first historical interaction data in the third row.

[0104] Table 3 Each first historical interaction data of a current object

[0105]

[0106] S340, determining the historical statistical information corresponding to the target parameter information according to a mapping relationship between each candidate parameter information corresponding to the preset statistical dimension parameter and historical statistical information.

[0107] The historical statistical information can refer to the object statistical result corresponding to each candidate parameter information obtained before the current time. Specifically, the embodiment can pre-store the mapping relationship between each candidate parameter information corresponding to the preset statistical dimension parameter and the historical statistical information, so as to determine the historical statistical information corresponding to the target parameter information based on the mapping relationship, that is, to obtain the historical statistical information currently required to be updated.

[0108] Exemplarily, when the preset statistical dimension parameter is the item identifier and the target parameter information is the item identifier 21, the historical statistical information corresponding to the target parameter information is: the historical user quantity of the purchased item identifier 21 statistically obtained before the current time.

[0109] S350, updating the historical statistical information corresponding to the target parameter information based on the preset statistical condition, according to the historical repeated appearance number of the second historical interaction data and the current interaction data, to obtain the current statistical information.

[0110] Exemplarily, when the preset statistical dimension parameter is the item identifier, based on the examples of Tables 2 and 3 above, the historical repeat occurrence times of the two second historical interaction data in Table 3 are 1 and 1 respectively, and the interaction mode identifier in the current interaction data in Table 2 is -1, so the total historical repeat occurrence times are determined to be 2, and the total current repeat occurrence times are determined to be 1. If the preset statistical condition is that the repeat occurrence times of the target parameter information are greater than or equal to one time, and only one time is counted, then based on the total historical repeat occurrence times 2, it is determined that the historical user quantity has counted the current object, and based on the total current repeat occurrence times 1, it is determined that the current object can be counted at the current time, so there is no need to modify the counted historical user quantity, that is, the historical user quantity is directly used as the current user quantity counted at the current time. If the preset statistical condition is that the repeat occurrence times of the target parameter information are greater than or equal to two times, and only one time is counted, then based on the total historical repeat occurrence times 2, it is determined that the historical user quantity has counted the current object, and based on the total current repeat occurrence times 1, it is determined that the current object cannot be counted at the current time, so the counted historical user quantity needs to be reduced by 1, and the historical user quantity after the reduction is determined as the current user quantity counted at the current time, so the user quantity of the item identifier 21 is counted in real time based on the current interaction data.

[0111] The technical scheme of the embodiment determines the target parameter information matched with the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data from each candidate parameter information corresponding to the preset statistical dimension parameter, determines the historical statistical information corresponding to the target parameter information according to the mapping relationship between each candidate parameter information corresponding to the preset statistical dimension parameter and the historical statistical information, and counts the historical statistical information in real time, so that each candidate parameter information of the preset statistical dimension parameter can be counted in real time based on the same interaction data, the performance overhead caused by the need to start a different Bloom filter for each candidate parameter information in the prior art is avoided, the device resources are greatly saved, and the dimension-based de-duplication counting is realized instead of de-duplication counting based on a specific dimension value.

[0112] The following is an embodiment of a data processing apparatus provided by the embodiment of the application. The data processing apparatus and the data processing method of each of the above embodiments belong to the same inventive concept. Details not described in the embodiment of the data processing apparatus can be referred to the embodiment of the data processing method.

[0113] Embodiment Four

[0114] Figure 5A structural schematic diagram of a data processing device provided for the fourth embodiment of the present application, the embodiment can be applied to the case of real-time deduplication statistics of data, and the device specifically comprises: an interactive data acquisition module 410, a second historical interactive data determination module 420, and a historical statistical information updating module 430.

[0115] The interactive data acquisition module 410 is configured to acquire current interactive data of a current object, at least one first historical interactive data, and a historical repeated occurrence number of the first historical interactive data that is pre-statistically acquired. The current interactive data comprises current parameter information corresponding to at least one preset interactive parameter, and the first historical interactive data comprises historical parameter information corresponding to each preset interactive parameter. The second historical interactive data determination module 420 is configured to, when detecting that the current interactive data contains target parameter information corresponding to a preset statistical dimension parameter, determine second historical interactive data containing the target parameter information from the first historical interactive data. The preset statistical dimension parameter is a parameter pre-selected from the preset interactive parameters. The historical statistical information updating module 430 is configured to, based on a preset statistical condition, update historical statistical information corresponding to the target parameter information according to the historical repeated occurrence number of the second historical interactive data and the current interactive data, and obtain current statistical information.

[0116] The technical scheme of the embodiment, by pre-statistically acquiring the historical repeated occurrence number of each first historical interactive data of each object, the historical repeated occurrence number of each element that needs to be deduplicated can be acquired. When real-time statistics of statistical information corresponding to the target parameter information of the preset statistical dimension parameter is performed, for example, when the number of objects corresponding to the target parameter information is counted, when the current interactive data of the current object contains the target parameter information, the second historical interactive data containing the target parameter information can be determined from each first historical interactive data of the current object. According to the historical repeated occurrence number of the second historical interactive data and the current interactive data, the historical statistical information corresponding to the target parameter information can be updated based on the preset statistical condition, and the current statistical information can be obtained, thereby realizing real-time deduplication statistics. The preset statistical condition can be set according to user demand, for example, the preset statistical condition can be set to count only once when the repeated occurrence number of the target parameter information is greater than or equal to one, or the preset statistical condition can be set to count only once when the repeated occurrence number of the target parameter information is greater than or equal to two, thereby being applied to the scene of counting only once when the occurrence is one or more times, or being applied to the scene of counting only once when the occurrence is two or more times, greatly improving the application scope of the application scene, and meeting the personalized demand of the user.

[0117] Optionally, the device further comprises:

[0118] The parameter information matching module is configured to match each current parameter information in the current interaction data with corresponding historical parameter information in each first historical interaction data after updating the historical statistical information corresponding to the target parameter information to obtain the current statistical information.

[0119] The historical repeat occurrence number updating module is configured to update a historical repeat occurrence number of third historical interaction data if there is third historical interaction data that matches each current parameter information in the current interaction data successfully.

[0120] The current interaction data storage module is configured to store the current interaction data in a current cache unit corresponding to the current object if there is no third historical interaction data that matches each current parameter information in the current interaction data successfully, wherein the current cache unit is configured to store the first historical interaction data of the current object.

[0121] Optionally, the current interaction data further comprises an interaction mode identifier of the current object, and the interaction mode identifier comprises an addition identifier and a cancellation identifier.

[0122] Correspondingly, the historical repeat occurrence number updating module is specifically configured to update the historical repeat occurrence number of the third historical interaction data by adding a preset value if the interaction mode identifier in the current interaction data is the addition identifier, and update the historical repeat occurrence number of the third historical interaction data by subtracting the preset value if the interaction mode identifier in the current interaction data is the cancellation identifier.

[0123] Optionally, the historical statistical information updating module 430 comprises:

[0124] The historical repeat occurrence total number determination unit is configured to determine a historical repeat occurrence total number according to the historical repeat occurrence numbers of the second historical interaction data.

[0125] The current repeat occurrence total number determination unit is configured to determine a current repeat occurrence total number according to the current interaction data and the historical repeat occurrence total number.

[0126] The historical statistical information updating unit is configured to judge whether the historical repeat occurrence total number and the current repeat occurrence total number satisfy a preset statistical condition, and update the historical statistical information corresponding to the target parameter information to obtain the current statistical information according to a judgment result.

[0127] Optionally, the historical statistic information updating unit is specifically configured to: when the total number of historical repeated occurrences meets the preset statistical condition and the total number of current repeated occurrences does not meet the preset statistical condition, determining, as the current statistical information, a result of subtracting the preset number from the historical statistical information corresponding to the target parameter information; when the total number of historical repeated occurrences does not meet the preset statistical condition and the total number of current repeated occurrences meets the preset statistical condition, determining, as the current statistical information, a result of adding the preset number to the historical statistical information corresponding to the target parameter information; and when the total number of historical repeated occurrences and the total number of current repeated occurrences both meet the preset statistical condition or both do not meet the preset statistical condition, determining, as the current statistical information, the historical statistical information corresponding to the target parameter information.

[0128] Optionally, the current interaction data further comprises an interaction mode identifier of the current object, and the interaction mode identifier comprises an adding identifier and a canceling identifier.

[0129] Correspondingly, the total number of current repeated occurrences determining unit is specifically configured to: if the interaction mode identifier in the current interaction data is the adding identifier, determining, as the total number of current repeated occurrences, a result of adding the preset value to the total number of historical repeated occurrences; and if the interaction mode identifier in the current interaction data is the canceling identifier, determining, as the total number of current repeated occurrences, a result of subtracting the preset value from the total number of historical repeated occurrences.

[0130] Optionally, the apparatus further comprises:

[0131] The historical byte array generating module is configured to, before obtaining the current interaction data of the current object, at least one first historical interaction data, and the historical repeated occurrence number of the first historical interaction data pre-statistically, convert the historical parameter information in the first historical interaction data of the current object into historical parameter information of a byte type, and generate a historical byte array corresponding to the first historical interaction data according to a preset arrangement order, the historical parameter information of each byte type, and the historical repeated occurrence number of the first historical interaction data pre-statistically.

[0132] The current byte array generating module is configured to convert each current parameter information in the current interaction data of the current object into current parameter information of a byte type, and generate a current byte array corresponding to the current interaction data according to a preset arrangement order and the current parameter information of each byte type.

[0133] Optionally, the historical byte array generation module is further configured to: if the target parameter information corresponding to the preset statistical dimension parameter contains at least two statistical parameter information, perform binary coding on the target parameter information and each statistical parameter information based on a preset coding mode to obtain binary coded information corresponding to the target parameter information and binary coded information corresponding to each statistical parameter information; and determine, according to the binary coded information corresponding to each statistical parameter information, historical binary coded information corresponding to the historical parameter information of the preset statistical dimension parameter in each first historical interaction data of the current object as the historical parameter information in byte type.

[0134] Correspondingly, the current byte array generation module is further configured to: determine, according to the binary coded information corresponding to each statistical parameter information, current binary coded information corresponding to the current parameter information of the preset statistical dimension parameter in the current interaction data of the current object as the current parameter information in byte type.

[0135] Optionally, the second historical interaction data determination module 420 is further configured to:

[0136] If the target parameter information corresponding to the preset statistical dimension parameter is binary coded information, the historical binary coded information corresponding to the preset statistical dimension parameter in the first historical interaction data is subjected to a bit operation with the target parameter information, and when the bit operation result is the same as the target parameter information, it is determined that the first historical interaction data contains the target parameter information; if the target parameter information corresponding to the preset statistical dimension parameter is non-binary coded information, it is detected whether the historical parameter information corresponding to the preset statistical dimension parameter in the first historical interaction data is the same as the target parameter information, and if yes, it is determined that the first historical interaction data contains the target parameter information.

[0137] Optionally, the apparatus further comprises:

[0138] The historical hash value adding module is configured to generate a historical hash value corresponding to the historical interaction data according to each historical parameter information in the first historical interaction data, and add the historical hash value to the historical byte array;

[0139] The current hash value adding module is configured to generate a current hash value corresponding to the current interaction data according to each current parameter information in the current interaction data, and add the current hash value to the current byte array.

[0140] Optionally, the second historical interaction data determination module 420 is specifically configured to: determine, from each candidate parameter information corresponding to the preset statistical dimension parameter, target parameter information matched with the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data, and determine, from the first historical interaction data, second historical interaction data containing the target parameter information;

[0141] Correspondingly, the apparatus further comprises a historical statistical information determination module configured to: before obtaining the current statistical information, determine the historical statistical information corresponding to the target parameter information according to a mapping relationship between each candidate parameter information corresponding to the preset statistical dimension parameter and the historical statistical information.

[0142] Optionally, the current interaction data of the current object refers to current item acquisition task data of the current user; the current item acquisition task data comprises at least one of current item acquisition platform information, current item acquisition parent task identification, current item acquisition subtask identification, current item identification, current item ownership identification and current item acquisition time.

[0143] The first historical interaction data of the current object refers to historical item acquisition task data of the current user; the historical item acquisition task data comprises at least one of historical item acquisition platform information, historical item acquisition parent task identification, historical item acquisition subtask identification, historical item identification, historical item ownership identification and historical item acquisition time.

[0144] The data processing apparatus provided by the embodiments of the present application can execute the data processing method provided by any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of executing the data processing method.

[0145] It should be noted that, in the embodiments of the above data processing apparatus, each unit and module included is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for convenient mutual distinction, and does not limit the protection scope of the present application.

[0146] Embodiment five

[0147] Figure 6 A structural schematic diagram of a device provided by the fifth embodiment of the present application. Figure 6 A block diagram of an exemplary device 12 suitable for use in implementing embodiments of the present application is shown. Figure 6 The device 12 shown is merely one example and should not be construed as limiting the scope of the present application.

[0148] As shown in Figure 6 The device 12 is shown in the form of a general-purpose computing device. The components of the device 12 can include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects the various system components, including the system memory 28 and the processing unit 16.

[0149] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0150] Device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by device 12, including volatile and non-volatile media, removable and non-removable media.

[0151] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 6 Not shown, often called a "hard drive"). Although Figure 6 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0152] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0153] Device 12 can also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; other devices that enable a user to interact with device 12; and / or any devices (e.g., net cards, modems, etc.) that enable device 12 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interface 22. Still yet, device 12 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network such as the Internet, via network adapter 20. As depicted, network adapter 20 communicates with the other components of device 12 via bus 18. It should be appreciated that although not shown, other hardware and / or software modules could be used in conjunction with device 12. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0154] Processing unit 16 performs various overall processing functions for device 12 by executing instructions stored in system memory 28. For example, processing unit 16 can implement a data processing method according to any embodiment of the present application, which includes the following steps:

[0155] obtaining current interaction data of a current object, at least one first historical interaction data, and a historical repeat occurrence number of the first historical interaction data, wherein the current interaction data includes current parameter information corresponding to at least one preset interaction parameter, and the first historical interaction data includes historical parameter information corresponding to each preset interaction parameter;

[0156] when it is detected that the current interaction data includes target parameter information corresponding to a preset statistical dimension parameter, determining second historical interaction data including the target parameter information from the first historical interaction data, wherein the preset statistical dimension parameter is a parameter preselected from the preset interaction parameters;

[0157] updating historical statistical information corresponding to the target parameter information based on a preset statistical condition, the historical repeat occurrence number of the second historical interaction data, and the current interaction data, to obtain current statistical information.

[0158] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the data processing method provided by any embodiment of the present application.

[0159] Embodiment six

[0160] The embodiment provides a computer readable storage medium, and a computer program is stored on the computer readable storage medium. The program is executed by a processor to implement the steps of the data processing method according to any embodiment of the present application, and the method includes the following steps:

[0161] Acquire current interaction data of the current object, at least one first historical interaction data and a pre-statistical number of historical repeated occurrences of the first historical interaction data, wherein the current interaction data includes current parameter information corresponding to at least one preset interaction parameter, and the first historical interaction data includes historical parameter information corresponding to each preset interaction parameter;

[0162] When it is detected that the current interaction data contains target parameter information corresponding to a preset statistical dimension parameter, second historical interaction data containing the target parameter information is determined from the first historical interaction data, wherein the preset statistical dimension parameter is a parameter preselected from the preset interaction parameter;

[0163] Based on a preset statistical condition, the historical statistical information corresponding to the target parameter information is updated according to the historical repeated occurrences of the second historical interaction data and the current interaction data, and current statistical information is obtained.

[0164] The computer storage medium of the embodiment of the application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or component.

[0165] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or component.

[0166] The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0167] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0168] Those skilled in the art will appreciate that the modules or steps of the present application described above can be implemented in a general purpose computer, and they can be centralized in a single computing device or distributed over a network of multiple computing devices. Alternatively, they can be implemented by computer executable program codes, which can be stored in a storage device and executed by a computing device, or they can be implemented by individual integrated circuit modules, or a plurality of modules or steps can be implemented by a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0169] Note that the above only describes the preferred embodiments of the present application and the principles of the applied technology. Those skilled in the art will understand that the present application is not limited to the specific embodiments described above, and that various obvious changes, re-adjustments and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the appended claims.

Claims

1. A data processing method, characterized by, The method comprises the following steps: obtaining current interaction data of a current object, at least one first historical interaction data and a historical repeat occurrence number of the first historical interaction data, wherein the current interaction data comprises current parameter information corresponding to at least one preset interaction parameter, and the first historical interaction data comprises historical parameter information corresponding to each preset interaction parameter; when it is detected that the current interaction data comprises target parameter information corresponding to a preset statistical dimension parameter, determining second historical interaction data comprising the target parameter information from the first historical interaction data, wherein the preset statistical dimension parameter is a parameter selected from the preset interaction parameter in advance; determining a total historical repeat occurrence number according to the historical repeat occurrence number of each second historical interaction data; determining a current total repeat occurrence number according to the current interaction data and the total historical repeat occurrence number; judging whether the total historical repeat occurrence number and the current total repeat occurrence number satisfy a preset statistical condition, updating historical statistical information corresponding to the target parameter information according to a judgment result, and obtaining current statistical information; wherein judging whether the total historical repeat occurrence number and the current total repeat occurrence number satisfy a preset statistical condition, updating historical statistical information corresponding to the target parameter information according to a judgment result, and obtaining current statistical information comprises: when the total historical repeat occurrence number satisfies the preset statistical condition and the current total repeat occurrence number does not satisfy the preset statistical condition, determining a result obtained by subtracting a preset number from the historical statistical information corresponding to the target parameter information as the current statistical information; when the total historical repeat occurrence number does not satisfy the preset statistical condition and the current total repeat occurrence number satisfies the preset statistical condition, determining a result obtained by adding a preset number to the historical statistical information corresponding to the target parameter information as the current statistical information.

2. The method of claim 1, wherein, After updating the historical statistical information corresponding to the target parameter information and obtaining the current statistical information, the method further comprises the following steps: matching each current parameter information in the current interaction data with corresponding historical parameter information in each first historical interaction data; if there is third historical interaction data that matches each current parameter information in the current interaction data successfully, updating the historical repeat occurrence number of the third historical interaction data; if there is no third historical interaction data that matches each current parameter information in the current interaction data successfully, storing the current interaction data in a current cache unit corresponding to the current object, wherein the current cache unit is used to store the first historical interaction data of the current object.

3. The method of claim 2, wherein, The current interaction data further comprises an interaction mode identifier of the current object, and the interaction mode identifier comprises an addition identifier and a cancellation identifier. Correspondingly, updating the historical repeat occurrence number of the third historical interaction data comprises: if the interaction mode identifier in the current interaction data is the addition identifier, updating the historical repeat occurrence number of the third historical interaction data by adding a preset value; If the interaction mode identifier in the current interaction data is the cancel identifier, the historical repeat occurrence number of the third historical interaction data is updated by subtracting a preset value.

4. The method of claim 1, wherein, It is judged whether the historical repeat occurrence total number and the current repeat occurrence total number satisfy a preset statistical condition, and the historical statistical information corresponding to the target parameter information is updated according to the judgment result to obtain current statistical information, and the method further includes: When the historical repeat occurrence total number and the current repeat occurrence total number both satisfy the preset statistical condition or both do not satisfy the preset statistical condition, the historical statistical information corresponding to the target parameter information is determined as the current statistical information.

5. The method of claim 1, wherein, The current interaction data further includes an interaction mode identifier of the current object, and the interaction mode identifier includes an add identifier and a cancel identifier. Accordingly, the current repeat occurrence total number is determined according to the current interaction data and the historical repeat occurrence total number, including: If the interaction mode identifier in the current interaction data is the add identifier, the historical repeat occurrence total number is determined as the current repeat occurrence total number by adding a preset value. If the interaction mode identifier in the current interaction data is the cancel identifier, the historical repeat occurrence total number is determined as the current repeat occurrence total number by subtracting a preset value.

6. The method of claim 1, wherein, Before obtaining the current interaction data of the current object, at least one first historical interaction data and the historical repeat occurrence number of the first historical interaction data, the method includes: Converting the historical parameter information in the first historical interaction data of the current object into historical parameter information of a byte type, and generating a historical byte array corresponding to the first historical interaction data according to a preset arrangement order, each historical parameter information of the byte type and the historical repeat occurrence number of the first historical interaction data pre-statistically obtained; Converting each current parameter information in the current interaction data of the current object into current parameter information of a byte type, and generating a current byte array corresponding to the current interaction data according to the preset arrangement order and each current parameter information of the byte type.

7. The method of claim 6, wherein, Converting the historical parameter information in each first historical interaction data of the current object into historical parameter information of a byte type, including: If the target parameter information corresponding to the preset statistical dimension parameter includes at least two statistical parameter information, the target parameter information and each statistical parameter information are binary coded based on a preset coding mode to obtain binary coded information corresponding to the target parameter information and binary coded information corresponding to each statistical parameter information; According to the binary coded information corresponding to each statistical parameter information, the historical binary coded information corresponding to the historical parameter information of the preset statistical dimension parameter in each first historical interaction data of the current object is determined as the historical parameter information of the byte type; Accordingly, converting each current parameter information in the current interaction data of the current object into current parameter information of a byte type, including: According to the binary coded information corresponding to each statistical parameter information, current binary coded information corresponding to current parameter information of the preset statistical dimension parameter in the current interaction data of the current object is determined as current parameter information of the byte type.

8. The method of claim 7, wherein, The second historical interaction data containing the target parameter information is determined from the first historical interaction data, including: If the target parameter information corresponding to the preset statistical dimension parameter is binary coded information, the historical binary coded information corresponding to the preset statistical dimension parameter in the first historical interaction data is subjected to a bit operation with the target parameter information, and when the bit operation result is the same as the target parameter information, it is determined that the first historical interaction data contains the target parameter information. If the target parameter information corresponding to the preset statistical dimension parameter is non-binary coded information, it is detected whether the historical parameter information corresponding to the preset statistical dimension parameter in the first historical interaction data is the same as the target parameter information, and if so, it is determined that the first historical interaction data contains the target parameter information.

9. The method of claim 6, wherein, The method further includes: According to each historical parameter information in the first historical interaction data, a historical hash value corresponding to the historical interaction data is generated, and the historical hash value is added to the historical byte array; According to each current parameter information in the current interaction data, a current hash value corresponding to the current interaction data is generated, and the current hash value is added to the current byte array.

10. The method according to any of claims 1 to 9, characterized in that, When it is detected that the current interaction data contains the target parameter information corresponding to the preset statistical dimension parameter, the second historical interaction data containing the target parameter information is determined from the first historical interaction data, including: From each candidate parameter information corresponding to the preset statistical dimension parameter, target parameter information matching the current parameter information corresponding to the preset statistical dimension parameter in the current interaction data is determined, and the second historical interaction data containing the target parameter information is determined from the first historical interaction data; Correspondingly, before judging whether the historical total number of repeated occurrences and the current total number of repeated occurrences satisfy the preset statistical condition, and updating the historical statistical information corresponding to the target parameter information according to the judgment result to obtain the current statistical information, it further includes: According to the mapping relationship between each candidate parameter information corresponding to the preset statistical dimension parameter and the historical statistical information, the historical statistical information corresponding to the target parameter information is determined.

11. The method of any one of claims 1-9, wherein: The current interaction data of the current object refers to current item acquisition task data of a current user; and the current item acquisition task data includes at least one of current item acquisition platform information, a current item acquisition parent task identifier, a current item acquisition subtask identifier, a current item identifier, a current item ownership identifier, and a current item acquisition time. The first historical interaction data of the current object refers to historical item acquisition task data of the current user; and the historical item acquisition task data includes at least one of historical item acquisition platform information, historical item acquisition parent task identification, historical item acquisition subtask identification, historical item identification, historical item ownership identification and historical item acquisition time.

12. A data processing apparatus, characterized by Comprise: An interaction data acquisition module, configured to acquire current interaction data of a current object, at least one first historical interaction data and a historical repetition number of the first historical interaction data pre-statistically acquired, wherein the current interaction data includes current parameter information corresponding to at least one preset interaction parameter, and the first historical interaction data includes historical parameter information corresponding to each preset interaction parameter; A second historical interaction data determination module, configured to determine second historical interaction data containing target parameter information corresponding to a preset statistical dimension parameter from the first historical interaction data when detecting that the current interaction data contains the target parameter information, wherein the preset statistical dimension parameter is a parameter pre-selected from the preset interaction parameters; A historical statistical information update module, configured to determine a total historical repetition number according to historical repetition numbers of each second historical interaction data; determine a current total repetition number according to current interaction data and the total historical repetition number; determine current statistical information corresponding to the target parameter information as a result of the historical statistical information minus a preset number when the total historical repetition number meets a preset statistical condition and the current total repetition number does not meet the preset statistical condition; and determine the current statistical information corresponding to the target parameter information as a result of the historical statistical information plus a preset number when the total historical repetition number does not meet the preset statistical condition and the current total repetition number meets the preset statistical condition.

13. A data processing device, characterized by The device comprises: One or more processors; Memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as claimed in any one of claims 1-11.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the data processing method as claimed in any one of claims 1-11.

Images