A method, device, terminal and medium for data deduplication

Abstract

This invention discloses a method, apparatus, terminal, and medium for data deduplication. The method includes: receiving deduplication rules and determining the deduplication attributes of target deduplicated data; reading initial reported data and determining the reporting scenario of the initial reported data based on the data state of the initial reported data, the reporting scenario including a reported scenario and a pre-reported scenario; performing single-attribute identification on the initial reported data of the reported scenario to obtain first single-attribute data, and determining first target reported data based on the first single-attribute data; performing single-attribute identification on the initial reported data of the pre-reported scenario to obtain second single-attribute data, and determining second target reported data and the target deduplicated data based on the first single-attribute data, the second single-attribute data, and the deduplication attributes. This invention can perform data deduplication with precision down to single attributes, improving the accuracy of reported data and broadening the applicable scenarios of the data deduplication method.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, terminal and medium for deduplication of data. Background Art

[0002] With the development of the Internet of Things, people have higher and higher requirements on the quality of data reported by devices. In the reporting scenario of state-changing data, the data changes repeatedly between certain values. Using data deduplication to report data can reduce the bandwidth cost during data transmission, while reducing data storage costs and improving data analysis efficiency.

[0003] However, the existing reporting scenarios of state-changing data can only roughly identify the entire report message. Only when the entire report message is identified as completely repeated with the entire report message of the previous time period, the data will be deduplicated and then reported. At the same time, the existing state-changing data will only be reported when the overall data state changes, and its reporting time is unstable. It can be seen that the existing data deduplication technology is applicable to a single scenario, cannot accurately identify single-attribute data for data deduplication, and cannot determine the data duplication reporting situation in scenarios such as single-attribute cross-reporting and single-attribute and multi-attribute mixed reporting.

[0004] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method, device, terminal and medium for data deduplication are provided, aiming to solve the problems in the prior art that the data reporting time of the state-changing data reporting scenario is unstable, the data reporting accuracy is low, and only the data status of the entire message can be judged, the adaptation scenario is single, and accurate data reporting effect cannot be achieved.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for deduplication of data, wherein the method comprises:

[0008] Receive deduplication rules and determine deduplication properties of target deduplication data;

[0009] Reading the initial reported data, and determining a reporting scenario of the initial reported data according to a data state of the initial reported data, wherein the reporting scenario includes a reported scenario and a pre-reporting scenario;

[0010] Performing single attribute recognition on the initial reported data of the reported scene to obtain first single attribute data, and determining first target reported data based on the first single attribute data;

[0011] Single attribute identification is performed on the initial reporting data of the pre-reporting scenario to obtain second single attribute data, and second target reporting data and the target deduplication data are determined based on the first single attribute data, the second single attribute data and the deduplication attribute.

[0012] In one implementation, the reading of the initial reported data and determining, according to the data state of the initial reported data, a reporting scenario of the initial reported data, wherein the reporting scenario includes a reported scenario and a pre-reporting scenario, include:

[0013] Acquire initial reported data, and monitor the data status of the initial reported data;

[0014] When the data state of the initially reported data has not changed, determining that the reporting scenario of the initially reported data is the reported scenario;

[0015] When the data state of the initially reported data changes, it is determined that the reporting scenario of the initially reported data is the pre-reporting scenario.

[0016] In one implementation, performing single attribute identification on the initial reported data of the reported scene to obtain first single attribute data, and determining first target reported data based on the first single attribute data includes:

[0017] Separating the initial reported data of the reported scene by attributes to obtain first single attribute data;

[0018] The first single attribute data is used as the first target reporting data.

[0019] In one implementation, the performing single attribute identification on the initial reporting data of the pre-reporting scenario to obtain second single attribute data, and determining second target reporting data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute, includes:

[0020] Performing single attribute recognition on the initial reporting data of the pre-reporting scenario to obtain second single attribute data;

[0021] Determining the target deduplicated data based on the first single attribute data, the second single attribute data and the deduplicated attribute;

[0022] The second target reporting data is obtained based on the second single attribute data and the target deduplication data.

[0023] In one implementation, the determining the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute includes:

[0024] Compare the second single attribute data with the deduplication attribute to obtain first deduplication data belonging to the deduplication attribute in the second single attribute data;

[0025] Comparing the first deduplicated data with the first single attribute data to obtain second deduplicated data in which the deduplicated data is equal to the first target reported data;

[0026] Based on the second deduplication data, the target deduplication data is determined.

[0027] In one implementation, obtaining the second target reported data based on the second single attribute data and the target deduplication data includes:

[0028] The second single attribute data is subtracted from the target deduplicated data to obtain the second target reported data.

[0029] In one implementation, the method further includes:

[0030] A memory time is set, and a reporting scenario of the initial reporting data is determined according to the memory time.

[0031] In a second aspect, an embodiment of the present invention further provides a device for deduplication of data, wherein the device comprises:

[0032] A deduplication attribute determination module, used for receiving deduplication rules and determining deduplication attributes of target deduplication data;

[0033] A reporting scenario determination module, used to read the initial reporting data, and determine the reporting scenario of the initial reporting data according to the data state of the initial reporting data, wherein the reporting scenario includes a reported scenario and a pre-reporting scenario;

[0034] A first target reported data determination module, configured to perform single attribute recognition on the initial reported data of the reported scene to obtain first single attribute data, and determine first target reported data based on the first single attribute data;

[0035] The second target reporting data determination module is used to perform single attribute identification on the initial reporting data of the pre-reporting scenario to obtain second single attribute data, and determine the second target reporting data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute.

[0036] In a third aspect, an embodiment of the present invention further provides a terminal, wherein the terminal includes a memory, a processor, and a data deduplication program stored in the memory and executable on the processor, and when the processor executes the data deduplication program, the steps of the data deduplication method of any one of the above-mentioned schemes are implemented.

[0037] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein a data deduplication program is stored on the computer-readable storage medium, and when the data deduplication program is executed by a processor, the steps of the data deduplication method described in any one of the above-mentioned schemes are implemented.

[0038] Beneficial effects: Compared with the prior art, the present invention provides a method, device, terminal and medium for data deduplication, which first receives deduplication rules and determines the deduplication attributes of the target deduplication data. Then, the initial reported data is read, and according to the data status of the initial reported data, the reporting scenario of the initial reported data is determined, and the reporting scenario includes a reported scenario and a pre-reported scenario. Next, single attribute recognition is performed on the initial reported data of the reported scenario to obtain first single attribute data, and the first target reported data is determined based on the first single attribute data. Finally, single attribute recognition is performed on the initial reported data of the pre-reporting scenario to obtain second single attribute data, and the second target reported data and the target deduplication data are determined based on the first single attribute data, the second single attribute data and the deduplication attributes. It can be seen that the present invention can perform data deduplication accurately to a single attribute, improve the accuracy of the reported data, and also broaden the applicable scenarios of the data deduplication method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flowchart of a specific implementation of the method for data deduplication provided by an embodiment of the present invention.

[0040] Figure 2 A schematic diagram of the implementation steps of existing data deduplication methods.

[0041] Figure 3 A schematic diagram of specific implementation steps of existing data deduplication methods.

[0042] Figure 4 A schematic diagram of specific implementation steps of the method for data deduplication provided by an embodiment of the present invention.

[0043] Figure 5 A functional schematic diagram of a data deduplication device provided in an embodiment of the present invention.

[0044] Figure 6 A functional block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0046] Those skilled in the art will appreciate that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms, and the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or their groups. It should be understood that when an element is "connected" or "coupled" to another element, it may be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0047] The present embodiment provides a method, device, terminal and medium for data deduplication. When implemented specifically, the present embodiment first receives deduplication rules and determines the deduplication attributes of the target deduplication data. Then, the initial reported data is read, and according to the data state of the initial reported data, the reporting scenario of the initial reported data is determined, and the reporting scenario includes a reported scenario and a pre-reported scenario. Next, single attribute recognition is performed on the initial reported data of the reported scenario to obtain first single attribute data, and the first target reported data is determined based on the first single attribute data. Finally, single attribute recognition is performed on the initial reported data of the pre-reporting scenario to obtain second single attribute data, and the second target reported data and the target deduplication data are determined based on the first single attribute data, the second single attribute data and the deduplication attribute. It can be seen that the present embodiment can accurately perform data deduplication to a single attribute, improve the accuracy of the reported data, and also broaden the applicable scenarios of the data deduplication method.

[0048] Exemplary Methods

[0049] The data deduplication method of this embodiment can be applied to a terminal, which can be a preset mobile terminal such as a mobile phone, a tablet, a computer, or an intelligent device such as a smart TV. In this embodiment, the terminal can be an external device connected to the data deduplication device, or a device built into the data deduplication device. Figure 1 As shown in , the data deduplication method of this embodiment includes the following steps:

[0050] Step S100: receiving a deduplication rule and determining a deduplication attribute of target deduplication data.

[0051] like Figure 2 , Figure 3 and Figure 4 As shown, the devices in this embodiment store various data to be reported. When these devices are connected to the controller (i.e. Figure 2 , Figure 3 and Figure 4 When L2 in the example is reached, all the data to be reported can be cleaned in the controller, and then the cleaned data can be reported to the corresponding data receiving platform, which can effectively reduce the bandwidth cost during data transmission, reduce the data storage cost of the platform, and improve the data analysis efficiency of the platform. The controller in this embodiment takes the Shenzhen Kaihong Super Device Controller based on the KaiHongOS system as an example.

[0052] Common data cleaning technologies usually include: data filtering, data deduplication and data aggregation. Specifically, the data cleaning technology used in this implementation is data deduplication. By deduplicating the reported data received by the controller and then reporting it, the data storage cost of the platform is effectively reduced and the data analysis efficiency of the platform is improved. Generally speaking, different data cleaning technologies correspond to different types of data to be reported. Common data types to be reported include: numerical fluctuation data, that is, filtering invalid or slightly changed data points according to the set threshold; numerical change data / continuous growth data, that is, only retaining the cumulative value of the last state; data repetitive data, that is, removing duplicate records through a deduplication algorithm; character-type state change data / state change data, that is, merging consecutive repeated states or only recording key state change events. Specifically, the data cleaning method of data deduplication used in this embodiment is precisely aimed at solving the reporting scenarios of character-type state change data / state change data.

[0053] In the specific implementation, this embodiment first needs to receive the deduplication rules issued by the data receiving platform (i.e. Figure 2 , Figure 3 and Figure 4 In the delivery rules in the example above), the deduplication attribute of the target deduplication data is determined according to the deduplication rule, and the deduplication attribute provides a reference indicator for subsequent data deduplication with precision to a single attribute. Figure 2 , Figure 3 and Figure 4 The attributes to be deduplicated are attribute a, attribute b, and attribute c.

[0054] Step S200: read the initial reporting data, and determine the reporting scenario of the initial reporting data according to the data status of the initial reporting data, wherein the reporting scenario includes a reported scenario and a pre-reporting scenario.

[0055] In one implementation, step S200 of this embodiment specifically includes the following steps:

[0056] Step S201: Acquire initial reported data and monitor the data status of the initial reported data;

[0057] Step S202: When the data state of the initially reported data has not changed, determining that the reporting scenario of the initially reported data is the reported scenario;

[0058] Step S203: When the data state of the initially reported data changes, determine that the reporting scenario of the initially reported data is the pre-reporting scenario.

[0059] Since the data cleaning object of this embodiment is state-changing data, the reported data of this type of data will be updated only when the data state changes, so this embodiment determines the reporting scenario of the initial reported data by monitoring the data state of the initial reported data. In this embodiment, the reporting scenarios of the initial reported data before the data state of the initial reported data has not changed are collectively referred to as reported scenarios; the reporting scenarios after the data state of the initial reported data has changed are collectively referred to as pre-reporting scenarios.

[0060] Step S300: Perform single attribute recognition on the initial reported data of the reported scene to obtain first single attribute data, and determine first target reported data based on the first single attribute data.

[0061] In specific implementation, this embodiment will deduplicate the initial reported data of the reported scenario before the state changes, accurately to the attributes, so as to solve the technical problem in the prior art that the state change data can only judge the data state of the entire message, affecting the accuracy of data deduplication.

[0062] In one implementation, step S300 of this embodiment specifically includes the following steps:

[0063] Step S301: separate the attributes of the initially reported data of the reported scene to obtain first single attribute data;

[0064] Step S302: Use the first single attribute data as the first target reporting data.

[0065] The initial reported data of the reported scenes in this embodiment are all the data stored in the device that is first read by the KaiHongOS system's Deep Kaihong Super Device Controller. At this time, all the initial reported data appear for the first time. Therefore, all the initial reported data of the reported scenes in this embodiment do not need to be deduplicated and are all used as the first target reported data. Specifically, the initial reported data of the reported scenes in this embodiment can be understood as Figure 2 The reported message {a=20, b=10} in , after the attribute separation, the first single attribute data can be obtained, for example, Figure 3 and Figure 4In the reported data part, the reporting messages {a=20} and {b=10} of the reported data, since the reporting message {a=20} of attribute a and the reporting message {b=10} of attribute b both appear for the first time, there is no need to deduplicate the first single attribute data and they are all used as the first target reporting data.

[0066] Step S400: perform single attribute identification on the initial reporting data of the pre-reporting scenario to obtain second single attribute data, and determine second target reporting data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute.

[0067] In one implementation, step S400 of this embodiment specifically includes the following steps:

[0068] Step S401: Perform single attribute recognition on the initial reporting data of the pre-reporting scenario to obtain second single attribute data;

[0069] Step S402: determining the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute;

[0070] Step S403: Obtain the second target reporting data based on the second single attribute data and the target deduplication data.

[0071] Specifically, the initial reporting data of the pre-reporting scenario in this embodiment is the data stored in the device that is subsequently read by the Deep Kaihong Super Device Controller based on the KaiHongOS system, so it is necessary to perform deduplication judgment on it. First, this embodiment performs single attribute recognition on the initial reporting data of the pre-reporting scenario to obtain second single attribute data. Then, the second single attribute data is compared with the first single attribute data to obtain a comparison result. Finally, based on the comparison result, it is determined whether to classify the second single attribute data as the second target reporting data or as the target deduplication data.

[0072] In this embodiment, when the second single attribute data is the same as the previously read data, it is necessary to deduplicate the second single attribute data, that is, to classify it as the target deduplicated data. When the second single attribute data is not the same as the previously read data, it is not necessary to deduplicate the second single attribute data, and the second single attribute data is retained and classified as the second target reported data. At the same time, the target reported data in this embodiment includes the first target reported data and the second target reported data.

[0073] In one implementation, step S402 of this embodiment specifically includes the following steps:

[0074] Step S4021: Compare the second single attribute data with the deduplication attribute to obtain first deduplication data belonging to the deduplication attribute in the second single attribute data;

[0075] Step S4022: Compare the first deduplicated data with the first single attribute data to obtain second deduplicated data in which the deduplicated data is equal to the first target reported data;

[0076] Step S4023: Determine the target deduplicated data based on the second deduplicated data.

[0077] like Figure 4 As shown, when the pre-reporting scenario of this embodiment is part A of the data to be reported, then the initial reported data of this embodiment is: {a=20, b=10}, and the second single attribute data is: {a=20} and {b=10}. Since the deduplication attributes of the target deduplication data determined in this embodiment are three attributes: a attribute, b attribute, and c attribute, by comparing the second single attribute data with the deduplication attributes, it can be obtained that the first single attribute data is {a=20} and {b=10}. Furthermore, by comparing the first deduplication data with the first single attribute data, it can be obtained that the first single attribute data {a=20} and {b=10} both appear in the reported data part, then the second deduplication data is {a=20} and {b=10}. Among them, in this embodiment, the second deduplication data is used as the target deduplication data, that is, the target deduplication data is {a=20} and {b=10}, that is, the target deduplication data {a=20} and {b=10} are cleaned in the L2 controller and are not reported.

[0078] In one implementation, step S403 of this embodiment specifically includes the following steps:

[0079] Step S4031: Subtract the target deduplicated data from the second single attribute data to obtain the second target reported data.

[0080] When the second single attribute data is the same as the previously read data, it is necessary to deduplicate the second single attribute data, that is, to classify it as the target deduplicated data. When the second single attribute data is not the same as the previously read data, it is not necessary to deduplicate the second single attribute data, and the second single attribute data is retained and classified as the second target reported data.

[0081] That is to say, in this embodiment, the second target reported data can be obtained by subtracting the target deduplication data from the second single attribute data. Figure 4In the example, when the pre-reporting scenario of this embodiment is the data part C to be reported, the first target reported data is {a=20} and {b=10}, the initial reported data is: {a=20, b=10, c=5}, the second single attribute data is: {a=20}, {b=10} and {c=5}, and the target deduplication data is: {a=20} and {b=10}. Then, the second target reported data is {c=5}, that is, the second target reported data is {c=5}, which is not cleaned in the L2 controller and is reported.

[0082] That is to say, the data deduplication method of this embodiment is accurate to a single attribute, which improves the accuracy of data deduplication and solves the problem of Figure 2 The existing data deduplication method can only judge the duplication of the reported data of the entire reported message. When there is repeated reported data of a single attribute in the initial reported data, it is impossible to achieve high-precision data deduplication (ie, data cleaning) effect. Figure 2 In the example, only when the reported data reported at the first time is {a=20, b=10} with multiple attributes, it is reported directly; when the reported data reported at the second time is still {a=20, b=10}, it will be identified and judged that the reported data is repeated, and it will be deduplicated (that is, the reported data is cleaned); however, if the reported data reported at the second time is {a=20} with a single attribute, although the reported data at the first time includes {a=20} and {b=10}, since the existing equipment can only identify the reported data of the entire message, the repeatability judgment result is that {a=20, b=10} is not equal to {a=20}, so {a=20} reported at the second time cannot be deduplicated (that is, the reported data cannot be cleaned), and will continue to be reported.

[0083] At the same time, the data deduplication method of this embodiment is more adaptable to various scenarios and solves the problem of Figure 3 The existing data deduplication methods cannot determine the problem of data duplication in scenarios such as single-attribute cross-reporting and mixed reporting of single and multiple attributes. Figure 3In the example, when the reported data parts are {a=20} and {b=10} in order of reporting time, and the data part A to be reported is {a=20} and {b=10} in order of reporting time, since the last reported data of the reported data part is {b=10}, and the first reported data of the data part A to be reported is not {b=10}, the same reported data of the data part A to be reported and the reported data part are not reported continuously, that is, the reported data {a=20} and {b=10} of the data part A to be reported continue to be reported and cannot be deduplicated. When the reported data of the data part B to be reported is {a=20, b=10}, although the reported data {a=20, b=10} contains {a=20} and {b=10}, since {a=20, b=10} is not equal to {a=20} and {a=20, b=10} is not equal to {b=10}, the reported data {a=20, b=10} of the data part B to be reported continues to be reported and cannot be deduplicated. When the reported data of the data part C to be reported is {a=20, b=10, c=5}, since {a=20, b=10, c=5} is not equal to {a=20} and {b=10} of the reported data part, the reported data {a=20, b=10, c=5} of the data part C to be reported continues to be reported and cannot be deduplicated.

[0084] In one implementation, the data deduplication method of this embodiment further includes the following steps:

[0085] Step S500: setting a memory time, and determining a reporting scenario for the initial reporting data according to the memory time.

[0086] Specifically, state-changing data is mainly divided into two categories: value-changing data and value-regular data. The value of value-changing data often changes between certain numbers, but the time of its value change is unstable. If its value remains unchanged for a long time, the data will not be reported and updated for a long time, affecting the timeliness of data reporting. The value of numerical regular data is fixed, but it will be reported along with the reporting process of each state-changing data, which undoubtedly increases the data storage cost of the platform and reduces the data analysis efficiency of the platform. For example, in the existing reported data about human body sensing radar, 6 attribute data are reported each time, of which 5 attributes are reported as fixed values ​​each time, such as: device ID, device type, device type name, device alias and device alias ID. The reported data of these 5 attributes are called numerical regular data.

[0087] Therefore, this embodiment adds a memory time to the reporting process of the initial reported data. When the data state has not changed, the data can be reported as long as the preset memory time is reached. At the same time, the memory time is also refreshed, further improving the reporting timeliness of the state-changing data. Optionally, this embodiment can also give priority to judging the attribute type of the reported data, screen out the regular attributes, divide the reporting process of the numerical change type and the numerical regular type data, and then set a memory time for the reporting data specifically for the regular attributes, so that the reported data of the regular attributes is only reported once within a cycle of memory time, thereby reducing the data storage cost and improving the transmission rate of the reported data of other state-changing attributes. Among them, the memory time in this embodiment is set by the user independently, which can be one second, one hour, or one year.

[0088] In summary, this embodiment provides a method for data deduplication. During specific implementation, this embodiment first receives deduplication rules and determines the deduplication attributes of the target deduplication data. Then, the initial reported data is read, and according to the data state of the initial reported data, the reporting scenario of the initial reported data is determined, and the reporting scenario includes a reported scenario and a pre-reported scenario. Next, single attribute identification is performed on the initial reported data of the reported scenario to obtain first single attribute data, and the first target reported data is determined based on the first single attribute data. Finally, single attribute identification is performed on the initial reported data of the pre-reporting scenario to obtain second single attribute data, and the second target reported data and the target deduplication data are determined based on the first single attribute data, the second single attribute data and the deduplication attribute. It can be seen that this embodiment can perform data deduplication accurately to a single attribute, which improves the accuracy of the reported data and also broadens the applicable scenarios of the data deduplication method.

[0089] Exemplary Devices

[0090] Based on the above embodiments, the present invention also provides a device for deduplication of data, such as Figure 5As shown in, the data deduplication device includes: a deduplication attribute determination module 10, a reporting scenario determination module 20, a first target reported data determination module 30 and a second target reported data determination module 40. Specifically, the deduplication attribute determination module 10 is used to receive a deduplication rule and determine the deduplication attribute of the target deduplication data. The reporting scenario determination module 20 is used to read the initial reported data and determine the reporting scenario of the initial reported data according to the data state of the initial reported data, and the reporting scenario includes a reported scenario and a pre-reported scenario. The first target reported data determination module 30 is used to perform single attribute recognition on the initial reported data of the reported scenario to obtain first single attribute data, and determine the first target reported data based on the first single attribute data. The second target reported data determination module 40 is used to perform single attribute recognition on the initial reported data of the pre-reporting scenario to obtain second single attribute data, and determine the second target reported data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute.

[0091] In one implementation, the reporting scenario determination module 20 includes:

[0092] A data status monitoring unit, used to obtain initial reported data and monitor the data status of the initial reported data;

[0093] a reported scenario determining unit, configured to determine, when the data state of the initially reported data has not changed, that the reporting scenario of the initially reported data is the reported scenario;

[0094] The pre-reporting scenario determining unit is configured to determine that the reporting scenario of the initial reporting data is the pre-reporting scenario when the data state of the initial reporting data changes.

[0095] In one implementation, the first target reporting data determining module 30 includes:

[0096] A first single attribute data determining unit, configured to perform attribute separation on the initial reported data of the reported scene to obtain first single attribute data;

[0097] The first target reporting data determining unit is configured to use the first single attribute data as the first target reporting data.

[0098] In one implementation, the second target reporting data determining module 40 includes:

[0099] a second single attribute data determining unit, configured to perform single attribute recognition on the initial reporting data of the pre-reporting scenario to obtain second single attribute data;

[0100] a target deduplication data determining unit, configured to determine the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute;

[0101] The second target reported data determining unit is used to obtain the second target reported data based on the second single attribute data and the target deduplication data.

[0102] In one implementation, the target deduplication data determination unit includes:

[0103] A first deduplication data determination subunit is used to compare the second single attribute data with the deduplication attribute to obtain first deduplication data belonging to the deduplication attribute in the second single attribute data;

[0104] A second deduplication data determination subunit is used to compare the first deduplication data with the first single attribute data to obtain second deduplication data in which the deduplication data is equal to the first target reported data;

[0105] The target deduplication data determination subunit is used to determine the target deduplication data based on the second deduplication data.

[0106] In one implementation, the second target reporting data determining unit includes:

[0107] The second target reported data determination subunit is used to subtract the target deduplication data from the second single attribute data to obtain the second target reported data.

[0108] In one implementation, the data deduplication device further includes:

[0109] The memory time setting module is used to set the memory time and determine the reporting scenario of the initial reporting data according to the memory time.

[0110] The working principles of each module in the data deduplication device of this embodiment are the same as the principles of each step in the above method embodiment, and will not be repeated here.

[0111] Based on the above embodiment, the present invention further provides a terminal, the principle block diagram of the terminal can be as follows: Figure 6 The terminal may include one or more processors 100 ( Figure 6Only one is shown in the figure), a memory 101 and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a data deduplication program. When one or more processors 100 execute the computer program 102, the various steps in the method embodiment for data deduplication can be implemented. Alternatively, when one or more processors 100 execute the computer program 102, the functions of the various modules / units in the device embodiment for data deduplication can be implemented, which is not limited here.

[0112] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0113] In one embodiment, the memory 101 may be an internal storage unit of an electronic device, such as a hard disk or memory of the electronic device. The memory 101 may also be an external storage device of the electronic device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 101 may also include both an internal storage unit of the electronic device and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0114] Those skilled in the art will understand that Figure 6 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, operating database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double operational data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0116] In summary, the present invention discloses a method, device, terminal and medium for data deduplication, the method comprising: receiving deduplication rules, determining the deduplication attributes of target deduplication data; reading initial reported data, and determining the reporting scenario of the initial reported data according to the data status of the initial reported data, the reporting scenario including a reported scenario and a pre-reported scenario; performing single attribute identification on the initial reported data of the reported scenario to obtain first single attribute data, and determining first target reported data based on the first single attribute data; performing single attribute identification on the initial reported data of the pre-reported scenario to obtain second single attribute data, and determining second target reported data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attributes. The present invention can accurately perform data deduplication to a single attribute, improves the accuracy of reported data, and also broadens the applicable scenarios of the data deduplication method.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for deduplication of data. It is characterized in that The method comprises: Receive deduplication rules and determine deduplication properties of target deduplication data; Reading the initial reported data, and determining a reporting scenario of the initial reported data according to a data state of the initial reported data, wherein the reporting scenario includes a reported scenario and a pre-reporting scenario; Performing single attribute recognition on the initial reported data of the reported scene to obtain first single attribute data, and determining first target reported data based on the first single attribute data; Performing single attribute identification on the initial reporting data of the pre-reporting scenario to obtain second single attribute data, and determining second target reporting data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute; The reading of the initial reported data and determining, according to the data state of the initial reported data, a reporting scenario of the initial reported data, wherein the reporting scenario includes a reported scenario and a pre-reporting scenario, includes: Acquire initial reported data, and monitor the data status of the initial reported data; When the data state of the initially reported data has not changed, determining that the reporting scenario of the initially reported data is the reported scenario; When the data state of the initially reported data changes, it is determined that the reporting scenario of the initially reported data is the pre-reporting scenario.

2. The method for deduplication of data according to claim 1, It is characterized in that The performing single attribute identification on the initial reported data of the reported scene to obtain first single attribute data, and determining first target reported data based on the first single attribute data, includes: Separating the initial reported data of the reported scene by attributes to obtain first single attribute data; The first single attribute data is used as the first target reporting data.

3. The method for deduplication of data according to claim 2, It is characterized in that The performing single attribute identification on the initial reporting data of the pre-reporting scenario to obtain second single attribute data, and determining second target reporting data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute, includes: Performing single attribute recognition on the initial reporting data of the pre-reporting scenario to obtain second single attribute data; Determining the target deduplicated data based on the first single attribute data, the second single attribute data and the deduplicated attribute; The second target reporting data is obtained based on the second single attribute data and the target deduplication data.

4. The method for deduplication of data according to claim 3, It is characterized in that The determining the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute includes: Compare the second single attribute data with the deduplication attribute to obtain first deduplication data belonging to the deduplication attribute in the second single attribute data; Comparing the first deduplicated data with the first single attribute data to obtain second deduplicated data in which the deduplicated data is equal to the first target reported data; Based on the second deduplication data, the target deduplication data is determined.

5. The method for deduplication of data according to claim 4, It is characterized in that The obtaining the second target reported data based on the second single attribute data and the target deduplication data includes: The second single attribute data is subtracted from the target deduplicated data to obtain the second target reported data.

6. The method for deduplication of data according to claim 1, It is characterized in that The method further comprises: A memory time is set, and a reporting scenario of the initial reporting data is determined according to the memory time.

7. A device for deduplication of data, It is characterized in that The device comprises: A deduplication attribute determination module, used for receiving deduplication rules and determining deduplication attributes of target deduplication data; A reporting scenario determination module, used to read the initial reporting data, and determine the reporting scenario of the initial reporting data according to the data state of the initial reporting data, wherein the reporting scenario includes a reported scenario and a pre-reporting scenario; A first target reported data determination module, configured to perform single attribute recognition on the initial reported data of the reported scene to obtain first single attribute data, and determine first target reported data based on the first single attribute data; A second target reported data determination module, configured to perform single attribute recognition on the initial reported data of the pre-reporting scenario to obtain second single attribute data, and determine second target reported data and the target deduplication data based on the first single attribute data, the second single attribute data and the deduplication attribute; The reporting scenario determination module includes: A data status monitoring unit, used to obtain initial reported data and monitor the data status of the initial reported data; a reported scenario determining unit, configured to determine, when the data state of the initially reported data has not changed, that the reporting scenario of the initially reported data is the reported scenario; The pre-reporting scenario determining unit is configured to determine that the reporting scenario of the initial reporting data is the pre-reporting scenario when the data state of the initial reporting data changes.

8. A terminal, It is characterized in that The terminal includes a memory, a processor, and a data deduplication program stored in the memory and executable on the processor. When the processor executes the data deduplication program, the steps of the data deduplication method as described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a data deduplication program, and when the data deduplication program is executed by the processor, the steps of the data deduplication method according to any one of claims 1 to 6 are implemented.

Images