Data duplicate checking algorithm determination method and device, equipment and medium

Abstract

The invention provides a data duplicate checking algorithm determination method and device, equipment and a medium, and is used for determining a data duplicate checking algorithm suitable for data. According to the method, data can be input into a pre-trained data feature analysis model, the data source of the data is obtained by performing feature analysis on the data, and a data duplicate checking algorithm corresponding to the data source of the data can be used as a data duplicate checking algorithm according to a pre-stored corresponding relation between the data source and the data duplicate checking algorithm. According to the data duplicate checking method and device, the suitable data duplicate checking algorithm can be selected for the data based on the data duplicate checking algorithm, data duplicate checking operation can be efficiently and accurately carried out based on the data duplicate checking algorithm, the data processing efficiency and precision are improved, and the data quality and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and medium for determining a data duplication checking algorithm. Background Art

[0002] In the related art, there may be duplicate data in the storage system, which not only occupies storage space, but also increases the burden of data transmission, backup and processing. Therefore, it is very necessary to check the data for duplicates based on the data duplication checking algorithm.

[0003] However, with the development of data diversification and data duplication detection algorithms, how to select a suitable data duplication detection algorithm for data has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method, device, equipment and medium for determining a data duplication checking algorithm, which are used to determine a data duplication checking algorithm suitable for data.

[0005] In a first aspect, the present application provides a method for determining a data duplication checking algorithm, the method comprising:

[0006] Inputting the data within the statistical period into a pre-trained data feature analysis model, the data feature analysis model performs feature analysis on the data and outputs the data source of the data;

[0007] According to the pre-saved correspondence between the data source and the data duplication checking algorithm, the data duplication checking algorithm corresponding to the data source of the data is determined as the data duplication checking algorithm applicable to the data.

[0008] Through the above method, since the data can be feature analyzed through the data feature analysis model to obtain the data source of the data, and based on the correspondence between the pre-saved data source and the data duplication checking algorithm, the data duplication checking algorithm corresponding to the data source of the data is determined as the data duplication checking algorithm suitable for the data, based on this, it is possible to select a suitable data duplication checking algorithm for the data, and subsequently based on the data duplication checking algorithm, data duplication checking operations can be performed efficiently and accurately, thereby improving data processing efficiency and accuracy, and improving the quality and accuracy of the data.

[0009] In a possible implementation, the data feature analysis model performs feature analysis on the data to output the data source of the data, including:

[0010] Performing feature analysis on the data in preset dimensions to determine feature values ​​of the data corresponding to each preset dimension; wherein the preset dimensions include at least one of data volume, data type, data update frequency, and data security;

[0011] The data source of the data is determined according to the characteristic value of each preset dimension corresponding to the data.

[0012] Through the above method, since data from different data sources usually have different data characteristics in dimensions such as data volume, data type, data update frequency, and data security, the data feature analysis model performs feature analysis on the data from these dimensions, and determines the data source based on the feature values ​​of the data corresponding to these dimensions, thereby improving the accuracy of the determined data source.

[0013] In a possible implementation, performing feature analysis on the data in a preset dimension to determine a feature value of the data corresponding to each preset dimension includes:

[0014] When the preset dimension includes data update frequency, for each sub-data in the data sorted according to the statistical time, by identifying whether the value of the target field of the sub-data is the same as the value of the previous sub-data at the target field, the number of times the sub-data at the target field is changed is counted;

[0015] The data update frequency of the data is determined according to the number of times and the total duration of the statistical period.

[0016] Through the above method, for each sub-data in the data sorted according to the statistical time, it is possible to identify whether the value of the sub-data at the target field is the same as the value of the previous sub-data at the target field, count the number of times the sub-data at the target field is changed, and determine the data update frequency based on the number and the total duration of the statistical period, thereby improving the accuracy and speed of determining the data update frequency.

[0017] In a possible implementation, performing feature analysis on the data in a preset dimension to determine a feature value of the data corresponding to each preset dimension includes:

[0018] When the preset dimension includes data security, an encryption method used for the data is identified, and based on the encryption method, the data security of the data is determined.

[0019] Through the above method, since the data security of the data can be determined by identifying the encryption method used for the data, the accuracy and speed of determining the data security can be improved.

[0020] In a possible implementation, the training process of the data feature analysis model includes:

[0021] Obtain any sample data in the sample set, the sample data corresponding to the sample data source;

[0022] Performing feature analysis on the sample data using a data feature analysis model to be trained, and outputting an identification data source of the sample data;

[0023] The data feature analysis model to be trained is trained according to the sample data source and the identification data source.

[0024] Through the above method, a trained data feature analysis model can be obtained, and based on the trained data feature analysis model, the data source can be determined with the greatest degree of accuracy.

[0025] In a second aspect, the present application provides a data duplication detection algorithm determination device, which has the function of implementing the behavior of the electronic device in the method embodiment of the first aspect above, and the beneficial effects can be found in the description of the first aspect and will not be repeated here. The data duplication detection algorithm determination device includes:

[0026] An analysis module, used for inputting data within a statistical period into a pre-trained data feature analysis model, wherein the data feature analysis model performs feature analysis on the data and outputs the data source of the data;

[0027] The determination module is used to determine the data duplication checking algorithm corresponding to the data source of the data as the data duplication checking algorithm applicable to the data according to the pre-saved correspondence between the data source and the data duplication checking algorithm.

[0028] In a possible implementation, the analysis module is specifically used to:

[0029] Performing feature analysis on the data in preset dimensions to determine feature values ​​of the data corresponding to each preset dimension; wherein the preset dimensions include at least one of data volume, data type, data update frequency, and data security;

[0030] The data source of the data is determined according to the characteristic value of each preset dimension corresponding to the data.

[0031] In a possible implementation, the analysis module is specifically used to:

[0032] When the preset dimension includes data update frequency, for each sub-data in the data sorted according to the statistical time, by identifying whether the value of the target field of the sub-data is the same as the value of the previous sub-data at the target field, the number of times the sub-data at the target field is changed is counted;

[0033] The data update frequency of the data is determined according to the number of times and the total duration of the statistical period.

[0034] In a possible implementation, the analysis module is specifically used to:

[0035] When the preset dimension includes data security, an encryption method used for the data is identified, and based on the encryption method, the data security of the data is determined.

[0036] In a possible implementation, the device further includes:

[0037] The training module is used to obtain any sample data in the sample set, and the sample data corresponds to a sample data source; perform feature analysis on the sample data through the data feature analysis model to be trained, and output the identification data source of the sample data; and train the data feature analysis model to be trained according to the sample data source and the identification data source.

[0038] In a third aspect, the present application further provides an electronic device, comprising at least a processor and a memory, wherein the processor is configured to implement the steps of any method described in the first aspect when executing a computer program stored in the memory.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer performs the steps of the method as described in any one of the first aspects.

[0040] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, which, when executed on a computer, enables the computer to execute the steps of any one of the methods described in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the implementation methods in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technology descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0042] Figure 1 A schematic diagram of a data duplication checking algorithm determination process provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of another data duplication checking algorithm determination process provided in an embodiment of the present application;

[0044] Figure 3 A schematic diagram of a determination process of another data duplication checking algorithm provided in an embodiment of the present application;

[0045] Figure 4A schematic diagram of the structure of a data duplication checking algorithm determination device provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0048] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0049] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0050] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0051] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0053] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0054] The embodiments of the present application provide a method, device, equipment and medium for determining a data duplication checking algorithm, which is used to determine a data duplication checking algorithm suitable for data. In this method, data can be input into a pre-trained data feature analysis model, and the data source of the data can be obtained by performing feature analysis on the data. The data duplication checking algorithm corresponding to the data source of the data can be determined as a data duplication checking algorithm suitable for the data based on the correspondence between the pre-saved data source and the data duplication checking algorithm. Based on this, it is possible to select a suitable data duplication checking algorithm for the data, and then based on the data duplication checking algorithm, data duplication checking operations can be performed efficiently and accurately, thereby improving data processing efficiency and accuracy, and improving the quality and accuracy of the data.

[0055] Embodiment 1:

[0056] Figure 1 A schematic diagram of a data duplication checking algorithm determination process provided in an embodiment of the present application, the process includes:

[0057] S101: Inputting data within a statistical period into a pre-trained data feature analysis model, the data feature analysis model performs feature analysis on the data and outputs the data source of the data.

[0058] The data duplication checking algorithm determination method provided in the embodiment of the present application is applied to an electronic device, which may be, for example, a personal computer (PC), a mobile terminal, a server, etc., and the present application does not make any specific limitation on this.

[0059] In one possible implementation, considering that data from different data sources usually have different data characteristics, data from different data sources are usually suitable for different data duplication checking algorithms. For a certain data, in order to select (determine) a data duplication checking algorithm suitable for the data, the data source of the data can be identified by first performing feature analysis on the data, and then a suitable data duplication checking algorithm can be selected for the data based on the data source. Among them, the data source can also be referred to as the business scenario or business model (Business Model) that generates the data. The data source can include databases, virtual machines, file systems, cloud storage, and blockchains, etc., and this application does not specifically limit this.

[0060] Specifically, when determining the source of data, the data can be input into a pre-trained data feature analysis model, and the data feature analysis model can determine (identify) the source of the data by performing feature analysis on the data. The data can be data collected within a preset statistical period, and the specific duration of the statistical period can be flexibly set according to demand, and this application does not specifically limit this.

[0061] To facilitate understanding, the training process of the data feature analysis model is explained below.

[0062] In a possible implementation, the sample set used for training the data feature analysis model includes multiple sample data, each of which may correspond to a sample data source, wherein the sample data may be data of a certain statistical period.

[0063] When training the data feature analysis model, any sample data in the sample set can be obtained and input into the data feature analysis model to be trained, and the data feature analysis model can be a multilayer perceptron (MLP) neural network model. The data feature analysis model can perform feature analysis on the sample data from several dimensions, determine the feature value of each preset dimension corresponding to the sample data, and can determine and output the data source of the sample data (for ease of description, referred to as identifying the data source) based on the feature value of each preset dimension corresponding to the sample data.

[0064] In a possible implementation, it is considered that data from different data sources have different characteristics in terms of data volume, data type, data update frequency, data security, etc. For example, the data volume of data from some data sources may be small, such as usually at the megabit (Mb) level, while the data volume of data from some data sources may be large, such as usually at the gigabyte (Gb) level; for another example, the data type of data from a database is usually numerical data, and the data type of data from a file system is usually text data; for another example, data with a high update frequency, such as data from a stock trading system or data from a real-time inventory management system, is usually stored in a database, so that the data update frequency of data from a database is usually large, while that of data from a file system is usually large. The data in the system is usually read more and does not need to be modified frequently. Therefore, the data update frequency of the data whose data source is the file system is usually low. For another example, due to the encryption, identity authentication and tamper-proof characteristics of databases and blockchains, the security of the encryption algorithms used for data whose data sources are databases and blockchains is usually higher, while the security of the encryption algorithms used for data whose data sources are file systems and cloud storage is relatively lower. In a possible implementation, the data feature analysis model can perform feature analysis on the sample data from the dimensions of data volume, data type, data update frequency, data security, etc., determine the feature values ​​of the sample data corresponding to these dimensions, and determine (predict) the data source of the sample data (identify the data source) according to the feature values ​​of the sample data corresponding to each preset dimension.

[0065] The error between the source of the identified data and the source of the sample data can be measured based on the configured error loss function. The larger the value of the error loss function, the greater the error between the source of the identified data and the source of the sample data, and the less accurate the recognition result (prediction result) of the model. The gradient descent algorithm can be used to back-propagate the gradient of the model parameters and adjust the model parameters to train the model. By minimizing the value of the loss function, the prediction performance of the model can be improved.

[0066] In a specific implementation, the above operation can be performed on each sample data in the sample set, and when the preset convergence condition is met, it is determined that the training of the data feature analysis model is completed. Among them, the preset convergence condition can be met for the sample data in the sample set to pass through the model, the number of sample data correctly identified is greater than the set number, or the number of iterations for training the model reaches the set maximum number of iterations, etc. In a specific implementation, the setting can be flexibly made, and no specific limitation is made here.

[0067] In one possible implementation, when determining the data source of data based on a trained data feature analysis model, the data can be input into the trained data feature analysis model, and the data feature analysis model performs feature analysis on the data on several preset dimensions including data volume, data type, data update frequency, and data security, and determines the feature value of the data corresponding to each preset dimension. After that, the data feature analysis model can determine the data source of the data based on the feature value of the data corresponding to each preset dimension.

[0068] Since data from different sources usually have different data characteristics in terms of data volume, data type, data update frequency, data security and other dimensions, the data feature analysis model performs feature analysis on the data from these dimensions and determines the data source based on the feature values ​​of the data corresponding to these dimensions, which can improve the accuracy of the determined data source.

[0069] In a possible implementation, when the preset dimension includes data volume, the data volume is subjected to feature analysis, and the feature value of the data corresponding to the data volume dimension is determined, the data size of the data can be identified, wherein the data size of the data can be at the Gb level, the Mb level, etc. The feature value of the data corresponding to the data volume dimension can be determined according to the data size of the data. For example, the data size value of the data can be directly determined as the feature value of the data corresponding to the data volume dimension. It is also possible to preset multiple data size intervals, compare the data size of the data with the multiple data size intervals, determine the target data size interval in which the data size of the data is located, and determine the target data size interval as the feature value of the data corresponding to the data volume dimension. In addition, the data volume level corresponding to each data size interval can be further preset, for example, the data volume level can be a large data volume, a medium data volume, a small data volume, or a first level, a second level, a third level, etc. After determining the target data size interval in which the data size of the data is located, the data volume level corresponding to the target data size interval can be determined as the feature value of the data corresponding to the data volume dimension.

[0070] In a possible implementation, when a data type is included in a preset dimension, a feature analysis is performed on the data type to determine the feature value of the dimension corresponding to the data type. The data type can be identified based on the file extension of the data (also known as the file name extension), and the data type can be directly determined as the feature value of the dimension corresponding to the data type. The data type can be numerical data, text data, etc., which is not specifically limited in this application.

[0071] In a possible implementation, when the preset dimension includes data update frequency, the data update frequency of the data is subjected to feature analysis to determine the feature value of the data corresponding to the dimension of data update frequency, the statistical time of each sub-data contained in the data can be first identified, and the sub-data can be sorted according to the statistical time, for example, the sub-data can be sorted in ascending order or in descending order according to the chronological order of the statistical time, and this application does not make specific restrictions on this. Afterwards, for each sorted sub-data, the number of times the sub-data at the target field is changed can be counted by identifying whether the value of the sub-data at the target field is the same as the value of the previous sub-data at the target field that is located before it in the sorting.

[0072] Exemplarily, for each sorted sub-data, when the value of the sub-data at the target field is the same as the value of the previous sub-data before it in the sorting at the target field, it can be considered that the sub-data at the target field has not been changed; conversely, when the value of the sub-data at the target field is different from the value of the previous sub-data before it in the sorting at the target field, it can be considered that the sub-data at the target field has been changed once. After performing the above comparison process to determine whether the values ​​of the target fields of two adjacent sub-data in the sorting are the same, the total number of times the sub-data at the target field has been changed can be obtained. Afterwards, the data update frequency of the data can be determined based on the total number of times and the total duration of the statistical period. Exemplarily, the quotient of the total number of times and the total duration of the statistical period can be determined as the data update frequency of the data, that is, the quotient of the total number of times and the total duration of the statistical period can be determined as the characteristic value of the dimension of data update frequency corresponding to the data.

[0073] In a possible implementation, the number of target fields may be one or more, and this application does not specifically limit this. When the number of target fields is more than one, the total number of times the sub-data at each target field is changed can be obtained separately in the manner described above. Afterwards, the average value, minimum value, maximum value or median value of the total number of times the sub-data at these multiple target fields are changed can be determined as the final number of times the sub-data at the target field is changed. Finally, the quotient of the final number and the total duration of the statistical period can be determined as the characteristic value of the dimension of data update frequency corresponding to the data, which will not be repeated here.

[0074] Since the present application can target each sub-data in the data sorted according to statistical time, by identifying whether the value of the sub-data at the target field is the same as the value of the previous sub-data at the target field, count the number of times the sub-data at the target field is changed, and can determine the data update frequency based on the number and the total duration of the statistical period, thereby improving the accuracy and speed of determining the data update frequency.

[0075] In one possible implementation, when data security is included in the preset dimension, when performing feature analysis on the data security of the data and determining the feature value of the data corresponding to the dimension of data security, it is possible to first identify the encryption method used for the data, and then determine the data security of the data based on the encryption method. In other words, the feature value of the data corresponding to the dimension of data security can be determined based on the encryption method.

[0076] Exemplarily, the encryption method of the data, such as whether it is encrypted, and the encryption algorithm used when encrypted, etc., can be directly determined as the characteristic value of the data corresponding to the dimension of data security. In addition, the encryption levels corresponding to different encryption methods can also be pre-configured, such as the encryption level can be no encryption, low level, medium level, high level, etc., which can be flexibly set according to needs. After the encryption method of the data is identified, the encryption level corresponding to the encryption method of the data can be determined according to the correspondence between the pre-configured encryption method and the encryption level, and the encryption level can be determined as the characteristic value of the data corresponding to the dimension of data security.

[0077] Since the present application can determine the data security of the data by identifying the encryption method used for the data, the accuracy and speed of determining the data security can be improved.

[0078] S102: According to the pre-saved correspondence between the data source and the data duplication checking algorithm, the data duplication checking algorithm corresponding to the data source of the data is determined as the data duplication checking algorithm applicable to the data.

[0079] In a possible implementation, after determining the data source of the data, the data duplication checking algorithm corresponding to the data source of the data can be determined according to the correspondence between the pre-saved data source and the data duplication checking algorithm, and the data duplication checking algorithm can be determined as the data duplication checking algorithm applicable to the data. Among them, the data source and the data duplication checking algorithm corresponding to the data source can be flexibly set according to demand, and this application does not make specific restrictions on this. Exemplarily, the data duplication checking algorithm may include: a duplication checking algorithm using a structured query language (SQL) statement, a hash duplication checking algorithm, a fingerprint duplication checking algorithm, a string matching duplication checking algorithm, a duplication checking method based on block division, a Merkle tree duplication checking algorithm, a fixed-length duplication checking algorithm, a variable-length duplication checking algorithm, etc.

[0080] As another example, considering that the characteristics of data whose source is a database are structured storage, data sharing and independence, the data is neatly arranged in a table format, each row of data in the table represents a record of information, each column of data represents the information of a specific field (also called an attribute), and each row of data has a unified field. This structured storage method can significantly improve the query efficiency and management convenience of the data. When the data source (business scenario) of the data is a database scenario, the data duplication detection algorithm corresponding to the data can be a hash duplication detection algorithm, a string matching duplication detection algorithm, etc.

[0081] As another example, considering that the characteristics of data whose source is a virtual machine are resource pooling, high flexibility and isolation, when the data source is a virtual machine scenario, the data duplication detection algorithm corresponding to the data can be a data duplication detection algorithm that detects duplication by comparing the similarity of virtual machine image files.

[0082] As another example, considering that the characteristics of data whose source is a file system are a unified file access interface, a clear directory structure, and sophisticated file attribute management, which allow users to easily perform operations such as creating, reading, writing, and deleting files, when the data source is a file system scenario, the data duplication checking algorithm corresponding to the data can be a duplication checking algorithm based on the file path and file content.

[0083] For example, when the data source is a cloud storage scenario, the data duplication checking algorithm corresponding to the data may be a hash duplication checking algorithm. When the data source is a blockchain scenario, the data duplication checking algorithm corresponding to the data may be a Merkle tree-based duplication checking algorithm, which will not be described in detail here.

[0084] For ease of understanding, the following is an explanation of the data duplication checking algorithm determination process provided by this application through a specific embodiment. Figure 2 , Figure 2 A schematic diagram of another data duplication checking algorithm determination process provided in an embodiment of the present application, the process includes the following steps:

[0085] S201: Input the data within the statistical period into a pre-trained data feature analysis model. The data feature analysis model performs feature analysis on the data in preset dimensions to determine the feature values ​​of the data corresponding to each preset dimension. The preset dimensions include at least one of data volume, data type, data update frequency, and data security. The source of the data is determined based on the feature values ​​of the data corresponding to each preset dimension.

[0086] S202: According to the pre-saved correspondence between the data source and the data duplication checking algorithm, the data duplication checking algorithm corresponding to the data source of the data is determined as the data duplication checking algorithm applicable to the data.

[0087] In an embodiment of the present application, data can be input into a pre-trained data feature analysis model, and the data source of the data can be obtained by performing feature analysis on the data. The data duplication checking algorithm corresponding to the data source of the data can be determined as a data duplication checking algorithm suitable for the data based on the correspondence between the pre-saved data source and the data duplication checking algorithm. Based on this, it is possible to select a suitable data duplication checking algorithm for the data, and subsequently, based on the data duplication checking algorithm, data duplication checking operations can be performed efficiently and accurately, thereby improving data processing efficiency and accuracy, and improving the quality and accuracy of the data.

[0088] Embodiment 2:

[0089] For ease of understanding, the following is an explanation of the data duplication checking algorithm determination process provided by this application through a specific embodiment. Figure 3 , Figure 3 A schematic diagram of another data duplication checking algorithm determination process provided in an embodiment of the present application, the process includes the following steps:

[0090] S300: training data feature analysis model.

[0091] Specifically, the process of training the data feature analysis model includes: obtaining any sample data in the sample set, the sample data corresponding to a sample data source; performing feature analysis on the sample data through the data feature analysis model to be trained, and outputting the identification data source of the sample data; and training the data feature analysis model to be trained according to the sample data source and the identification data source.

[0092] S301: Input the data within the statistical period into a pre-trained data feature analysis model. The data feature analysis model performs feature analysis on the data in preset dimensions to determine the feature values ​​of the data corresponding to each preset dimension. The preset dimensions include at least one of data volume, data type, data update frequency, and data security. The source of the data is determined based on the feature values ​​of the data corresponding to each preset dimension.

[0093] The data may be structured data presented in a table form from a database, for example, product information of an e-commerce website, user order information, etc. The data may also be unstructured data presented in a file form from a file system, for example, documents, pictures, videos, etc., which are not specifically limited in this application.

[0094] S302: According to the pre-saved correspondence between the data source and the data duplication checking algorithm, the data duplication checking algorithm corresponding to the data source of the data is determined as the data duplication checking algorithm applicable to the data.

[0095] Embodiment 3:

[0096] Based on the same technical concept, on the basis of the above embodiments, the present application provides a data duplication checking algorithm determination device, Figure 4 A schematic diagram of a data duplication checking algorithm determining device structure provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the device comprises:

[0097] Analysis module 401, used for inputting data within a statistical period into a pre-trained data feature analysis model, wherein the data feature analysis model performs feature analysis on the data and outputs the data source of the data;

[0098] The determination module 402 is used to determine the data duplication checking algorithm corresponding to the data source of the data as the data duplication checking algorithm applicable to the data according to the pre-saved correspondence between the data source and the data duplication checking algorithm.

[0099] In a possible implementation, the analysis module 401 is specifically configured to:

[0100] Performing feature analysis on the data in preset dimensions to determine feature values ​​of the data corresponding to each preset dimension; wherein the preset dimensions include at least one of data volume, data type, data update frequency, and data security;

[0101] The data source of the data is determined according to the characteristic value of each preset dimension corresponding to the data.

[0102] In a possible implementation, the analysis module 401 is specifically configured to:

[0103] When the preset dimension includes data update frequency, for each sub-data in the data sorted according to the statistical time, by identifying whether the value of the target field of the sub-data is the same as the value of the previous sub-data at the target field, the number of times the sub-data at the target field is changed is counted;

[0104] The data update frequency of the data is determined according to the number of times and the total duration of the statistical period.

[0105] In a possible implementation, the analysis module 401 is specifically configured to:

[0106] When the preset dimension includes data security, an encryption method used for the data is identified, and based on the encryption method, the data security of the data is determined.

[0107] In a possible implementation, the device further includes:

[0108] The training module 403 is used to obtain any sample data in the sample set, and the sample data corresponds to a sample data source; perform feature analysis on the sample data through the data feature analysis model to be trained, and output the identification data source of the sample data; and train the data feature analysis model to be trained according to the sample data source and the identification data source.

[0109] Embodiment 4:

[0110] Based on the same technical concept, the present application also provides an electronic device, Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, it includes: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504;

[0111] The memory 503 stores a computer program, and when the program is executed by the processor 501, the processor 501 implements any of the above embodiments. Since the principle of the electronic device to solve the problem is similar to the method for determining the data duplication detection algorithm, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0112] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0113] The communication interface 502 is used for communication between the electronic device and other devices.

[0114] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0115] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0116] Embodiment 5:

[0117] Based on the same technical concept, an embodiment of the present application provides a computer-readable storage medium, in which a computer program executable by an electronic device is stored, and when the program is run on the electronic device, the electronic device implements any of the above embodiments when executing. Since the principle of solving the problem by the computer-readable storage medium is similar to the method for determining the data duplication checking algorithm, the implementation of the computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0118] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc., optical storage such as CD, DVD, BD, HVD, etc., and semiconductor storage such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD), etc.

[0119] Based on the same technical concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes any of the above embodiments. Since the principle of solving the problem by the above computer program product is similar to the method for determining the data duplication detection algorithm, the implementation of the above computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0120] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for determining a data duplication checking algorithm, characterized in that: The method comprises: Inputting the data within the statistical period into a pre-trained data feature analysis model, the data feature analysis model performs feature analysis on the data and outputs the data source of the data; According to the pre-saved correspondence between the data source and the data duplication checking algorithm, the data duplication checking algorithm corresponding to the data source of the data is determined as the data duplication checking algorithm applicable to the data.

2. The method according to claim 1, characterized in that The data feature analysis model performs feature analysis on the data and outputs the data source of the data, including: Performing feature analysis on the data in preset dimensions to determine feature values ​​of the data corresponding to each preset dimension; wherein the preset dimensions include at least one of data volume, data type, data update frequency, and data security; The data source of the data is determined according to the characteristic value of each preset dimension corresponding to the data.

3. The method according to claim 2, characterized in that The performing feature analysis on the data in the preset dimension to determine the feature value of the data corresponding to each preset dimension includes: When the preset dimension includes data update frequency, for each sub-data in the data sorted according to the statistical time, by identifying whether the value of the target field of the sub-data is the same as the value of the previous sub-data at the target field, the number of times the sub-data at the target field is changed is counted; The data update frequency of the data is determined according to the number of times and the total duration of the statistical period.

4. The method according to claim 2, characterized in that: The performing feature analysis on the data in the preset dimension to determine the feature value of the data corresponding to each preset dimension includes: When the preset dimension includes data security, an encryption method used for the data is identified, and based on the encryption method, the data security of the data is determined.

5. The method according to any one of claims 1 to 4, characterized in that: The training process of the data feature analysis model includes: Obtain any sample data in the sample set, the sample data corresponding to the sample data source; Performing feature analysis on the sample data using a data feature analysis model to be trained, and outputting an identification data source of the sample data; The data feature analysis model to be trained is trained according to the sample data source and the identification data source.

6. A device for determining a data duplication checking algorithm, characterized in that: The device comprises: An analysis module, used for inputting data within a statistical period into a pre-trained data feature analysis model, wherein the data feature analysis model performs feature analysis on the data and outputs the data source of the data; The determination module is used to determine the data duplication checking algorithm corresponding to the data source of the data as the data duplication checking algorithm applicable to the data according to the pre-saved correspondence between the data source and the data duplication checking algorithm.

7. The device according to claim 6, characterized in that The analysis module is specifically used for: Performing feature analysis on the data in preset dimensions to determine feature values ​​of the data corresponding to each preset dimension; wherein the preset dimensions include at least one of data volume, data type, data update frequency, and data security; The data source of the data is determined according to the characteristic value of each preset dimension corresponding to the data.

8. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is configured to implement the steps of the method according to any one of claims 1 to 5 when executing a computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 5.

Images