A data analysis method for digital resource usage construction

Abstract

The present invention discloses a data analysis method for digital resource usage and construction, comprising the following steps: A. categorizing data to be used according to content, and then performing data cleaning on each category of data; B. establishing an index table for the data processed in step A, and integrating the index table with a resource library index table by adding foreign keys; and C. performing a simulation operation on the integrated resource library, and updating the resource library index table integrated in step B based on the operation results. The present invention can overcome the shortcomings of existing technologies and increase the data update speed of digital resource libraries.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data analysis method for digital resource usage construction. Background Art

[0002] Digital resource libraries are widely used databases across various industries. To ensure real-time data availability, digital resource libraries need to be updated regularly. However, the large amount of data required for each update results in a slow update process, hindering the ease of use of digital resource libraries. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a data analysis method for digital resource usage construction, which can solve the shortcomings of the existing technology and improve the data update speed of the digital resource library.

[0004] In order to solve the above technical problems, the technical solutions adopted by the present invention are as follows.

[0005] A data analysis method for digital resource usage construction includes the following steps:

[0006] A. Classify the data to be used according to its content, and then clean each type of data;

[0007] B. Create an index table for the data processed in step A and integrate the index table with the resource library index table by adding foreign keys;

[0008] C. Perform simulation operations on the integrated resource library and update the resource library index table integrated in step B according to the operation results.

[0009] Preferably, in step A, data cleaning includes the following steps:

[0010] A1. Extract the sensitive factors of each type of data;

[0011] A2. Cluster the sensitive factors based on similarity, and then assign the same priority to the sensitive factors in the same group based on the number of sensitive factors in each group.

[0012] A3. Delete data that does not contain sensitive factors;

[0013] A4. For data containing sensitive factors, group them according to the highest priority sensitive factor they contain;

[0014] A5. Delete duplicate data from each set of data.

[0015] A6. Perform a simulation on the remaining data, then swap the non-highest priority sensitive factors in the remaining data, perform another simulation, compare the two simulation results, and merge the data whose simulation results before and after the sensitive factor swap have a deviation less than the set threshold.

[0016] A7. Repeat step A6 until there is no more data that meets the merge criteria.

[0017] Preferably, in step A1, extracting the sensitivity factor of each type of data includes the following steps:

[0018] A11. Mark the data content. The number of tags for a piece of data should be greater than or equal to 2.

[0019] A12. Randomly replace the content at the marked positions of the data, use the test function to perform a test operation on the data before and after the replacement, and calculate the deviation between the two operation results.

[0020] A13. Repeat step A12, changing the data mark position before each execution of step A12 until the deviation exceeds the preset threshold or the number of repetitions reaches the preset number, ending the test operation and selecting the mark content with the largest deviation as the sensitive factor.

[0021] Preferably, in step B, establishing an index table of the data processed in step A includes the following steps:

[0022] B11. Establish a sensitive factor set for each sensitive factor contained in each data, and establish a correlation function between the sensitive factor set and the data;

[0023] B12. Create a two-level index table. The objects in the first-level index table are correlation functions, which are stored in a grouping manner. The correlation functions are grouped according to similarity. The objects in the second-level index table are sensitive factor sets, which are stored in a queue manner.

[0024] B13. When retrieving data, first search for a sensitive factor set that is the same and / or similar to the sensitive factor set of the target data through the second-level index table, then search for the association function related to the sensitive factors in the second-level index table through the first-level index table, and finally search for the target data through the association function in the group where the found association function is located.

[0025] Preferably, in step C, updating the resource library index table integrated in step B includes the following steps:

[0026] C1. Update the sensitive factor set based on the simulation results;

[0027] C2. Update the second-level index table according to the updated sensitive factor set.

[0028] The beneficial effects of the above technical solution are as follows: by extracting sensitive factors and using them as limiting parameters for data cleaning, the present invention effectively reduces the amount of data verification computation required during the data cleaning process, while simultaneously improving the accuracy of data cleaning. Furthermore, during the index table creation process, a two-level index structure comprising correlation functions and sensitive factors is established, thereby improving data retrieval efficiency. Furthermore, each time the index table is updated, only the second-level index table containing the set of sensitive factors needs to be updated, reducing the update computational effort. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flow chart of a specific embodiment of the present invention. DETAILED DESCRIPTION

[0030] Reference Figure 1 , a specific embodiment of the present invention includes the following steps:

[0031] A. Classify the data to be used according to its content, and then clean each type of data;

[0032] B. Create an index table for the data processed in step A and integrate the index table with the resource library index table by adding foreign keys;

[0033] C. Perform simulation operations on the integrated resource library and update the resource library index table integrated in step B according to the operation results.

[0034] In step A, data cleaning includes the following steps:

[0035] A1. Extract the sensitive factors of each type of data;

[0036] A2. Cluster the sensitive factors based on similarity, and then assign the same priority to the sensitive factors in the same group based on the number of sensitive factors in each group.

[0037] A3. Delete data that does not contain sensitive factors;

[0038] A4. For data containing sensitive factors, group them according to the highest priority sensitive factor they contain;

[0039] A5. Delete duplicate data from each set of data.

[0040] A6. Perform a simulation on the remaining data, then swap the non-highest priority sensitive factors in the remaining data, perform another simulation, compare the two simulation results, and merge the data whose simulation results before and after the sensitive factor swap have a deviation less than the set threshold.

[0041] A7. Repeat step A6 until there is no more data that meets the merge criteria.

[0042] In step A1, extracting the sensitivity factor of each type of data includes the following steps:

[0043] A11. Mark the data content. The number of tags for a piece of data should be greater than or equal to 2.

[0044] A12. Randomly replace the content at the marked positions of the data, use the test function to perform a test operation on the data before and after the replacement, and calculate the deviation between the two operation results.

[0045] A13. Repeat step A12, changing the data mark position before each execution of step A12 until the deviation exceeds the preset threshold or the number of repetitions reaches the preset number, ending the test operation and selecting the mark content with the largest deviation as the sensitive factor.

[0046] In step B, establishing an index table of the data processed in step A includes the following steps:

[0047] B11. Establish a sensitive factor set for each sensitive factor contained in each data, and establish a correlation function between the sensitive factor set and the data;

[0048] B12. Create a two-level index table. The objects in the first-level index table are correlation functions, which are stored in a grouping manner. The correlation functions are grouped according to similarity. The objects in the second-level index table are sensitive factor sets, which are stored in a queue manner.

[0049] B13. When retrieving data, first search for a sensitive factor set that is the same and / or similar to the sensitive factor set of the target data through the second-level index table, then search for the association function related to the sensitive factors in the second-level index table through the first-level index table, and finally search for the target data through the association function in the group where the found association function is located.

[0050] In step C, updating the resource library index table integrated in step B includes the following steps: C1, updating the sensitive factor set according to the simulation operation result;

[0051] C2. Update the second-level index table according to the updated sensitive factor set.

[0052] The present invention effectively increases the data updating speed of the digital resource library by improving the data cleaning process.

[0053] In the description of the present invention, it should be understood that the terms "longitudinal", "transverse", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" and the like to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings and are only for the convenience of describing the present invention, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0054] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A data analysis method for digital resource usage construction, characterized in that The following steps are involved: A. Classify the data to be used according to the content, and then clean each type of data. The following steps are included: A1. Extract the sensitive factors of each type of data. The following steps are included: A11. Mark the data content. The number of tags for a piece of data should be greater than or equal to 2. A12. Randomly replace the content at the marked positions of the data, use the test function to perform a test operation on the data before and after the replacement, and calculate the deviation between the two operation results. A13. Repeat step A12, replacing the data marker position before each step A12 execution, until the deviation exceeds a preset threshold or the number of repetitions reaches a preset number. Terminate the test operation and select the marker content with the largest deviation as the sensitivity factor. A2. Cluster the sensitive factors based on similarity, and then assign the same priority to the sensitive factors in the same group based on the number of sensitive factors in each group. A3. Delete data that does not contain sensitive factors; A4. For data containing sensitive factors, group them according to the highest priority sensitive factor they contain; A5. Delete duplicate data from each set of data. A6. Perform a simulation on the remaining data, then swap the non-highest priority sensitive factors in the remaining data, perform another simulation, compare the two simulation results, and merge the data whose simulation results before and after the sensitive factor swap have a deviation less than the set threshold. A7. Repeat step A6 until no more data meets the merge criteria. B. Create an index table for the data processed in step A and integrate the index table with the resource library index table by adding foreign keys; The creation of the index table includes the following steps: B11. Establish a sensitive factor set for each sensitive factor contained in each data, and establish a correlation function between the sensitive factor set and the data; B12. Create a two-level index table. The objects in the first-level index table are correlation functions, which are stored in a grouping manner. The correlation functions are grouped according to similarity. The objects in the second-level index table are sensitive factor sets, which are stored in a queue manner. B13. When retrieving data, first search for a set of sensitive factors that are identical and / or similar to the sensitive factor set of the target data through the second-level index table, then search for the correlation function related to the sensitive factors in the second-level index table through the first-level index table, and finally search for the target data through the correlation function in the group where the found correlation function is located. C. Perform simulation calculation on the integrated resource library and update the resource library index table integrated in step B according to the calculation result, including the following steps: C1. Update the sensitive factor set based on the simulation results; C2. Update the second-level index table according to the updated sensitive factor set.

Images