Computer data analysis processing method based on big data

Abstract

The invention discloses a computer data analysis processing method based on big data, and belongs to the technical field of data processing, and the method specifically comprises the following steps: collecting all storage data in a computer space, screening out storage data greater than a preset byte quantity, and marking the storage data as undetermined storage data; dividing the to-be-stored data into text data and video data, performing duplicate removal processing on the text data and the video data, and screening out repeated redundant text and video data; the evaluation indexes include but not limited to the number of days from last reading to today, the reading frequency and the byte size, the evaluation indexes are converted into extremely large indexes, the score F of each piece of to-be-stored data is calculated based on a superior and inferior solution distance method, and the score F of each piece of to-be-stored data is calculated; marking a plurality of pieces of undetermined storage data with top scores as recommended cleaning data; according to the method, large-scale rapid analysis of the duplicated data of the computer storage space is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a computer data analysis and processing method based on big data. Background Art

[0002] In today's information age, with the rapid development of technologies such as the Internet, the Internet of Things, and cloud computing, the speed of data generation, storage, and transmission has increased exponentially. Computers accumulate a large amount of data resources during operation, and these data contain valuable information and knowledge, which are of great value to individuals, enterprises, and society. However, the challenge that comes with it is how to process and analyze these huge data sets in order to extract useful information from them and transform them into effective information.

[0003] Moreover, with limited computing resources, how to effectively process large-scale data sets is a technical challenge. Identifying and removing duplicates in massive data is a basic and critical task, which can increase the value of data, reduce the waste of storage space, and provide a more accurate data foundation for subsequent analysis work. This requires not only the efficiency of the algorithm, but also its scalability and fault tolerance.

[0004] Therefore, it is very necessary to develop a computer data analysis and processing method based on big data that can automatically and accurately process various types of data and improve the efficiency and accuracy of data processing. Summary of the invention

[0005] The purpose of the present invention is to provide a computer data analysis and processing method based on big data to solve the following technical problems:

[0006] With limited computing resources, how to effectively process large-scale data sets, identify and remove duplicates, increase the value of data, and reduce the waste of storage space.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] A computer data analysis and processing method based on big data comprises the following steps:

[0009] Collect all the storage data in the computer space, filter out the storage data that is larger than a preset byte amount, and mark it as pending storage data;

[0010] The pending stored data is divided into text data and video data, and the text data and the video data are deduplicated respectively to filter out duplicate and redundant text and video data;

[0011] Obtain evaluation indicators of the pending storage data after deduplication, the evaluation indicators include but are not limited to the number of days since the last reading, the reading frequency and the byte size, convert the evaluation indicators into extremely large indicators, calculate the score F of each pending storage data based on the superior and inferior solution distance method, and mark several pending storage data with higher scores as recommended cleaning data.

[0012] As a further solution of the present invention: directly mark the number of days since today and the byte size as the maximum index, and mark the inverse of the reading frequency as the maximum index.

[0013] As a further solution of the present invention: the process of deduplication of text data is:

[0014] Arrange the characters in the text data according to the set text format, keep the number of characters in each line uniform, assign corresponding numbering sequences to all characters in the text data, including but not limited to Chinese characters, numbers and English letters, obtain the maximum number value N, convert the characters into corresponding grayscale values ​​under N grayscale levels with the corresponding numbering sequence values ​​for display, convert the text data into a grayscale image, divide the text data with a difference in the number of characters lower than a preset percentage into the same set, compare the grayscale images corresponding to the text data in the same set, obtain the grayscale histogram of each text data respectively, compare the texts pairwise, count the sum of the difference values ​​of all grayscale levels in the two grayscale histograms, and mark the text data corresponding to the two grayscale histograms whose sum of the difference is lower than a preset threshold as duplicate text.

[0015] As a further solution of the present invention: when the bytes of any two text data are greater than a preset threshold, the grayscale image of the pair of text data is divided into a plurality of regions, and each region is compared in turn.

[0016] As a further solution of the present invention: the deduplication process of video data is:

[0017] The video data with a byte amount difference less than a preset threshold are divided into the same set. In any set, characteristic indicators of each video data are obtained, and the characteristic indicators include frame rate, resolution and playback time. The resolution is the number of pixels on the short side of the video frame. The video data are compared in pairs, and the characteristic indicators of one of the video data are marked as a1, b1, and c1 in sequence, and the characteristic indicators of the other video data are marked as a2, b2, and c2 in sequence. The characteristic indicators are normalized and used as geometric parameters of the trapezoidal image. The frame rate is used as With S1 as the upper base, resolution as the lower base, and playback time as the height, generate two corresponding trapezoidal images, calculate the areas of the two trapezoidal images, which are S1=(a1+b1)c1 / 2 and S2=(a2+b2)c2 / 2 respectively, superimpose the two trapezoidal images in three-dimensional space, select the superposition state when the superposition area of ​​the two trapezoidal images is the largest, obtain the area S0 of the superposition area at this time, calculate the value of 2S0 / (S1+S2), and mark the value as the similarity of the two video data. When the similarity is greater than 90%, it is a repeated video.

[0018] As a further solution of the present invention: the process of calculating the score is:

[0019] The number of pending storage data is marked as n, the number of evaluation indicators is marked as m, and the extremely large indicator X is marked as ij Convert to normalized index Z ij , i∈n, j∈m, normalized index Z ij With very large index X ij The conversion formula is:

[0020]

[0021] Get the maximum and minimum values ​​of any evaluation index in all pending stored data and mark them as Z + j and Z - j , calculate each evaluation index of any pending storage data and the corresponding maximum value Z + The Euclidean distance D i + , and the corresponding minimum value Z - The Euclidean distance D i - ; Then the unnormalized score of any pending stored data is:

[0022]

[0023] The score f i Perform normalization to obtain the final score F of any pending stored data i , the formula is:

[0024]

[0025] As a further solution of the present invention: Euclidean distance D i + and Euclidean distance D i - The calculation formula is:

[0026]

[0027] Beneficial effects of the present invention:

[0028] The present invention utilizes grayscale image comparison and trapezoidal image similarity calculation methods to deduplicate text and video data, and can effectively identify duplicate or similar data items, thereby ensuring the uniqueness and accuracy of the data; and based on multi-dimensional evaluation indicators, and through the superior-inferior solution distance method, these indicators are integrated to calculate the score, so that the importance assessment of the data is more comprehensive and objective; the function of automatically marking and recommending cleaned data helps users or system administrators to identify data that occupies a large amount of storage space and is not frequently used, and then make reasonable storage resource management decisions to release valuable storage space; the present invention is applicable to various types of big data environments, and can process and analyze both text data and video data, and has strong adaptability and universality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below in conjunction with the accompanying drawings.

[0030] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0032] See also Figure 1 As shown, the present invention is a computer data analysis and processing method based on big data, comprising the following steps:

[0033] Collect all the storage data in the computer space, filter out the storage data that is larger than a preset byte amount, and mark it as pending storage data;

[0034] The pending stored data is divided into text data and video data, and the text data and the video data are deduplicated respectively to filter out duplicate and redundant text and video data;

[0035] Obtain evaluation indicators of the pending storage data after deduplication, the evaluation indicators include but are not limited to the number of days since the last reading, the reading frequency and the byte size, convert the evaluation indicators into extremely large indicators, calculate the score F of each pending storage data based on the superior and inferior solution distance method, and mark several pending storage data with higher scores as recommended cleaning data.

[0036] The present invention collects all storage data in the computer space, screens out storage data larger than a preset byte amount, and marks it as pending storage data. The pending storage data is divided into text data and video data, and deduplication is performed. The evaluation index of the pending storage data after deduplication is obtained, including the number of days since the last reading, the reading frequency and the byte size, and these evaluation indexes are converted into extremely large indexes. The score F of each pending storage data is calculated based on the superior and inferior solution distance method, and several pending storage data with the highest scores are marked as recommended cleaning data.

[0037] During the text data deduplication process, the present invention converts the text data into grayscale images, compares the grayscale images of the text data in the same set, and counts the sum of the differences of all grayscale levels in the two grayscale histograms. When the sum of the differences is lower than a preset threshold, the text data is marked as duplicate text.

[0038] In the process of video data deduplication, firstly, the video data with byte difference less than the preset threshold are divided into the same set. Then, in this set, the characteristic indicators of each video data (including frame rate, resolution and playback time) are obtained, and these characteristic indicators are normalized as the geometric parameters of the trapezoidal image to generate the corresponding trapezoidal image. Finally, the similarity of the two trapezoidal images is calculated. When the similarity is greater than a certain value, these video data are marked as duplicate videos.

[0039] In the process of calculating the score, the method first converts the extremely large index into a standardized index, then calculates the Euclidean distance between each evaluation index of any pending storage data and the corresponding maximum and minimum values, and calculates the unnormalized score of the pending storage data based on these Euclidean distances. Finally, the score is normalized to obtain the final score of the pending storage data.

[0040] In summary, the present invention effectively processes and analyzes computer data through big data technology, thereby improving the efficiency and accuracy of data processing.

[0041] In another preferred embodiment of the present invention, the number of days since today and the byte size are directly marked as extreme large indicators, and the inverse of the reading frequency is marked as the extreme large indicator.

[0042] In another preferred embodiment of the present invention, the process of deduplication of text data is as follows:

[0043] Arrange the characters in the text data according to the set text format, keep the number of characters in each line uniform, assign corresponding numbering sequences to all characters in the text data, including but not limited to Chinese characters, numbers and English letters, obtain the maximum number value N, convert the characters into corresponding grayscale values ​​under N grayscale levels with the corresponding numbering sequence values ​​for display, convert the text data into a grayscale image, divide the text data with a difference in the number of characters lower than a preset percentage into the same set, compare the grayscale images corresponding to the text data in the same set, obtain the grayscale histogram of each text data respectively, compare the texts pairwise, count the sum of the difference values ​​of all grayscale levels in the two grayscale histograms, and mark the text data corresponding to the two grayscale histograms whose sum of the difference is lower than a preset threshold as duplicate text.

[0044] In another preferred embodiment of the present invention, when the bytes of any two text data are greater than a preset threshold, the grayscale image of the pair of text data is divided into a plurality of regions, and each region is compared in turn.

[0045] In another preferred embodiment of the present invention, the deduplication process of video data is as follows:

[0046] The video data with a byte amount difference less than a preset threshold are divided into the same set. In any set, characteristic indicators of each video data are obtained, and the characteristic indicators include frame rate, resolution and playback time. The resolution is the number of pixels on the short side of the video frame. The video data are compared in pairs, and the characteristic indicators of one of the video data are marked as a1, b1, and c1 in sequence, and the characteristic indicators of the other video data are marked as a2, b2, and c2 in sequence. The characteristic indicators are normalized and used as geometric parameters of the trapezoidal image. The frame rate is used as With S1 as the upper base, resolution as the lower base, and playback time as the height, generate two corresponding trapezoidal images, calculate the areas of the two trapezoidal images, which are S1=(a1+b1)c1 / 2 and S2=(a2+b2)c2 / 2 respectively, superimpose the two trapezoidal images in three-dimensional space, select the superposition state when the superposition area of ​​the two trapezoidal images is the largest, obtain the area S0 of the superposition area at this time, calculate the value of 2S0 / (S1+S2), and mark the value as the similarity of the two video data. When the similarity is greater than 90%, it is a repeated video.

[0047] In another preferred embodiment of the present invention, the process of calculating the score is:

[0048] The number of pending storage data is marked as n, the number of evaluation indicators is marked as m, and the extremely large indicator X is marked as ij Convert to normalized index Z ij , i∈n, j∈m, normalized index Z ij With very large index Xij The conversion formula is:

[0049]

[0050] Get the maximum and minimum values ​​of any evaluation index in all pending stored data and mark them as Z + j and Z - j , calculate each evaluation index of any pending storage data and the corresponding maximum value Z + The Euclidean distance D i + , and the corresponding minimum value Z - The Euclidean distance D i - ; Then the unnormalized score of any pending stored data is:

[0051]

[0052] The score f i Perform normalization to obtain the final score F of any pending stored data i , the formula is:

[0053]

[0054] In another preferred embodiment of the present invention, the Euclidean distance D i + and Euclidean distance D i - The calculation formula is:

[0055]

[0056] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A computer data analysis and processing method based on big data, characterized in that: The following steps are involved: Collect all the storage data in the computer space, filter out the storage data that is larger than a preset byte amount, and mark it as pending storage data; The pending stored data is divided into text data and video data, and the text data and the video data are deduplicated respectively to filter out duplicate and redundant text data and video data; Obtain evaluation indicators of the pending storage data after deduplication, the evaluation indicators include but are not limited to the number of days since the last reading, the reading frequency and the byte size, convert the evaluation indicators into extremely large indicators, calculate the score F of each pending storage data based on the superior and inferior solution distance method, and mark several pending storage data with higher scores as recommended cleaning data.

2. A computer data analysis and processing method based on big data according to claim 1, characterized in that: Directly mark the number of days since the last read and the byte size as extremely large indicators, and mark the inverse of the reading frequency as an extremely large indicator.

3. The computer data analysis and processing method based on big data according to claim 1 is characterized in that: The process of deduplication of text data is as follows: Arrange the characters in the text data according to the set text format, keep the number of characters in each line uniform, assign corresponding numbering sequences to all characters in the text data, including but not limited to Chinese characters, numbers and English letters, obtain the maximum number value N, convert the characters into corresponding grayscale values ​​under N grayscale levels with the corresponding numbering sequence values ​​for display, convert the text data into a grayscale image, divide the text data with a difference in the number of characters lower than a preset percentage into the same set, compare the grayscale images corresponding to the text data in the same set, obtain the grayscale histogram of each text data respectively, compare the text data pairwise, count the sum of the difference values ​​of all grayscale levels in the two grayscale histograms, and mark the text data corresponding to the two grayscale histograms with the sum of the difference lower than a preset threshold as duplicate text.

4. A computer data analysis and processing method based on big data according to claim 3, characterized in that: When the bytes of any two text data are greater than a preset threshold, the grayscale image of the pair of text data is divided into a number of regions, and each region is compared in turn.

5. The computer data analysis and processing method based on big data according to claim 1 is characterized in that: The deduplication process of video data is as follows: The video data with a byte amount difference less than a preset threshold are divided into the same set. In any set, characteristic indicators of each video data are obtained, and the characteristic indicators include frame rate, resolution and playback time. The resolution is the number of pixels on the short side of the video frame. The video data are compared in pairs, and the characteristic indicators of one of the video data are marked as a1, b1, and c1 in sequence, and the characteristic indicators of the other video data are marked as a2, b2, and c2 in sequence. The characteristic indicators are normalized and used as geometric parameters of the trapezoidal image. The frame rate is used as With S1 as the upper base, resolution as the lower base, and playback time as the height, generate two corresponding trapezoidal images, calculate the areas of the two trapezoidal images, which are S1=(a1+b1)c1 / 2 and S2=(a2+b2)c2 / 2 respectively, superimpose the two trapezoidal images in three-dimensional space, select the superposition state when the superposition area of ​​the two trapezoidal images is the largest, obtain the area S0 of the superposition area at this time, calculate the value of 2S0 / (S1+S2), and mark the value as the similarity of the two video data. When the similarity is greater than 90%, it is a repeated video.

6. The computer data analysis and processing method based on big data according to claim 1 is characterized in that: The process of calculating the score is: The number of pending storage data is marked as n, the number of evaluation indicators is marked as m, and the extremely large indicator X is marked as ij Convert to normalized index Z ij , i∈n, j∈m, normalized index Z ij With very large index X ij The conversion formula is: Get the maximum and minimum values ​​of any evaluation index in all pending stored data and mark them as Z + j and Z - j , calculate each evaluation index of any pending storage data and the corresponding maximum value Z + The Euclidean distance D i + , and the corresponding minimum value Z - The Euclidean distance D i - ; Then the unnormalized score of any pending stored data is: The score f i Perform normalization to obtain the final score F of any pending stored data i , the formula is:

7. A computer data analysis and processing method based on big data according to claim 6, characterized in that: Euclidean distance D i + and Euclidean distance D i - The calculation formula is:

Images