A computer data analysis and processing method based on big data

Abstract

The application discloses a computer data analysis processing method based on big data and belongs to the technical field of data processing, and specifically comprises the following steps: collecting all storage data in computer space, screening out storage data greater than a preset byte amount and marking the storage data as pending storage data; dividing the pending storage data into text data and video data, respectively performing deduplication processing on the text data and the video data, and screening out redundant text and video data; obtaining evaluation indexes of the deduplicated pending storage data, wherein the evaluation indexes include but are not limited to the number of days from the last reading to today, reading frequency and byte size, converting the evaluation indexes into maximum indexes, calculating the score F of each pending storage data based on the distance between superior and inferior solutions, and marking a plurality of pending storage data with higher scores as recommended cleaning data; and the application realizes large-scale and rapid analysis of repeated data in the computer storage space.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically to a computer data analysis and processing method based on big data. Background Technology

[0002] In today's information age, with the rapid development of technologies such as the Internet, the Internet of Things, and cloud computing, the speed of data generation, storage, and transmission is increasing exponentially. Computers accumulate vast amounts of data resources during operation, containing valuable information and knowledge that is of significant value to individuals, businesses, and society. However, the ensuing challenge is how to process and analyze these massive datasets to extract useful information and transform it into effective tools.

[0003] Moreover, effectively processing large-scale datasets with limited computing resources is a technical challenge. Identifying and removing duplicates from massive datasets is a fundamental and crucial task. This can increase the value of the data, reduce the waste of storage space, and provide a more accurate data foundation for subsequent analysis. This requires not only efficient algorithms but also scalability and fault tolerance.

[0004] Therefore, it is essential to develop a computer data analysis and processing method based on big data that can automatically and accurately process various types of data and improve the efficiency and accuracy of data processing. Summary of the Invention

[0005] The purpose of this invention is to provide a computer data analysis and processing method based on big data, and to solve the following technical problems:

[0006] How can we effectively process large-scale datasets, identify and remove duplicates, increase the value of data, and reduce the waste of storage space under limited computing resources?

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] A computer data analysis and processing method based on big data includes the following steps:

[0009] Collect all stored data in the computer space, filter out stored data larger than a preset number of bytes, and mark them as pending stored data;

[0010] The data to be stored is divided into text data and video data. The text data and video data are deduplicated separately to filter out duplicate and redundant text and video data.

[0011] The evaluation metrics for the deduplicated pending storage data are obtained. These evaluation metrics include, but are not limited to, the number of days since the last read, the read frequency, and the byte size. The evaluation metrics are converted into extremely large metrics. The score F of each pending storage data is calculated based on the superior-inferior solution distance method. The pending storage data with the highest scores are marked as recommended cleanup data.

[0012] As a further aspect of the present invention: the number of days since the last read and the size of the bytes are directly marked as extremely large indicators, and the reciprocal of the reading frequency is marked as an extremely large indicator.

[0013] As a further aspect of the present invention: the process of deduplicating text data is as follows:

[0014] The characters in the text data are arranged according to the set text format, and the number of characters in each line is kept uniform. All characters in the text data, including but not limited to Chinese characters, numbers, and English letters, are assigned corresponding number order. The maximum number value N is obtained. The characters are converted into grayscale values ​​corresponding to N grayscale levels according to their corresponding number order values ​​and displayed. The text data is converted into a grayscale image. Text data with a character quantity difference of less than a preset percentage are grouped into the same set. The grayscale images corresponding to the text data in the same set are compared, and the grayscale histogram of each text data is obtained. The texts are compared pairwise, and the sum of the differences of all grayscale levels in the two grayscale histograms is calculated. The text data corresponding to the two grayscale histograms with a sum of differences of less than a preset threshold are marked as duplicate text.

[0015] As a further aspect of the present invention: when the number of bytes of any two text data is greater than a preset threshold, the grayscale image of the pair of text data is divided into several regions, and each region is compared sequentially.

[0016] As a further aspect of the present invention: the deduplication process for video data is as follows:

[0017] Video data with a byte size difference less than a preset threshold are grouped into the same set. Within any set, feature metrics are obtained for each video data point, including frame rate, resolution, and playback duration. Resolution is the number of pixels on the shorter side of the video frame. The video data are compared pairwise, and the feature metrics of one video data point are sequentially labeled a1, b1, c1, and the feature metrics of the other video data point are sequentially labeled a2, b2, c2. The feature metrics are normalized and used as geometric parameters of a trapezoidal image. The frame rate is then used as... Using the resolution as the bottom base and the playback duration as the height, two trapezoidal images are generated. The areas of the two trapezoidal images are calculated as S1 = (a1 + b1)c1 / 2 and S2 = (a2 + b2)c2 / 2, respectively. The two trapezoidal images are superimposed in three-dimensional space. The superposition state when the superposition area of ​​the two trapezoidal images is the maximum is selected, and the area S0 of the superposition region at this time is obtained. The value of 2S0 / (S1 + S2) is calculated and marked as the similarity between the two video data. When the similarity is greater than 90%, it is a duplicate video.

[0018] As a further aspect of the present invention: the process of calculating the score is as follows:

[0019] Let n be the number of data to be stored, m be the number of evaluation metrics, and X be the largest metric. ij Convert to standardized index Z ij , i∈n, j∈m, standardized index Z ij With the extremely large indicator X ij The conversion formula is:

[0020]

[0021] Obtain the maximum and minimum values ​​of any evaluation metric from all pending stored data and label them as Z. + j and Z - j Calculate the value of each evaluation index and its corresponding maximum value Z for any undetermined stored data. + Euclidean distance D i + and the corresponding minimum value Z - Euclidean distance D i - Then the unnormalized score of any undetermined stored data is:

[0022]

[0023] The score f i Normalization is performed to obtain the final score F for any undetermined stored data. i The formula is:

[0024]

[0025] As a further aspect of the present invention: Euclidean distance D i + Euclidean distance D i - The calculation formula is:

[0026]

[0027] The beneficial effects of this invention are:

[0028] This invention utilizes methods such as grayscale image comparison and trapezoidal image similarity calculation to deduplicate text and video data, effectively identifying duplicate or similar data items to ensure data uniqueness and accuracy. Based on multi-dimensional evaluation indicators, and using the superior-inferiority distance method to synthesize these indicators and calculate scores, the invention provides a more comprehensive and objective assessment of data importance. The automatic tagging and recommendation function helps users or system administrators identify data that occupies significant storage space but is infrequently used, enabling them to make reasonable storage resource management decisions and free up valuable storage space. This invention is applicable to various types of big data environments, processing and analyzing both text and video data, demonstrating strong adaptability and universality. Attached Figure Description

[0029] The invention will now be further described with reference to the accompanying drawings.

[0030] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see Figure 1 As shown, this invention is a computer data analysis and processing method based on big data, comprising the following steps:

[0033] Collect all stored data in the computer space, filter out stored data larger than a preset number of bytes, and mark them as pending stored data;

[0034] The data to be stored is divided into text data and video data. The text data and video data are deduplicated separately to filter out duplicate and redundant text and video data.

[0035] The evaluation metrics for the deduplicated pending storage data are obtained. These evaluation metrics include, but are not limited to, the number of days since the last read, the read frequency, and the byte size. The evaluation metrics are converted into extremely large metrics. The score F of each pending storage data is calculated based on the superior-inferior solution distance method. The pending storage data with the highest scores are marked as recommended cleanup data.

[0036] This invention collects all stored data in computer space, filters out stored data larger than a preset byte size, and marks them as pending storage data. The pending storage data is divided into text data and video data, and deduplication is performed. Evaluation metrics for the deduplicated pending storage data are obtained, including the number of days since the last read, read frequency, and byte size, and these metrics are converted into extremely large metrics. A score F is calculated for each pending storage data based on the superior-inferior solution distance method, and the top-scoring pending storage data are marked as recommended data for cleaning.

[0037] In the process of deduplicating text data, this invention converts the text data into grayscale images and compares the grayscale images of text data in the same set. It calculates the sum of the differences between all grayscale levels in the two grayscale histograms. When the sum of the differences is lower than a preset threshold, these text data are marked as duplicate text.

[0038] In the video data deduplication process, video data with a byte size difference less than a preset threshold are first grouped into the same set. Then, within this set, the feature metrics (including frame rate, resolution, and playback duration) of each video data are obtained, and these feature metrics are normalized and used as the geometric parameters for a trapezoidal image, generating the corresponding trapezoidal image. Finally, the similarity between two trapezoidal images is calculated; when the similarity exceeds a certain value, these video data are marked as duplicate videos.

[0039] In calculating the score, this method first converts the extremely large metrics into standardized metrics. Then, it calculates the Euclidean distance between each evaluation metric of any undetermined storage data and its corresponding maximum and minimum values, and calculates the unnormalized score of the undetermined storage data based on these Euclidean distances. Finally, it normalizes this score to obtain the final score of the undetermined storage data.

[0040] In summary, this invention effectively processes and analyzes computer data through big data technology, improving the efficiency and accuracy of data processing.

[0041] In another preferred embodiment of the present invention, the number of days since the last read and the size of the bytes are directly marked as extremely large indicators, and the reciprocal of the reading frequency is marked as an extremely large indicator.

[0042] In another preferred embodiment of the present invention, the process of deduplicating text data is as follows:

[0043] The characters in the text data are arranged according to the set text format, and the number of characters in each line is kept uniform. All characters in the text data, including but not limited to Chinese characters, numbers, and English letters, are assigned corresponding number order. The maximum number value N is obtained. The characters are converted into grayscale values ​​corresponding to N grayscale levels according to their corresponding number order values ​​and displayed. The text data is converted into a grayscale image. Text data with a character quantity difference of less than a preset percentage are grouped into the same set. The grayscale images corresponding to the text data in the same set are compared, and the grayscale histogram of each text data is obtained. The texts are compared pairwise, and the sum of the differences of all grayscale levels in the two grayscale histograms is calculated. The text data corresponding to the two grayscale histograms with a sum of differences of less than a preset threshold are marked as duplicate text.

[0044] In another preferred embodiment of the present invention, when the number of bytes of any two text data is greater than a preset threshold, the grayscale image of the pair of text data is divided into several regions, and each region is compared in turn.

[0045] In another preferred embodiment of the present invention, the deduplication process for video data is as follows:

[0046] Video data with a byte size difference less than a preset threshold are grouped into the same set. Within any set, feature metrics are obtained for each video data point, including frame rate, resolution, and playback duration. Resolution is the number of pixels on the shorter side of the video frame. The video data are compared pairwise, and the feature metrics of one video data point are sequentially labeled a1, b1, c1, and the feature metrics of the other video data point are sequentially labeled a2, b2, c2. The feature metrics are normalized and used as geometric parameters of a trapezoidal image. The frame rate is then used as... Using the resolution as the bottom base and the playback duration as the height, two trapezoidal images are generated. The areas of the two trapezoidal images are calculated as S1 = (a1 + b1)c1 / 2 and S2 = (a2 + b2)c2 / 2, respectively. The two trapezoidal images are superimposed in three-dimensional space. The superposition state when the superposition area of ​​the two trapezoidal images is the maximum is selected, and the area S0 of the superposition region at this time is obtained. The value of 2S0 / (S1 + S2) is calculated and marked as the similarity between the two video data. When the similarity is greater than 90%, it is a duplicate video.

[0047] In another preferred embodiment of the present invention, the process of calculating the score is as follows:

[0048] Let n be the number of data to be stored, m be the number of evaluation metrics, and X be the largest metric. ij Convert to standardized index Z ij , i∈n, j∈m, standardized index Z ij With the extremely large indicator Xij The conversion formula is:

[0049]

[0050] Obtain the maximum and minimum values ​​of any evaluation metric from all pending stored data and label them as Z. + j and Z - j Calculate the value of each evaluation index and its corresponding maximum value Z for any undetermined stored data. + Euclidean distance D i + and the corresponding minimum value Z - Euclidean distance D i - Then the unnormalized score of any undetermined stored data is:

[0051]

[0052] The score f i Normalization is performed to obtain the final score F for any undetermined stored data. i The formula is:

[0053]

[0054] In another preferred embodiment of the invention, the Euclidean distance D i + Euclidean distance D i - The calculation formula is:

[0055]

[0056] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A big data based computer data analysis processing method, characterized by, The method comprises the following steps: Collect all stored data in the computer space, filter out the stored data larger than the preset byte amount, and mark as pending stored data; Divide the pending stored data into text data and video data, and perform deduplication processing on the text data and the video data respectively, and filter out redundant text data and video data; Obtain evaluation indexes of the deduplicated pending stored data, the evaluation indexes include but are not limited to the number of days from the last reading to today, the reading frequency and the byte size, convert the evaluation indexes into maximum indexes, calculate the score F of each pending stored data based on the distance between the superior and inferior solutions, and mark the pending stored data with higher scores as recommended cleaning data; The deduplication process of the video data is as follows: Divide the video data with a byte amount difference less than a preset threshold into the same set, obtain the feature indexes of each video data in any set, the feature indexes include the frame rate, the resolution and the playing time length, the resolution is the pixel number of the short side of the video frame, compare the video data two by two, mark the feature indexes of one video data as a1, b1 and c1 in turn, mark the feature indexes of the other video data as a2, b2 and c2 in turn, normalize the feature indexes, take the feature indexes as the geometric parameters of the trapezoidal image, take the frame rate as the upper base, the resolution as the lower base and the playing time length as the height respectively, generate corresponding two trapezoidal images, calculate the areas of the two trapezoidal images, which are S1=(a1+b1)c1 / 2 and S2=(a2+b2)c2 / 2 respectively, superimpose the two trapezoidal images in the three-dimensional space, select the superimposed state when the superimposed area of the two trapezoidal images is maximum, obtain the area S0 of the superimposed region at this time, calculate the value of 2S0 / (S1+S2), mark the value as the similarity of the two video data, and when the similarity is greater than 90%, the two video data are repeated video data.

2. The computer data analysis processing method based on big data according to claim 1, characterized in that, Directly mark the number of days from the last reading to today and the byte size as the maximum index, and mark the reciprocal of the reading frequency as the maximum index.

3. The computer data analysis processing method based on big data according to claim 1, characterized in that, The text data deduplication process is as follows: Arrange the characters in the text data according to the set line format, keep the number of characters in each line uniform, assign all characters including but not limited to Chinese characters, numbers and English letters in the text data with corresponding numbering sequences, obtain the maximum number N, convert the characters into corresponding gray values under N gray levels for display according to the corresponding numbering sequence values, convert the text data into a gray image, divide the text data with a character number difference lower than a preset percentage into the same set, compare the gray images of the text data in the same set, obtain the gray histogram of each text data respectively, compare the text data two by two, count the sum of the difference values of all gray levels in the two gray histograms, and mark the text data corresponding to the two gray histograms with a difference value sum lower than a preset threshold as repeated text data.

4. The computer data analysis processing method based on big data according to claim 3, characterized in that, When the byte amount of any two text data is greater than a preset threshold, divide the gray images of the text data into several regions, and compare each region in turn.

5. The computer data analysis processing method based on big data according to claim 1, characterized in that, The score calculation process is as follows: Let the number of the data to be stored be marked as n, the number of the evaluation indexes be marked as m, and the maximum index X ij be converted into the standardized index Z ij , i∈n, j∈m, the standardized index Z ij is converted from the maximum index X ij by the following formula: ； Obtain the maximum and minimum of any evaluation index in all pending storage data and mark as Z + j and Z - j , calculate the Euclidean distance D + of each evaluation index of any pending storage data with the corresponding maximum value Z i + , and the Euclidean distance D - of each evaluation index of any pending storage data with the corresponding minimum value Z i - ; then the unnormalized score of any pending storage data is: ； The score f i The final score F of any pending storage data is obtained by normalization processing i The formula is: 。 6. The computer data analysis processing method based on big data according to claim 5, characterized in that, Euclidean distance D i + and Euclidean distance D i - The calculation formula is: ， 。

Images