An artificial intelligence public data platform based on big data processing self-updating de-duplication
By constructing feature matrices and sparse feature matrices, and combining entropy analysis, the self-updating redundancy removal process of big data storage systems is optimized, solving the problem of low efficiency in traditional methods and achieving efficient redundancy removal processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGDA WEIYE (BEIJING) TECHNOLOGY CO LTD
- Filing Date
- 2024-08-14
- Publication Date
- 2026-05-12
AI Technical Summary
Traditional big data storage systems face challenges such as large data volumes and complex data types during self-updating and redundancy removal processes, and have failed to effectively utilize artificial intelligence technology to improve processing efficiency.
We employ an eigenvalue-based comparison and sparse data processing method. By constructing a feature matrix and a sparse feature matrix, and combining entropy analysis, we optimize the data partitioning and comparison process to achieve self-updating and redundancy removal.
It improves the redundancy removal efficiency of big data storage systems, reduces computational complexity, and is suitable for efficient redundancy removal of complex data types.
Smart Images
Figure CN119106275B_ABST
Abstract
Description
[Technical Field]
[0001] This invention belongs to the field of big data technology, and in particular relates to an artificial intelligence public data platform based on big data processing for self-updating and redundancy removal. [Background Technology]
[0002] The rapid development and widespread application of artificial intelligence (AI) technology are propelling society into a new era of intelligence and digitalization. With continuous technological advancements and strong policy support, AI is expected to play a crucial role in more fields, promoting high-quality economic and social development.
[0003] "Big data" typically refers to massive datasets that are difficult to collect, process, and analyze, as well as data that is stored long-term in traditional infrastructure. Storage for big data involves persistently storing these massive datasets in computers. Big data storage and management require using storage devices to store the collected data, establishing corresponding databases, and managing and accessing them. With the explosive growth of data volume, the storage and management needs of big data are rapidly increasing, posing new challenges to the performance, capacity, and architecture of storage systems. The complex data types of big data, including structured, semi-structured, and unstructured big data, make redundancy removal even more challenging. In traditional technologies, different redundancy removal strategies can be formulated based on factors such as data access frequency and modification frequency; for data with high access frequency, stricter deduplication strategies can be used to reduce the storage and transmission of duplicate data; for data with high modification frequency, more flexible deduplication strategies can be used to ensure data integrity and consistency.
[0004] In summary, self-updating redundancy removal for big data faces some new problems and challenges. Big data exhibits two significant characteristics during self-updating redundancy removal: large data volume and complex data types. Traditional redundancy removal methods often focus on comparing individual data items or merging different data types. Technological advancements tend to focus on data regularization and improving comparison algorithms, failing to directly address the issue of large data volume in big data storage. Furthermore, they have not applied artificial intelligence technology to big data processing. Based on these issues, this invention is applicable to big data processing. By utilizing artificial intelligence technology, it transforms data content comparison into feature value comparison, combining feature value comparison with sparse data content comparison. This significantly improves the processing efficiency of self-updating redundancy removal without compromising redundancy removal accuracy. [Summary of the Invention]
[0005] To address the aforementioned problems in the prior art, this invention proposes a self-updating redundancy removal method based on big data processing and its artificial intelligence public data platform. The method includes:
[0006] Step S1: For each data entry stored in the public data platform, divide it into one or more data regions; calculate the feature value of each data region, and use the feature value to construct a feature matrix corresponding to each data entry; each feature value type corresponds to a feature matrix;
[0007] Step S2: Calculate the feature matrix M corresponding to each eigenvalue type. u The entropy of the data; select the feature matrices corresponding to the top U feature value types with the largest entropy from all feature value types, and perform sparse processing on the feature matrices corresponding to the top U feature value types to obtain U sample sparse feature matrices; associate and store the U feature value types with the data entry.
[0008] The calculation of the feature matrix corresponding to each eigenvalue type u entropy S u Specifically, the entropy S is calculated using the following formulas (1) to (3). u Where: u = 1 to U; (i, j) are the element numbers in the characteristic matrix; It is the eigenvalue located at position (i, j);
[0009]
[0010] The sparse processing is performed to obtain U sample sparse feature matrices MS. u Specifically, this refers to the feature matrix M corresponding to each type u of feature. u In the characteristic matrix M u Delete some elements such that when the sparse window slides arbitrarily in the feature matrix, any deleted element in the sparse window has at least XS undeleted adjacent elements, excluding boundary elements; set the element values of the deleted elements to 0.
[0011] Step S3: Receive the data to be stored and divide it into one or more data regions; calculate the feature value of each data region, and use the feature value to construct a feature matrix corresponding to different feature value types for the data entry; perform sparsity processing on the feature matrix to obtain the current sparse feature matrix MC corresponding to the data to be stored. u ;
[0012] Step S4: For each data entry with the same data type as the data to be stored, compare the U feature value types associated with each data entry and their corresponding sample sparse feature matrices with the current sparse feature matrix of the corresponding feature value type of the data to be stored; when there is a data entry such that the sample sparse feature matrix corresponding to each feature value type is similar to the current sparse feature matrix of the corresponding feature value type, take the existing data entry as the data entry to be determined.
[0013] Step S5: Perform eigenvalue padding on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix; compare the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update to remove redundancy;
[0014] The step of performing eigenvalue completion based on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix is as follows: the sparse window slides in the feature matrix, the elements in the feature matrix within the sparse window that have element values in the sparse feature matrix are used as input data in the completion model, and the output data of the completion model is used as completion to fill in the corresponding positions of the elements that do not exist in the sparse window with eigenvalues to obtain the feature completion matrix.
[0015] The step of comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal specifically involves: comparing the feature completion matrix with the feature matrix of the data to be determined; if the feature completion matrix corresponding to each feature value type is similar to the feature matrix of the data to be determined of the corresponding feature value type, self-update redundancy removal is performed directly, and the data to be stored is deleted; otherwise, return to step S3 to process the next data to be stored.
[0016] Furthermore, the partitioning method is related to the type of data to be stored.
[0017] Furthermore, the data types to be stored include structured, semi-structured, and / or unstructured big data.
[0018] Furthermore, the types of data to be stored include images, audio, text, and / or video.
[0019] Furthermore, the feature values of each data region are of one or more types, and each feature value type corresponds to a feature matrix.
[0020] Furthermore, the division is a semantically independent data segmentation.
[0021] An artificial intelligence public data platform based on big data processing self-updating and redundancy removal is provided, wherein the artificial intelligence public data platform is used to implement the aforementioned big data processing self-updating and redundancy removal method.
[0022] An artificial intelligence public data server based on big data processing self-updating and redundancy removal includes a processor coupled to a memory. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the aforementioned big data processing self-updating and redundancy removal method is implemented.
[0023] A self-updating redundancy removal system based on big data processing is provided, the system being used to implement the aforementioned self-updating redundancy removal method based on big data processing.
[0024] A computer-readable storage medium includes a program that, when run on a computer, causes the computer to perform the aforementioned self-updating redundancy removal method based on big data processing.
[0025] The beneficial effects of this invention include:
[0026] (1) Applicable to big data processing, based on feature matrix and sparse feature data, the comparison of data content is converted into the comparison of feature values. Through simple numerical quantification calculation, the redundancy confirmation result is obtained quickly and effectively in big data information. When the prediction result cannot guarantee a high degree of redundancy confirmation and a vague confirmation result is obtained, the redundancy is further confirmed by the comparison of sparse data content. Thus, based on the hierarchical processing of feature matrix comparison and sparse data content comparison, the processing efficiency of self-updating redundancy removal is greatly improved without reducing the accuracy of redundancy removal.
[0027] (2) Applicable to complex data types. For different data types, the corresponding feature matrix is determined. The sparse feature matrix reduces the computational cost of subsequent comparisons. First, the feature matrix is described from multiple dimensions through multiple feature value types. The diversity of data content is discovered from multiple dimensions, thereby achieving dimensionality increase. Then, the entropy is used to discover the main analysis dimensions so that the sparse feature matrix can contain the maximum amount of information, thereby achieving dimensionality reduction. By combining dimensionality increase and dimensionality reduction, the computational complexity of self-updating and redundancy removal is greatly reduced. [Attached Image Description]
[0028] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to unduly limit the scope of the invention. In the drawings:
[0029] Figure 1 This is a schematic diagram of the self-updating redundancy removal method based on big data processing provided by the present invention.
Detailed Implementation Methods
[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The illustrative embodiments and descriptions are only used to explain the present invention and are not intended to limit the present invention.
[0031] This invention proposes a self-updating redundancy removal method based on big data processing and its artificial intelligence public data platform, as shown in the appendix. Figure 1 As shown, the method includes the following steps:
[0032] Step S1: For each data entry stored in the public data platform, divide the data entry to obtain one or more data regions; calculate the feature value of each data region, and use the feature value to construct a feature matrix corresponding to the data entry;
[0033] Preferably, the partitioning method is related to the type of data to be stored; for example, the type of data to be stored includes images, audio, text, video, etc.
[0034] Replaceable types: Data types to be stored include structured, semi-structured, and / or unstructured big data, etc.
[0035] Preferably, the feature values of each data region are of one or more types, and each feature value type corresponds to a feature matrix;
[0036] Preferably, the division is a semantically independent data segmentation;
[0037] Preferably, the feature value is one or more of the following: mean, median, mode, variance, extreme value, keyword, word frequency and / or word distance;
[0038] Preferred approach: Pre-set the correspondence between data types and their partitioning methods; of course, a data type can be configured with multiple partitioning methods; further select a partitioning method from the multiple corresponding partitioning methods based on the data size;
[0039] Step S2: Calculate the feature matrix M corresponding to each eigenvalue type. u The entropy of the data is calculated; the feature matrices corresponding to the top U feature value types with the largest entropy are selected from all feature value types UA, and the feature matrices corresponding to the top U feature value types are sparsified to obtain the sample sparse feature matrices; the U feature value types are associated with the data entry and stored; that is, the number of sparse feature matrices is U; when comparing the data entry later, only the U sparse feature matrices need to be compared; this greatly reduces the number of samples used for comparison, thereby reducing the storage and computational costs of comparison.
[0040] Preferred values: UA and U are preset values, for example: UA = 10, U = 2;
[0041] The calculation of the feature matrix corresponding to each eigenvalue type u entropy S u Specifically, the entropy S is calculated using the following formulas (1) to (3). uWhere: u = 1 to U; (i, j) are the element numbers in the characteristic matrix; It is the eigenvalue located at position (i, j);
[0042]
[0043] The sparse processing is performed to obtain the sample sparse feature matrix MS. u Specifically, in the characteristic matrix M u Delete some elements such that when the sparse window slides arbitrarily in the feature matrix, any deleted element in the sparse window has at least XS undeleted adjacent elements, excluding boundary elements; set the element values of the deleted elements to 0.
[0044] Alternatively, the sparse processing is performed to obtain the sample sparse feature matrix MS. u Specifically, in the characteristic matrix M u Deleting some elements ensures that when the sparse window slides in the feature matrix, every deleted element in the sparse window has exactly XS undeleted adjacent elements, excluding boundary elements; the sliding method is to slide according to a preset step size, for example: the preset step size is equal to the length or width of the sparse window.
[0045] Preferably, the size and XS value of the sparse window are both preset values; for example, the sparse window size is set to 3×3; and the XS is set to 2, 4, or 6.
[0046] Preferably, the adjacent elements are 4 adjacent elements or 8 adjacent elements;
[0047] Preferred setting: The XS setting is related to the degree of redundancy removal. When storage space is limited, a higher degree of redundancy removal is used, which requires a lower XS value, thereby improving the granularity of redundancy removal.
[0048] Alternatively, the sparse processing to obtain a sparse feature matrix can be performed by: expanding the feature matrix into rows and columns, and deleting one feature value at a preset interval to obtain a sparse feature matrix; for example, the preset interval is 1; after deletion, the rows and columns are restored.
[0049] Step S3: Receive the data to be stored and divide it into one or more data regions; calculate the feature value of each data region, and use the feature value to construct a feature matrix corresponding to different feature value types for the data entry; perform sparsity processing on the feature matrix to obtain the current sparse feature matrix MC corresponding to the data to be stored. u Obviously, the methods for dividing the data to be stored, constructing the feature matrix, and determining the sparse feature matrix, as well as the processing methods for data entries in the public data platform in steps S1 and S2, are the same here.
[0050] Step S4: For each data entry with the same data type as the data to be stored, compare the U feature value types associated with each data entry and their corresponding sample sparse feature matrix with the current sparse feature matrix of the data to be stored to obtain the data entry of the data to be determined in the public data platform;
[0051] Step S4 specifically includes the following steps:
[0052] Step S41: Treat each data entry with the same data type as the data to be stored as an unprocessed data entry;
[0053] Step S42: Obtain an unprocessed data entry; obtain the U eigenvalue types associated with the unprocessed data entry and their corresponding sample sparse feature matrix MS. u ;
[0054] Step S43: For each feature type, sequentially process the sparse feature matrix of the samples. and the current sparse feature matrix Comparison to obtain sparse matrix similarity SM u Specifically, the sparse matrix similarity SM is calculated using the following formula (4) or (5). u Where: m and n are the row and column numbers of the sparse feature matrix elements; ε is the local minimum.
[0055]
[0056] Step S44: Based on entropy and sparse matrix similarity SM u Obtain the comprehensive matrix similarity (SSM) u Specifically, the comprehensive matrix similarity SSM is calculated using the following formula (6). u ;
[0057] SSM u =∑ u S u ×SM u (6);
[0058] Step S45: Determine whether all data entries have been processed. If yes, return to step S42; otherwise, proceed to the next step.
[0059] Step S46: When there is a data entry with a comprehensive matrix similarity greater than the sparse matrix similarity threshold, select the data entry with the largest comprehensive matrix similarity as the data entry to be determined; otherwise, do not perform sub-update to remove redundancy, and return to step S3 to process the next data to be stored.
[0060] Preferably, the sparse matrix similarity threshold is a preset value;
[0061] Alternatively, step S4 specifically involves: for each data entry with the same data type as the data to be stored, comparing the U feature value types associated with each data entry and their corresponding sample sparse feature matrices with the current sparse feature matrix of the data to be stored; when there exists a data entry such that the sample sparse feature matrix corresponding to each of its feature value types is similar to the current sparse feature matrix of the corresponding feature value type, the data entry is taken as the data entry of the data to be determined.
[0062] Step S5: Perform eigenvalue completion based on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix; compare the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update to remove redundancy;
[0063] Preferably, the completion is either complete completion or partial completion;
[0064] The step of performing eigenvalue completion based on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix is as follows: the sparse window slides within the feature matrix, and the elements in the feature matrix within the sparse window that have values in the sparse feature matrix are used as input data in the completion model, while the output data of the completion model is used as completion data to fill in the corresponding positions of the elements that do not exist in the sparse window with eigenvalues to obtain the feature completion matrix; the sliding method here corresponds to the sliding method in sparse processing.
[0065] Preferred approach: When there is a discrepancy between the feature values used for completion at the same position, select the output data with higher confidence as the feature value used for completion; discrepancies may exist during arbitrary sliding.
[0066] Alternative: When there are discrepancies between the feature values used for completion at the same position, select the output data obtained when the input data is longer or more numerous as the feature value used for completion;
[0067] Preferably, the completion model is an artificial intelligence model, trained using data entries from a public data platform; furthermore, the data of the artificial intelligence model includes output data and its confidence level.
[0068] The step of comparing the feature completion matrix and the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal is specifically as follows: comparing the feature completion matrix and the feature matrix of the data to be determined to obtain the comprehensive matrix similarity corresponding to each feature value type and the comprehensive matrix similarity; when the comprehensive matrix similarity is greater than the high redundancy similarity, proceed to step S7 to directly perform self-update redundancy removal; when the similarity between the two is less than the low redundancy similarity, return to step S3 to process the next data to be stored; otherwise, proceed to step S6 to compare the data content; the calculation method of the matrix similarity corresponding to each feature value type is the same as the calculation method of the sparse matrix similarity (formula (4) or (5)), and the calculation method of the comprehensive matrix similarity is the same as the calculation method of the comprehensive matrix similarity (formula (6));
[0069] Alternatively, the step of comparing the feature completion matrix and the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal specifically involves: comparing the feature completion matrix and the feature matrix of the data to be determined to obtain a comprehensive matrix similarity corresponding to the matrix similarity and the comprehensive matrix similarity for each feature value type; when the comprehensive matrix similarity is greater than the high redundancy similarity, proceed to step S7 to directly perform self-update redundancy removal; otherwise, return to step S3 to process the next data to be stored.
[0070] Preferably, the high redundancy similarity is greater than the low redundancy similarity, and both are preset values;
[0071] When the prediction accuracy is high, or the default prediction accuracy is high and the similarity of the completed feature matrix is high, it reflects a high degree of redundancy. Direct self-update and redundancy removal are performed to avoid subsequent time-consuming data content comparison. Furthermore, when the prediction result cannot guarantee a high degree of redundancy and a vague confirmation result is obtained, redundancy is further confirmed by sparse data content comparison. Thus, based on the hierarchical processing of feature matrix comparison and sparse data content comparison, the processing efficiency of self-update redundancy removal is greatly improved without reducing the redundancy removal accuracy.
[0072] Alternatively: Step S5 specifically involves comparing the feature matrix of the data to be stored with the feature matrix of the data to be determined to obtain a comprehensive matrix similarity corresponding to each feature value type and a comprehensive matrix similarity. When the comprehensive matrix similarity (or all matrix similarities) is greater than the second highest redundancy similarity, proceed to step S7 to directly perform self-update and redundancy removal. When the similarity is less than the low redundancy similarity, return to step S3 to process the next data to be stored. Otherwise, proceed to step S6 to compare data content. The completion-based determination method considers the compatibility between the data to be stored and the overall data entries in the public data platform. Therefore, compared with the completion-based determination method, the redundancy is lower, the requirements for the data to be stored are higher, the redundancy removal efficiency is lower, and the pressure on content comparison is greater. It is suitable for cases where the data size of the data to be stored is small. In other words, by adopting the completion-based determination method, the processing capacity of the targeted big data and the big data capacity of the public data platform can be further improved.
[0073] Preferably, the second high redundancy similarity is greater than or equal to the first high redundancy similarity;
[0074] Alternatively: The step of comparing the feature completion matrix and the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal specifically involves: comparing the feature matrix of the data to be stored and the feature matrix of the data to be determined to obtain a comprehensive matrix similarity corresponding to each feature value type and a comprehensive matrix similarity; when the comprehensive matrix similarity (or all matrix similarities) is greater than the high redundancy similarity, proceed to step S7 to directly perform self-update redundancy removal; otherwise, proceed to step S6 to perform data content comparison;
[0075] Preferably, the high redundancy similarity and low redundancy similarity are related to the size of the feature matrix;
[0076] Alternatively, the step of comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal specifically involves: comparing the feature completion matrix with the feature matrix of the data to be determined; if the feature completion matrix corresponding to each feature value type is similar to the feature matrix of the data to be determined for the corresponding feature value type, then self-update redundancy removal is performed directly, and the data to be stored is deleted; otherwise, the process returns to step S3 to process the next data to be stored.
[0077] Step S6: Obtain the first data region corresponding to the current sparse feature matrix of the data to be stored; obtain the second data region in the data to be stored that is not the first data region; obtain the third data region corresponding to the sample sparse feature matrix of the data to be determined; obtain the fourth data region in the data to be determined that is not the third data region; compare the data content of the data items in the third data region and the fourth data region. When the similarity between the two is greater than the high similarity threshold, proceed to step S7; otherwise, return to step S3 to process the next data to be stored.
[0078] The data content comparison process specifically involves: using string comparison, numerical comparison, or other data comparison algorithms to identify the differences between various data items in the data area; setting a data content similarity threshold, and determining that two data items can be considered duplicates when their similarity exceeds the data content similarity threshold;
[0079] Preferred method: Use fuzzy matching to identify data items that are similar but not identical in content;
[0080] Preferred approach: Before comparing data content, convert data items into a uniform format, such as date format, text case, etc., to reduce erroneous comparison results due to format differences;
[0081] Step S7: Perform self-update to remove redundancy and delete data to be stored;
[0082] Alternatively: Step S7 specifically involves setting a self-update timer, and deleting the data to be stored when the timer reaches zero;
[0083] Alternatively: Step S7 specifically involves: archiving the data to be stored to free up storage space;
[0084] Alternatively: Step S7 specifically involves: reducing the storage priority of the data to be stored and setting a self-update condition; when the self-update condition is met, deleting the data to be stored; the sub-update condition is that the storage priority reaches a preset priority and / or the timer is reset to zero.
[0085] Based on the same inventive concept, the present invention also provides a self-updating redundancy removal system based on big data processing, the system being used to implement the above-mentioned self-updating redundancy removal method based on big data processing;
[0086] Based on the same inventive concept, the present invention also provides a self-updating redundancy removal server based on big data processing, the server being used to implement the above-mentioned self-updating redundancy removal method based on big data processing.
[0087] Based on the same inventive concept, the present invention also provides a self-updating and redundancy removal device based on big data processing, the device being used to implement the above-mentioned self-updating and redundancy removal method based on big data processing.
[0088] Based on the same inventive concept, the present invention also provides a big data processing self-updating redundancy removal platform, the platform being used to implement the above-mentioned big data processing self-updating redundancy removal method;
[0089] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including assembly or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple co-located files (e.g., a file storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a communications network.
[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A self-updating redundancy removal method based on big data processing, characterized in that, The method includes: Step S1: For each data entry stored in the public data platform, divide it into one or more data regions; calculate the feature value of each data region, and use the feature value to construct a feature matrix corresponding to each data entry; each feature value type corresponds to a feature matrix; in; The partitioning method is related to the type of data to be stored; the types of data to be stored include images, audio, text, and / or video. Step S2: Calculate the feature matrix corresponding to each eigenvalue type. The entropy of the first U eigenvalue types is selected from all eigenvalue types, and the eigenvalue matrices corresponding to the top U eigenvalue types are sparsed to obtain U sample sparse feature matrices. Associate and store the U feature value types with the data entry; The calculation of the feature matrix corresponding to each eigenvalue type u entropy Specifically, the entropy is calculated using the following formulas (1) to (3). ;in: It is the element number in the feature matrix; It is located in position eigenvalues; (1); (2); (3); The sparse processing is performed to obtain U sample sparse feature matrices. Specifically, this refers to the feature matrix corresponding to each type u of feature. In the feature matrix Deleting some elements ensures that, as the sparse window slides arbitrarily across the feature matrix, every deleted element within the sparse window exists and at least exists. The elements that have not been deleted, excluding boundary elements, are set to 0; the element values of the elements to be deleted are set to 0. Step S3: Receive the data to be stored and divide it into one or more data regions; calculate the feature value of each data region, and use the feature value to construct the feature matrix corresponding to different feature value types for the data entry; perform sparse processing on the feature matrix to obtain the current sparse feature matrix corresponding to the data to be stored; Step S4: For each data entry with the same data type as the data to be stored, compare the U feature value types associated with each data entry and their corresponding sample sparse feature matrices with the current sparse feature matrix of the corresponding feature value type of the data to be stored; when there is a data entry such that the sample sparse feature matrix corresponding to each feature value type is similar to the current sparse feature matrix of the corresponding feature value type, take the existing data entry as the data entry to be determined. Step S5: Perform eigenvalue padding on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix; compare the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update to remove redundancy; Specifically, for the storage and management of the big data, a storage device is used to store the collected data, a corresponding database is established, and the data is managed and accessed. Step S6: Obtain the first data region corresponding to the current sparse feature matrix of the data to be stored; obtain the second data region in the data to be stored that is not the first data region; obtain the third data region corresponding to the sample sparse feature matrix of the data to be determined; obtain the fourth data region in the data to be determined that is not the third data region; compare the data content of the data items in the third data region and the fourth data region, and when the similarity between the two is greater than the high similarity threshold, proceed to step S7; Step S7: Perform self-update to remove redundancy and delete data to be stored.
2. The self-updating redundancy removal method based on big data processing according to claim 1, characterized in that, The types of data to be stored include structured, semi-structured, and / or unstructured big data.
3. The self-updating redundancy removal method based on big data processing according to claim 2, characterized in that, Each data region has one or more feature values of different types, and each feature value type corresponds to a feature matrix.
4. The self-updating redundancy removal method based on big data processing according to claim 3, characterized in that, The division is a semantically independent data segmentation.
5. An artificial intelligence public data platform based on big data processing for self-updating and redundancy removal, characterized in that: The artificial intelligence public data platform is used to implement the big data processing-based self-updating redundancy removal method as described in any one of claims 1-4.
6. A public data server for artificial intelligence based on big data processing, characterized in that: The system includes a processor coupled to a memory, the memory storing program instructions, which, when executed by the processor, implement the self-updating redundancy removal method based on big data processing as described in any one of claims 1-4.
7. A self-updating and redundancy-removing system based on big data processing, characterized in that, The system is used to implement the self-updating redundancy removal method based on big data processing as described in any one of claims 1-4.
8. A computer-readable storage medium, characterized in that, Includes a program that, when run on a computer, causes the computer to perform the self-updating redundancy removal method based on big data processing as described in any one of claims 1-4.