Artificial intelligence public data platform based on big data processing self-updating redundancy removal

Abstract

The invention relates to a self-updating redundancy elimination method based on big data processing and an artificial intelligence public data platform thereof. The method comprises the following steps: for each data entry with the same data type as the to-be-stored data, comparing a sparse feature matrix corresponding to the data entry with a sparse feature matrix of the to-be-stored data to obtain a data entry of the to-be-determined data in the public data platform; performing feature value completion based on the sparse feature matrix of the to-be-stored data to obtain a complete feature completion matrix; and comparing the feature completion matrix with the feature matrix of the to-be-determined data to determine whether to perform self-updating redundancy elimination. The method is suitable for big data processing, data content comparison is converted into feature value comparison through the artificial intelligence technology, and the processing efficiency of self-updating redundancy elimination is greatly improved on the basis that the redundancy elimination accuracy is not reduced.

Description

[Technical field]

[0001] The present invention belongs to the field of big data technology, and in particular relates to an artificial intelligence public data platform based on big data processing, self-updating and redundancy removal. [Background technology]

[0002] The rapid development and widespread application of artificial intelligence technology are driving society into a new era of intelligence and digitalization. With the continuous advancement of technology and strong policy support, artificial intelligence is expected to play a key role in more fields and promote high-quality development of the economy and society.

[0003] "Big data" usually refers to data sets that are huge in number and difficult to collect, process, and analyze, and also refers to data that is stored for a long time in traditional infrastructure. Storage for big data is to persist data sets that are huge in number and difficult to collect, process, and analyze in computers. Big data storage and management requires the use of memory to store the collected data, establish a corresponding database, and manage and call it. With the explosive growth of data scale, the storage and management requirements of big data are growing rapidly with the increase in data volume, which poses new challenges to the performance, capacity, and architecture of storage systems. The data types of big data are complex, including structured, semi-structured, and unstructured big data, which makes de-duplication face greater challenges. In traditional technologies, different de-duplication strategies can be formulated based on factors such as data access frequency and modification frequency; for data with higher access frequency, more stringent de-duplication strategies can be adopted to reduce the storage and transmission of duplicate data; for data with higher modification frequency, more flexible de-duplication strategies can be adopted to ensure data integrity and consistency.

[0004] In summary, the self-updating and de-redundancy of big data faces some new problems and challenges; big data presents two significant characteristics when performing self-updating and de-redundancy, one is that the data volume is large, and the other is that the data type is relatively complex. Traditional de-redundancy methods are often carried out by comparing data items one by one and fusing and comparing different data types. The development of technology often focuses on improving data regularization and comparison algorithms, and fails to face and focus on the problem of large data volume in big data storage; it also fails to use artificial intelligence technology in the process of big data processing; based on the above problems, the present invention is suitable for big data processing, and uses artificial intelligence technology to convert data content comparison into feature value comparison, and combines feature value comparison with sparse data content comparison, which greatly improves the processing efficiency of self-updating and de-redundancy without reducing the accuracy of de-redundancy. [Summary of the invention]

[0005] In order to solve the above problems in the prior art, the present invention proposes a self-updating redundancy removal method based on big data processing and an artificial intelligence public data platform thereof, the method comprising:

[0006] Step S1: for each data entry stored in the public data platform, divide it to obtain one or more data regions; obtain the eigenvalue of each data region, and use the eigenvalue to construct a feature matrix corresponding to each data entry; each eigenvalue type corresponds to a feature matrix;

[0007] Step S2: Calculate the characteristic matrix M corresponding to each eigenvalue type u entropy; selecting the feature matrices corresponding to the first U eigenvalue types with the largest entropy from all eigenvalue types, performing sparse processing on the feature matrices corresponding to the first U eigenvalue types to obtain U sample sparse feature matrices; and associating and storing the U eigenvalue types with the data entry;

[0008] The characteristic matrix corresponding to each eigenvalue type u is calculated The entropy S u Specifically, the entropy S is calculated using the following formulas (1) to (3): u ; Where: u = 1 ~ U; (i, j) is the element number in the feature matrix; is the eigenvalue at position (i, j);

[0009]

[0010] The sparse processing is performed to obtain U sample sparse feature matrices MS u ; Specifically: for each feature type u, the feature matrix M corresponding to u , in the feature matrix M u Delete some elements from the sparse window so that when the sparse window slides arbitrarily in the feature matrix, any deleted element in the sparse window exists and has at least XS adjacent elements that are not deleted, except for the boundary elements; the element values ​​of some deleted elements are set to 0;

[0011] Step S3: Receive the data to be stored and divide it into one or more data regions; obtain the eigenvalue of each data region, and use the eigenvalue to construct a feature matrix corresponding to different eigenvalue types of the data entry; perform sparse processing on the feature matrix to obtain the current sparse feature matrix MC corresponding to the data to be stored u ;

[0012] Step S4: for each data entry of the same data type as the data to be stored, compare the U eigenvalue types and the corresponding sample sparse feature matrices associated with each data entry with the current sparse feature matrix of the corresponding eigenvalue type of the data to be stored; when there is a data entry such that the sample sparse feature matrices corresponding to each eigenvalue type are similar to the current sparse feature matrix of the corresponding eigenvalue type, use the existing data entry as the data entry of the data to be determined;

[0013] Step S5: completing the eigenvalues ​​of the current sparse feature matrix of the data to be stored to obtain a feature completion matrix; comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update to remove redundancy;

[0014] The method of performing eigenvalue completion based on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix is ​​specifically as follows: a sparse window is made to slide in the feature matrix, and elements in the feature matrix in the sparse window that have element values ​​in the sparse feature matrix are used as input data in the completion model, and output data of the completion model are used as completion eigenvalues ​​to complete corresponding positions of elements that do not exist in the sparse window to obtain a feature completion matrix;

[0015] The step of comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update and redundancy removal is specifically as follows: comparing the feature completion matrix with the feature matrix of the data to be determined, and when the feature completion matrix corresponding to each eigenvalue type is similar to the feature matrix of the data to be determined of the corresponding eigenvalue type, directly performing self-update and redundancy removal, and deleting the data to be stored; otherwise, returning to step S3 to process the next data to be stored.

[0016] Furthermore, the division method is related to the type of data to be stored.

[0017] Furthermore, the types of data to be stored include structured, semi-structured and / or unstructured big data.

[0018] Furthermore, the types of data to be stored include images, voices, texts and / or videos.

[0019] Furthermore, the eigenvalues ​​of each data region are of one or more types, and each eigenvalue type corresponds to a feature matrix.

[0020] Furthermore, the division is semantically irrelevant data segmentation.

[0021] An artificial intelligence public data platform based on big data processing self-update and redundancy removal, the artificial intelligence public data platform is used to implement the method based on big data processing self-update and redundancy removal.

[0022] An artificial intelligence public data server based on big data processing self-update and redundancy removal includes a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method for self-update and redundancy removal based on big data processing is implemented.

[0023] A self-updating and redundancy removal system based on big data processing, the system is used to implement the self-updating and redundancy removal method based on big data processing.

[0024] A computer-readable storage medium includes a program, which, when running on a computer, enables the computer to execute the self-updating and redundancy-removing method based on big data processing.

[0025] The beneficial effects of the present invention include:

[0026] (1) It is suitable for big data processing. Based on the feature matrix and sparse feature data, the data content comparison is converted into the feature value comparison. Through simple numerical quantization calculation, the redundancy confirmation result is quickly and effectively obtained in the big data information. When the prediction result cannot guarantee a high redundancy confirmation and obtains a fuzzy confirmation result, the sparse data content comparison is used to make further redundancy confirmation. Thus, based on the hierarchical processing of feature matrix comparison and sparse data content comparison, the processing efficiency of self-update redundancy removal is greatly improved without reducing the accuracy of redundancy removal.

[0027] (2) It is applicable to complex data types. It determines the corresponding feature matrix for different data types. The sparse feature matrix reduces the amount of calculation for subsequent comparisons. First, the feature matrix is ​​described from multiple dimensions through multiple eigenvalue types. The diversity of data content is discovered from multiple dimensions, thereby achieving dimensionality increase. Then, the main analysis dimension is discovered through entropy so that the sparse feature matrix can contain the maximum amount of information, thereby achieving dimensionality reduction. Through the combination of dimensionality increase and dimensionality reduction, the computational complexity of self-update redundancy removal is greatly reduced.

Brief Description of the Drawings

[0028] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:

[0029] Figure 1 A schematic diagram of the self-updating and redundancy removal method based on big data processing provided by the present invention. [Specific implementation method]

[0030] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0031] The present invention proposes a self-updating redundancy removal method based on big data processing and an artificial intelligence public data platform thereof, as shown in the attached Figure 1 As shown, the method comprises the following steps:

[0032] Step S1: for each data entry stored in the public data platform, divide the data entry to obtain one or more data regions; obtain the eigenvalue of each data region, and use the eigenvalue to construct a feature matrix corresponding to the data entry;

[0033] Preferably, the division method is related to the type of data to be stored; for example, the type of data to be stored includes images, voice, text, video, etc.;

[0034] Interchangeable: The types of data to be stored include structured, semi-structured and / or unstructured big data;

[0035] Preferably: the eigenvalues ​​of each data region are of one or more types, and each eigenvalue type corresponds to a feature matrix;

[0036] Preferably: the division is semantically irrelevant data segmentation;

[0037] Preferably: the characteristic value is one or more of mean, median, mode, variance, extreme value, keyword, word frequency and / or distance between words;

[0038] Preferably: the corresponding relationship between the data type and its division method is preset; of course, a data type can be set with multiple division methods corresponding thereto; and a division method is further selected from the multiple corresponding division methods according to the data size;

[0039] Step S2: Calculate the characteristic matrix M corresponding to each eigenvalue type u entropy; select the feature matrices corresponding to the first U eigenvalue types with the largest entropy from all eigenvalue types UA, and perform sparse processing on the feature matrices corresponding to the first U eigenvalue types to obtain a sample sparse feature matrix; store the U eigenvalue types in association with the data entry; that is, the number of sparse feature matrices is U; when comparing the data entry later, only the U sparse feature matrices need to be compared; the amount of samples used for comparison is greatly reduced, thereby reducing the amount of storage and the amount of calculation used for comparison;

[0040] Preferably: UA and U are preset values, for example: UA=10, U=2;

[0041] The characteristic matrix corresponding to each eigenvalue type u is calculated The entropy S u Specifically, the entropy S is calculated using the following formulas (1) to (3): u; Where: u = 1 ~ U; (i, j) is the element number in the feature matrix; is the eigenvalue at position (i, j);

[0042]

[0043] The sparse processing is performed to obtain a sample sparse feature matrix MS u ; Specifically: in the feature matrix M u Delete some elements from the sparse window so that when the sparse window slides arbitrarily in the feature matrix, any deleted element in the sparse window exists and has at least XS adjacent elements that are not deleted, except for the boundary elements; the element values ​​of some deleted elements are set to 0;

[0044] Alternatively, the sparse processing is performed to obtain a sample sparse feature matrix MS u ; Specifically: in the feature matrix M u , so that when the sparse window slides in the feature matrix, any deleted element in the sparse window has and only has XS adjacent elements that are not deleted, except for boundary elements; the sliding method is sliding according to a preset step size, for example, the preset step size is equal to the length or width of the sparse window;

[0045] Preferably: the size and XS value of the sparse window are both preset values; for example: the sparse window size is set to a 3×3 window; XS is set to 2, 4, 6;

[0046] Preferably: the adjacent elements are 4 adjacent elements or 8 adjacent elements;

[0047] Optimum: The setting of XS is related to the degree of redundancy reduction. When the storage space is limited, a higher degree of redundancy reduction is adopted, which requires a lower XS value, thereby improving the granularity of redundancy reduction.

[0048] Alternatively, the sparse processing is performed to obtain a sparse feature matrix; specifically, after the feature matrix is ​​expanded in rows and columns, one eigenvalue is deleted at every preset interval to obtain a sparse feature matrix; for example, the preset interval is 1; and after the deletion is completed, the rows and columns are restored;

[0049] Step S3: receiving the data to be stored and dividing it to obtain one or more data regions; obtaining the eigenvalues ​​of each data region, and using the eigenvalues ​​to construct a feature matrix corresponding to different eigenvalue types of the data entry; performing sparse processing on the feature matrix to obtain a current sparse feature matrix MC corresponding to the data to be stored u ; Obviously, the division of the stored data, the construction of the feature matrix, and the determination of the sparse feature matrix are the same as the processing of the data entries in the public data platform in step S1 and step S2;

[0050] Step S4: for each data entry of the same data type as the data to be stored, compare the U eigenvalue types associated with each data entry and the corresponding sample sparse feature matrix with the current sparse feature matrix of the data to be stored to obtain the data entry of the data to be determined in the public data platform;

[0051] The step S4 specifically includes the following steps:

[0052] Step S41: treating each data entry of the same data type as the data to be stored as an unprocessed data entry;

[0053] Step S42: Obtain an unprocessed data entry; obtain U eigenvalue types associated with the unprocessed data entry and their corresponding sample sparse feature matrix MS u ;

[0054] Step S43: For each eigenvalue type, the sample sparse feature matrix and the current sparse feature matrix Compare to get the sparse matrix similarity SM u Specifically, the sparse matrix similarity SM is calculated using the following formula (4) or (5): u ; Where: m, n are the row and column numbers of the sparse feature matrix elements; ε is the minimum value;

[0055]

[0056] Step S44: Based on the entropy and sparse matrix similarity SM u Get the comprehensive matrix similarity SSM u ; Specifically: Use the following formula (6) to calculate the comprehensive matrix similarity SSM u ;

[0057] SSM u =∑ u S u ×SM u (6);

[0058] Step S45: determine whether all data items have been processed, if yes, return to step S42; otherwise, proceed to the next step;

[0059] Step S46: when there is a data entry whose comprehensive matrix similarity is greater than the sparse matrix similarity threshold, select the data entry with the largest comprehensive matrix similarity as the data entry of the data to be determined; otherwise, do not perform sub-update de-redundancy, and return to step S3 to process the next data to be stored;

[0060] Preferably: the sparse matrix similarity threshold is a preset value;

[0061] Alternatively, the step S4 is specifically as follows: for each data entry of the same data type as the data to be stored, the U eigenvalue types associated with each data entry and the corresponding sample sparse feature matrix are compared with the current sparse feature matrix of the data to be stored; when there is a data entry such that the sample sparse feature matrix corresponding to each eigenvalue type is similar to the current sparse feature matrix of the corresponding eigenvalue type, the data entry is used as the data entry of the data to be determined;

[0062] Step S5: performing eigenvalue completion based on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix; comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update to remove redundancy;

[0063] Preferably: the completion is complete completion or partial completion;

[0064] The method of performing eigenvalue completion based on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix is ​​specifically as follows: the sparse window is slid in the feature matrix, and the elements in the feature matrix in the sparse window that have element values ​​in the sparse feature matrix are used as input data in the completion model, and the output data of the completion model are used as the completion eigenvalues ​​to complete the corresponding positions of the elements that do not exist in the sparse window to obtain the feature completion matrix; the sliding method here corresponds to the sliding method in the sparse processing;

[0065] Preferably: when there is a disagreement between the complement feature values ​​at the same position, select the output data with higher confidence as the complement feature value; there may be disagreement during any sliding;

[0066] Alternative: When there is a disagreement between the complement feature values ​​at the same position, the output data obtained when the input data is longer or more is selected as the complement feature value;

[0067] Preferably: the completion model is an artificial intelligence model, and the artificial intelligence model is trained using data items in a public data platform; further, the data of the artificial intelligence model includes output data and its confidence level;

[0068] The step of comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal is specifically as follows: comparing the feature completion matrix with the feature matrix of the data to be determined to obtain a matrix similarity corresponding to each eigenvalue type and a comprehensive matrix similarity of the comprehensive matrix similarity; when the comprehensive matrix similarity is greater than the high redundancy similarity, entering step S7 to directly perform self-update redundancy removal; when the similarity between the two is less than the low redundancy similarity, returning to step S3 to process the next data to be stored; otherwise, entering step S6 to perform data content comparison; the calculation method of the matrix similarity corresponding to each eigenvalue type is the same as the calculation method of the sparse matrix similarity (Formula (4) or (5)), and the calculation method of the comprehensive matrix similarity is the same as the calculation method of the comprehensive matrix similarity (Formula (6));

[0069] Alternatively, the comparison of the feature completion matrix and the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal is specifically: comparing the feature completion matrix and the feature matrix of the data to be determined to obtain a comprehensive matrix similarity of a matrix similarity corresponding to each eigenvalue type and a comprehensive matrix similarity; when the comprehensive matrix similarity is greater than the high redundancy similarity, entering step S7 to directly perform self-update redundancy removal; otherwise, returning to step S3 to process the next data to be stored;

[0070] Preferably: the high redundancy similarity is greater than the low redundancy similarity, and both are preset values;

[0071] When the prediction result accuracy is high, or the default prediction result prediction accuracy is high and the similarity of the completed feature matrix is ​​high, it reflects a high redundancy, and self-update de-redundancy is directly performed to avoid subsequent time-consuming data content comparison. Furthermore, when the prediction result cannot guarantee a high redundancy confirmation and obtains a vague confirmation result, further redundancy confirmation is performed through sparse data content comparison, thereby greatly improving the processing efficiency of self-update de-redundancy based on hierarchical processing of feature matrix comparison and sparse data content comparison without reducing the accuracy of de-redundancy.

[0072] Alternatively: the step S5 is specifically as follows: compare the characteristic matrix of the data to be stored and the characteristic matrix of the data to be determined to obtain the matrix similarity corresponding to each eigenvalue type and the comprehensive matrix similarity of the comprehensive matrix similarity; when the comprehensive matrix similarity (or all matrix similarities) of the similarity of the two is greater than the second highest redundancy similarity, enter step S7 to directly perform self-update redundancy removal; when the similarity of the two is less than the low redundancy similarity, return to step S3 to process the next data to be stored; otherwise, enter step S6 to compare the data content; the determination method of complementation is used to consider the compatibility between the data to be stored and the data entries in the public data platform as a whole. Therefore, compared with the determination method of complementation, the determination method has lower redundancy, higher requirements for the data to be stored, low redundancy removal efficiency, greater pressure on content comparison, and is suitable for the case where the data size of the data to be stored is small; that is, by adopting the determination method of complementation, the processing capability of the targeted big data and the big data capacity of the public data platform can be further improved;

[0073] Preferably: the second high redundancy similarity is greater than or equal to the first high redundancy similarity;

[0074] Alternatively: the step of comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal is specifically as follows: comparing the feature matrix of the data to be stored with the feature matrix of the data to be determined to obtain a comprehensive matrix similarity of the matrix similarity and the comprehensive matrix similarity corresponding to each eigenvalue type; when the comprehensive matrix similarity (or all matrix similarities) of the similarities of the two is greater than the high redundancy similarity, entering step S7 to directly perform self-update redundancy removal; otherwise, entering step S6 to perform data content comparison;

[0075] Preferably: the high redundancy similarity and the low redundancy similarity are related to the size of the feature matrix;

[0076] Alternatively, the step of comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update redundancy removal is specifically as follows: comparing the feature completion matrix with the feature matrix of the data to be determined, and when the feature completion matrix corresponding to each eigenvalue type is similar to the feature matrix of the data to be determined of the corresponding eigenvalue type, directly performing self-update redundancy removal and deleting the data to be stored; otherwise, returning to step S3 to process the next data to be stored;

[0077] Step S6: obtaining a first data region corresponding to the current sparse feature matrix of the data to be stored; obtaining a second data region other than the first data region in the data to be stored; obtaining a third data region corresponding to the sample sparse feature matrix of the data to be determined; obtaining a fourth data region other than the third data region in the data to be determined; comparing the data contents of the data items in the third data region and the fourth data region, and when the similarity between the two is greater than the high similarity threshold, proceeding to step S7; otherwise, returning to step S3 to process the next data to be stored;

[0078] The data content comparison is specifically: using string comparison, numerical comparison or other data comparison algorithms to identify the differences between the data items in the data region; setting a data content similarity threshold, and when the similarity between the two is greater than the data content similarity threshold, determining that the two data contents can be considered to be duplicates;

[0079] Preferred: Use fuzzy matching to identify data items that are similar but not identical in content;

[0080] Preferably: before comparing data content, convert the data items into a uniform format, such as date format, text case, etc., to reduce erroneous comparison results due to format differences;

[0081] Step S7: Perform self-update to remove redundancy and delete the data to be stored;

[0082] Alternatively: the step S7 specifically includes: setting a self-update timer, and deleting the data to be stored when the timer returns to zero;

[0083] Alternatively: the step S7 specifically includes: archiving the data to be stored to release storage space;

[0084] Alternatively: the step S7 is specifically: lowering the storage priority of the data to be stored and setting a self-update condition, and when the self-update condition is met, deleting the data to be stored; the sub-update condition is that the storage priority reaches a preset priority and / or the timer is reset to zero;

[0085] Based on the same inventive concept, the present invention also provides a self-updating redundancy removal system based on big data processing, and the system is used to implement the self-updating redundancy removal method based on big data processing;

[0086] Based on the same inventive concept, the present invention also provides a server for self-updating and de-redundancy based on big data processing, and the server is used to implement the above-mentioned method for self-updating and de-redundancy based on big data processing;

[0087] Based on the same inventive concept, the present invention also provides a self-updating redundancy removal device based on big data processing, and the device is used to implement the self-updating redundancy removal method based on big data processing;

[0088] Based on the same inventive concept, the present invention also provides a self-updating and redundancy-removing platform based on big data processing, and the platform is used to implement the self-updating and redundancy-removing method based on big data processing;

[0089] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of a programming language, including an assembled or interpreted language, a declarative or procedural language, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple collaborative files (e.g., files storing one or more modules, subroutines, or code portions). A computer program can be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network.

[0090] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0092] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0093] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A self-updating and redundancy removal method based on big data processing, characterized in that: The method comprises: Step S1: for each data entry stored in the public data platform, divide it to obtain one or more data regions; obtain the eigenvalue of each data region, and use the eigenvalue to construct a feature matrix corresponding to each data entry; each eigenvalue type corresponds to a feature matrix; Step S2: Calculate the characteristic matrix M corresponding to each eigenvalue type u entropy; select the feature matrices corresponding to the first U eigenvalue types with the largest entropy from all eigenvalue types, and perform sparse processing on the feature matrices corresponding to the first U eigenvalue types to obtain U sample sparse feature matrices; Associating and storing the U characteristic value types with the data entry; The characteristic matrix corresponding to each eigenvalue type u is calculated The entropy S u Specifically, the entropy S is calculated using the following formulas (1) to (3): u ; Where: u = 1 to U; (i, j) is the element number in the feature matrix; is the eigenvalue at position (i, j); The sparse processing is performed to obtain U sample sparse feature matrices MS u ; Specifically: for each feature type u, the feature matrix M corresponding to u , in the feature matrix M u Delete some elements from the sparse window so that when the sparse window slides arbitrarily in the feature matrix, any deleted element in the sparse window exists and has at least XS adjacent elements that are not deleted, except for the boundary elements; the element values ​​of some deleted elements are set to 0; Step S3: receiving the data to be stored and dividing it to obtain one or more data regions; obtaining the eigenvalues ​​of each data region, and using the eigenvalues ​​to construct a feature matrix corresponding to different eigenvalue types of the data entry; performing sparse processing on the feature matrix to obtain a current sparse feature matrix MC corresponding to the data to be stored u ; Step S4: for each data entry of the same data type as the data to be stored, compare the U eigenvalue types and the corresponding sample sparse feature matrices associated with each data entry with the current sparse feature matrix of the corresponding eigenvalue type of the data to be stored; when there is a data entry such that the sample sparse feature matrices corresponding to each eigenvalue type are similar to the current sparse feature matrix of the corresponding eigenvalue type, use the existing data entry as the data entry of the data to be determined; Step S5: completing the eigenvalues ​​of the current sparse feature matrix of the data to be stored to obtain a feature completion matrix; comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update to remove redundancy; The method of performing eigenvalue completion based on the current sparse feature matrix of the data to be stored to obtain a feature completion matrix is ​​specifically as follows: a sparse window is made to slide in the feature matrix, and elements in the feature matrix in the sparse window that have element values ​​in the sparse feature matrix are used as input data in the completion model, and output data of the completion model are used as completion eigenvalues ​​to complete corresponding positions of elements that do not exist in the sparse window to obtain a feature completion matrix; The step of comparing the feature completion matrix with the feature matrix of the data to be determined to determine whether to perform self-update and redundancy removal is specifically as follows: comparing the feature completion matrix with the feature matrix of the data to be determined, and when the feature completion matrix corresponding to each eigenvalue type is similar to the feature matrix of the data to be determined of the corresponding eigenvalue type, directly performing self-update and redundancy removal, and deleting the data to be stored; otherwise, returning to step S3 to process the next data to be stored.

2. The self-updating redundancy removal method based on big data processing according to claim 1 is characterized in that: The division method is related to the type of data to be stored.

3. The self-updating and redundancy removal method based on big data processing according to claim 2 is characterized in that: The types of data to be stored include structured, semi-structured and / or unstructured big data.

4. The self-updating and redundancy removal method based on big data processing according to claim 3 is characterized in that: The types of data to be stored include images, voices, texts and / or videos.

5. The self-updating and redundancy removal method based on big data processing according to claim 4 is characterized in that: The eigenvalues ​​of each data region are of one or more types, and each eigenvalue type corresponds to a feature matrix.

6. The self-updating redundancy removal method based on big data processing according to claim 5 is characterized in that: The division is a semantically independent data segmentation.

7. An artificial intelligence public data platform based on big data processing, self-updating and redundancy removal, characterized in that: The artificial intelligence public data platform is used to implement the self-updating and redundancy removal method based on big data processing as described in any one of claims 1 to 5.

8. An artificial intelligence public data server based on big data processing, self-updating and redundancy removal, characterized in that: It includes a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the self-updating redundancy removal method based on big data processing described in any one of claims 1 to 5 is implemented.

9. A self-updating redundancy removal system based on big data processing, characterized in that: The system is used to implement the self-updating redundancy removal method based on big data processing as described in any one of claims 1-5.

10. A computer-readable storage medium, characterized in that: It includes a program which, when executed on a computer, enables the computer to execute the self-updating redundancy removal method based on big data processing as described in any one of claims 1 to 5.

Images