A method, system and storage medium for continuous data clustering to save storage space

By calculating and comparing the standard deviation number of data, data clustering and storage in low-performance and low storage space environments are realized, the problems of data loss and performance decline are solved, and the information expression efficiency and storage are improved ability.

CN117311643BActive Publication Date: 2024-11-01CHINA CONSTRUCTION THIRD ENGINEERING BUREAU GENERAL CONTRACTING CONSTRUCTION CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311406267.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-10-27
Publication Date
2024-11-01
Estimated Expiration
2043-10-27

AI Technical Summary

Technical Problem

The prior art has problems such as insufficient storage space, data loss and performance degradation in the data clustering process. Traditional storage strategies reduce space occupation through compression and deletion but reduce user access efficiency.

Method used

By recording individual data, add new categories and store category parameters, calculate standard deviations and compare them, find the category corresponding to the smallest standard deviation number. If the standard deviation number is ≤2, modify the category parameters and add new categories to add data. Recording, using entropy load maximum as the result of the data packet structure for storage.

Benefits of technology

The clustering and storage of continuous input data is realized in a low-performance and low-storage environment, which is suitable for various low-performance and low-storage environments, improving the information expression efficiency and storage ability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117311643B_ABST
    Figure CN117311643B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, system and storage medium for continuous data clustering that saves storage space, and the method steps are as follows: record a single data, add and store a category, and record category parameters: according to a number of recorded data X, perform calculations with each category in a loop, calculate the standard deviation and compare, and find the category corresponding to the smallest standard deviation; make a judgment based on the obtained standard deviation, and if the standard deviation is ≤2, modify the current category parameters; add a category, and add a single data record to this category. Beneficial effects of the present invention: The present invention is suitable for continuous data collection in various low-performance, low-storage environments; it realizes clustering of continuously input continuous data and storing the results in a low-performance, low-storage space environment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of data clustering, and in particular to a method, system and storage medium for continuous data clustering that saves storage space. Background Art

[0002] In the era of the Internet of Things, a large amount of data collected by sensors needs to be processed by artificial intelligence, among which data clustering is one of the basic processing methods of machine learning. However, due to the limitations of the processing power and storage space of IoT devices, these data often cannot be stored and processed in a timely manner.

[0003] 1. For example, a Chinese patent discloses a data processing method, device, equipment and computer-readable storage medium (publication number: CN113434471A), which includes: obtaining target data to be processed, the target data includes several data items; based on a clustering algorithm model, classifying the several data items into at least one data group; determining the current center value of each data group according to the data items currently contained in each data group; obtaining the historical center value of each data group, and determining the target data group according to the current center value and the historical center value of each data group; storing the data items in the target data group to the target storage address. It is possible to retain historical data while reducing the storage space occupied when data changes. The present application also relates to blockchain technology, and the target data group can be stored in the blockchain.

[0004] 2. A data clustering method, system and storage medium (publication number: CN113806610A), comprising the following steps: determining data clustering conditions; clustering data according to the data clustering conditions to obtain at least one first clustering result, and calculating the entropy load of each first clustering result; the entropy load represents the size of the average amount of information carried by the corresponding first clustering result; taking the maximum entropy load among the entropy loads, and the corresponding first clustering result is the data clustering result. The present invention performs clustering based on the overall data, realizes the integrity of data clustering, and the obtained clustering results are more complete and accurate; and there is no dependence on and processing of any special data in the clustering process, and no restrictions on any data type, so it is generally applicable to the clustering of any data; the maximum average amount of information carried is used as the basis for determining the clustering results. For a computer system with a certain storage space, the amount of information that can be stored is larger, which improves the efficiency of information expression.

[0005] The clustering algorithms in the existing technology can effectively process data sets with low dimensions and small numbers. However, as the amount of data increases, many clustering algorithms exceed the capabilities of ordinary computers in terms of computing power and memory consumption and can no longer process them.

[0006] There are some problems with the current related technologies:

[0007] 1. There are problems such as data loss and performance degradation during the storage process;

[0008] 2. Traditional storage strategies use compression, deduplication and other technologies to reduce the space occupied by data. However, these technologies will reduce user access efficiency.

[0009] Therefore, it is necessary to propose a data continuous clustering method, system and storage medium that saves storage space to solve the above problems. Summary of the invention

[0010] In view of the above-mentioned deficiencies in the prior art, an object of the present invention is to provide a data continuous clustering method, system and storage medium that saves storage space to solve the above-mentioned problems.

[0011] A method for continuous data clustering that saves storage space, the method steps are as follows:

[0012] S1. Record a single piece of data, add and store a category, and record the category parameters;

[0013] S2. Based on the recorded data, loop and calculate each category, calculate the standard deviation and compare them, and find the category corresponding to the smallest standard deviation;

[0014] S3, judging based on the obtained standard deviation, if the standard deviation is ≤ 2, modifying the current category parameters;

[0015] S4. Add a new category and add a single data record to this category.

[0016] The category parameters in step S1 include mean value U, standard deviation S and quantity N.

[0017] In step S1, the average value of the category parameter = input value, the standard deviation S = 0, and the number N = 1.

[0018] The data recording method in step 1 is:

[0019] S11, encapsulating the recorded data of the application system into a data packet structure that complies with a unified specification through the data update layer;

[0020] S12, encapsulating the data packet structure into a message of a unified format through a message processing layer;

[0021] S13, sending the message to a designated storage device through an interface processing layer;

[0022] S14. Take the maximum entropy load among the entropy loads corresponding to each data packet structure result, and the data packet structure result corresponding to the maximum entropy load is the data recording result.

[0023] The calculation method of step S2 is as follows: S21, calculate the standard deviation D = |(XU) / S|, if S = 0, directly record the result as D = ∞; S22, compare the D of each category, and find the category corresponding to the smallest D; wherein D is the standard deviation, X is the data, U is the average value; S is the standard deviation.

[0024] The limit of the standard deviation number in step S3 can be adjusted according to actual needs.

[0025] The category parameter modification formula in step S3 is:

[0026]

[0027] Among them, U is the mean, X is the number of data, S is the standard deviation, and N is the quantity.

[0028] The standard deviation ranges from 0.1 to 3.

[0029] A data continuous clustering system that saves storage space, the system comprises a memory, a processor and a program stored in the memory and executable on the processor, the program implementing the steps of the above-mentioned clustering method when executed by the processor.

[0030] A computer-readable storage medium stores at least one program, which can be executed by at least one processor. When the at least one program is executed by the at least one processor, the steps of the above-mentioned data continuous clustering method are implemented.

[0031] Compared with the prior art, the present invention has the following beneficial effects: the present invention is applicable to continuous data collection in various low-performance, low-storage environments; and realizes clustering of continuously input continuous data and storage of results in low-performance, low-storage space environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method for continuous data clustering to save storage space of the present invention;

[0033] Figure 2 It is a flow chart of the data recording method of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0035] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0036] like Figure 1 Combined with Figure 2 As shown, a method for continuous data clustering that saves storage space, the method steps are:

[0037] S1. Record a single piece of data, add and store a category, and record the category parameters;

[0038] S2. Based on the recorded data, loop and calculate each category, calculate the standard deviation and compare them, and find the category corresponding to the smallest standard deviation;

[0039] S3, judging based on the obtained standard deviation, if the standard deviation is ≤ 2, modifying the current category parameters;

[0040] S4. Add a new category and add a single data record to this category.

[0041] The category parameters in step S1 include mean value U, standard deviation S and quantity N.

[0042] In step S1, the average value of the category parameter = input value, the standard deviation S = 0, and the number N = 1.

[0043] The data recording method in step 1 is:

[0044] S11, encapsulating the recorded data of the application system into a data packet structure that complies with a unified specification through the data update layer;

[0045] S12, encapsulating the data packet structure into a message of a unified format through a message processing layer;

[0046] S13, sending the message to a designated storage device through an interface processing layer;

[0047] S14. Take the maximum entropy load among the entropy loads corresponding to each data packet structure result, and the data packet structure result corresponding to the maximum entropy load is the data recording result.

[0048] Data packets all contain designated fields and data areas, etc. The unified and standardized data packet structure can be defined by the data structure of the corresponding interface and program. Messages in a unified format refer to messages that meet the same designated rules, such as including a designated message part of a designated length. In the embodiment of the present invention, the unified and standardized data packet structure means that when different data packets are generated or transmitted, their structure can meet the same predetermined rules, but it does not limit all data packets to adopt the same structure.

[0049] The recorded data is divided into two layers according to the recording protocol. The recorded data is encapsulated and sent layer by layer through different modules (layers), which is convenient for adjusting the implementation of data recording methods according to different recording types or data sending requirements without modifying the overall recording function, so as to adapt to the generation and transmission of recorded data in various scenarios. Each link of data recording is also loosely coupled and independently controlled, with clear logic, which is convenient for each layer to make full use of resources and optimize.

[0050] The data packet structure consists of two parts, the data packet header and the data packet content. The data packet header includes the packet ID (NID_PACKET) and the packet length (L_PACKET). The packet ID should be unique and must be different between different data packets. The data packet content can be filled as needed and is divided into two types. The first type of data packet content has the following expression structure for the field variable record. The data packet content includes multiple fields as needed.

[0051] The calculation method of step S2 is as follows: S21, calculate the standard deviation D = |(XU) / S|, if S = 0, directly record the result as D = ∞; S22, compare the D of each category, and find the category corresponding to the smallest D; wherein D is the standard deviation, X is the data, U is the average value; S is the standard deviation.

[0052] The limit of the standard deviation number in step S3 can be adjusted according to actual needs.

[0053] The category parameter modification formula in step S3 is:

[0054]

[0055] Among them, U is the mean, X is the number of data, S is the standard deviation, and N is the quantity.

[0056] The standard deviation ranges from 0.1 to 3.

[0057] A data continuous clustering system that saves storage space, the system comprises a memory, a processor and a program stored in the memory and executable on the processor, the program implementing the steps of the above-mentioned clustering method when executed by the processor.

[0058] A computer-readable storage medium stores at least one program, which can be executed by at least one processor. When the at least one program is executed by the at least one processor, the steps of the above-mentioned data continuous clustering method are implemented.

[0059] Compared with the prior art, the present invention has the following beneficial effects: the present invention is applicable to continuous data collection in various low-performance, low-storage environments; and realizes clustering of continuously input continuous data and storage of results in low-performance, low-storage space environments.

[0060] The calculation method of entropy load is:

[0061]

[0062]

[0063] Among them, a mj For the sequence {a mj}, {a mj} is the data v of the jth dimension j The difference between them is arranged in ascending order. a is the base of the logarithmic function. a>1, and the entropy load is Ia mj Indicates v j Take the sequence {a mj} the mth item a mj The average amount of information carried by the first clustering result obtained by clustering; n is v j Take the sequence {a mj} the mth item a mj The number of data sets contained in the first clustering result obtained by clustering; k i is the number of elements in the i-th data set, N is the total number of data, p i It is the ratio of the number of elements in the i-th data set to the total number of data.

[0064] The result of each clustering is several data sets, each of which corresponds to a data category. When the computer system stores the clustering results, each data category has a corresponding fixed-length code. The average amount of information that each code can store is certain, and the information expression efficiency of each code is also certain. It is expected that the fixed-length code can store the largest average amount of information, thereby achieving the highest information expression efficiency.

[0065] Entropy Load Indicates the average amount of information carried by the clustering results obtained in this clustering. The larger the value, the greater the average amount of information in each data category in this clustering result. The larger the average amount of information that can be stored in the encoding corresponding to each data category, the higher the information expression efficiency of the encoding corresponding to each data category. For a computer system with a certain storage space, the larger the amount of information that can be stored, the higher the efficiency of its information expression.

[0066] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for continuous data clustering that saves storage space, characterized by: The method steps are: S1. Record a single piece of data, add and store a category, and record the category parameters; S2. Based on the recorded data, loop and calculate each category, calculate the standard deviation and compare them, and find the category corresponding to the smallest standard deviation; S3, making a judgment based on the obtained standard deviation number, if the standard deviation number is ≤ 2, modifying the current category parameter; S4. Add a new category and add a single data record to this category; The data recording method in step S1 is: S11, encapsulating the recorded data of the application system into a data packet structure that complies with a unified specification through the data update layer; S12, encapsulating the data packet structure into a message of a unified format through a message processing layer; S13, sending the message to a designated storage device through an interface processing layer; S14, taking the maximum entropy load among the entropy loads corresponding to each data packet structure result, and the data packet structure result corresponding to the maximum entropy load is the data recording result; The calculation method of entropy load is: Among them, a mj For the sequence {a mj }, {a mj } is the data v of the jth dimension j The difference between them is arranged in ascending order. a is the base of the logarithmic function. a>1, and the entropy load is Ia mj Indicates v j Take the sequence {a mj } the mth item a mj The average amount of information carried by the first clustering result obtained by clustering; n is v j Take the sequence {a mj } the mth item a mj The number of data sets contained in the first clustering result obtained by clustering; k i is the number of elements in the i-th data set, N is the total number of data, p i is the ratio of the number of elements in the i-th data set to the total number of data; Entropy Load Indicates the average amount of information carried by the clustering results obtained in this clustering. The larger the value, the larger the average amount of information of each data category in the clustering result. The larger the average amount of information that can be stored in the code corresponding to each data category, the higher the information expression efficiency of the code corresponding to each data category. For a computer system with a certain storage space, the larger the amount of information that can be stored. The category parameters in step S1 include a mean value U, a standard deviation S, and a quantity N; wherein the mean value of the category parameters in step S1 = the input value, the standard deviation S = 0, and the quantity N = 1; The calculation method of step S2 is as follows: S21, calculate the standard deviation number D = |(XU) / S|, if S = 0, directly record the result as D = ∞; S22, compare the D of each category, and find the category corresponding to the smallest D; where D is the standard deviation number, X is the data, U is the average value; S is the standard deviation; The limit of the standard deviation number in step S3 is adjusted according to actual needs; the category parameter modification formula in step S3 is: Where U is the mean, X is the number of data, S is the standard deviation, and N is the quantity; the standard deviation ranges from 0.1 to 3.

2. A data continuous clustering system that saves storage space, characterized in that: The system includes a memory, a processor, and a program stored in the memory and executable on the processor. When the program is executed by the processor, the steps of the clustering method according to claim 1 are implemented.

3. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program can be executed by at least one processor. When the at least one program is executed by the at least one processor, the steps of the data continuous clustering method according to claim 1 are implemented.