A method for identity-based industrial data delta registration

Abstract

The application discloses a kind of industrial data increment registration based on identification method, its steps include: 1) industrial data increment registration system is assigned to each registered user unique user id;2) whether the corresponding user is first registered by user id determination, if first registration, the data in the database of the user is registered with full data;If not first registration, the database of the user is monitored, and the increment data registration frequency of the user is calculated according to the data change in the database setting period N days;Then the database is registered with increment data at frequency;3) when the registration time of the user exceeds the set duration, the registration frequency of the next period is predicted according to the recent increment data of the database periodically, and then the database is registered with increment data at frequency in the next period.The application can effectively register increment data.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology and relates to a method for incremental registration of industrial data based on identifiers. Background Technology

[0002] Industrial data is crucial to industrial development. A reasonable and effective registration method for industrial data can increase enterprises' enthusiasm for data registration, thereby promoting the future development of industrial data governance. Currently, very few mainstream registration systems are designed for incremental registration.

[0003] The current status of incremental industrial data registration in my country reveals the following problems: First, most enterprises, considering the timeliness of data, hope for a system that can ensure data security and automatically track and register incremental data in real time; second, there is currently no mature system for registering incremental industrial data; and third, for some enterprises with large amounts of data, the registration time is long, relying on manual estimation of the registration frequency.

[0004] Therefore, there is an urgent need for an identifier-based method for incremental registration of industrial data that can automatically track incremental data in the database in real time and automatically adjust the registration frequency according to the amount of data. Summary of the Invention

[0005] To address the problems existing in the prior art, the purpose of this invention is to provide a method for incremental registration of industrial data based on identifiers, achieving data registration without leaving the database. During the data registration process, the database data is divided into existing data and incremental data using timestamps; incremental data that has not been previously registered is registered instead of re-registering all data.

[0006] This invention effectively solves the problem of registering real-time, growing, and valid data, avoiding the issue of duplicate registration of existing data. It shifts enterprise data registration from batch and delayed processes to streaming and real-time processes, accelerating the business loop. Simultaneously, incremental registration is a crucial step for enterprises moving from extensive data storage to refined data operation, further encouraging more enterprises to participate in data registration. In the long run, establishing methods and systems for incremental data registration also lays the foundation for building high-quality datasets.

[0007] This invention provides a method for real-time tracking of incremental industrial data registration, which involves periodically monitoring incremental data and employing a registration method that automatically adjusts the registration frequency based on the data volume. The method includes: Multi-dimensional integration: It considers both data volume trends and system load and business cycle.

[0008] Smooth response: Avoid frequency abrupt changes by using smoothing functions and exponential smoothing.

[0009] Forward-looking adjustments: By using a forecasting module to anticipate data fluctuations, registration delays caused by sudden peaks can be reduced.

[0010] Adaptive Business: The business cycle coefficient C_business can be dynamically set according to factors such as enterprise production schedule and holidays.

[0011] This invention protects the security of original data by encrypting the header of industrial data tables to generate codes and then registering them.

[0012] The technical solution of this invention is a method for incremental registration of industrial data based on identifiers, the steps of which include: 1) The industrial data incremental registration system assigns a unique user ID to each registered user; 2) The industrial data incremental registration system determines whether a user is registering for the first time by using the user ID of the currently logged-in user. If it is, the system performs a full data registration of the user's database. If it is not, the system monitors the user's database and calculates the incremental data registration frequency based on the data changes within a set period of N days. Then register the frequency using incremental data. Register incremental data in the database; 3) Once a user's registration period exceeds a set duration, the registration frequency for the next N days will be predicted periodically based on the most recent incremental data in the database. Then, within the next cycle N days, the registration frequency will be used. Incremental data registration is performed on the database.

[0013] Preferably, the incremental data registration frequency ;in, This represents the number of incremental data entries on day i within a given period of N days. Let be the total registration duration on day i, where i = 1 to N.

[0014] Preferably, the cycle is set to 5 days.

[0015] Preferably, the registration frequency is calculated. The method is as follows: 1) Collect the number of new data entries Dt added to the database within the time window t, and the total time spent completing the registration within the time window t. Normalized CPU utilization of the registration plugin Network latency normalization value ; Calculate the registered load And predict the data increment within the next time window (t+1). ;in, This is the moving average over the past m time windows. This is a seasonal factor calculated based on historical data from the same period. , , For the weighting coefficients, satisfying + + =1; 2) Calculate the frequency of dynamic registration ;in, For the set load threshold, (·) represents the hyperbolic tangent function, used to smooth the effects of load. The set business cycle coefficient; 3) Frequency of dynamic registration Exponential smoothing is obtained ;in, The registration interval for the previous cycle, It is a smoothing factor; 4) According to =max( Determine the registration frequency ;in, The minimum time interval is set. This is the maximum time interval set.

[0016] Preferably, after a user registration period of more than 7 calendar days, the registration frequency for the next period (N days) is predicted based on the historical incremental data of the database. .

[0017] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the methods described above.

[0018] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program implements the above-described method when executed by a processor.

[0019] The advantages of this invention are as follows: By using the identifier-based incremental industrial data registration method provided by this invention, incremental data can be effectively registered, effectively adapting to the data registration needs of various industries, increasing the data registration volume, and eliminating concerns about data leakage and security issues caused by data leaving the database. Furthermore, the registration system automatically adjusts the registration frequency based on the incremental user data, eliminating the need for manual changes or monitoring.

[0020] This patent improves the efficiency of video data registration while ensuring security. Compared to other methods, this patent's video data registration method encrypts database table header names to generate ciphertext and registers the ciphertext instead of the data itself. This method can both understand user data to adapt to the business scenario and ensure data confidentiality. It is highly adaptable to various industries and is a more suitable incremental data registration method and system for actual industrial enterprises. Attached Figure Description

[0021] Figure 1 This is a flowchart of the incremental data registration method of the present invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0023] The method flow of this invention is as follows: Figure 1 As shown, firstly, users (enterprises) register an account in this industrial data incremental registration system and, after being assigned a unique user ID, can register data. During registration, the system will determine whether the user is registering for the first time based on the user ID. If it is the first time registering, the system will register the existing data (all data); if it is not the first time registering, the system will read the timestamp of the most recent registration based on the calculated registration frequency as the start time (the timestamp format is: yyyy-MM-dd HH:mm:ss.SSS, i.e., "year-month-day hour:minute:second.millisecond"), and perform incremental data registration on the data generated after that start time in the user database.

[0024] The system defaults to a 5-day cycle and automatically calculates the basic update frequency of data. The calculation method is: Incremental data registration frequency (registered once every N hours) = Total number of incremental data entries in the last five days / Total registration time in the last five days (in minutes) * (Maximum value among incremental data in the last five days) / 60. This gives the interval in hours for incremental registration. The specific calculation formula is as follows: in, Registration interval (hours) Let i be the number of incremental data entries for day i. The total registration duration (in minutes) for day i.

[0025] The system features dynamic frequency adjustment and forecasting capabilities, which predict the growth trend of new data through time series analysis. This function will be activated after a user has used the system for more than seven calendar days, providing data forecasts. The specific implementation method and formula are as follows: 1. Sample Collection: The system defaults to a 5-day statistical period. Let Dt represent the number of new data entries (i.e., incremental data volume) within the time window t (e.g., one day). This represents the total time (minutes) taken to complete registration within the time window t, the CPU usage of the registration plugin, and the network latency (seconds).

[0026] 2. Forecast: Calculated using trend forecasting formulas. .

[0027] The data increment trend is predicted over a future period using a weighted moving average plus a seasonal factor. The formula for calculating the data increment trend prediction is as follows: in, The predicted incremental data volume for the next time slice. This is the moving average over the past m time slices. This is a seasonal factor, calculated based on historical data from the same period last week (e.g., the same time period last week). , , For the weighting coefficients, satisfying + + =1.

[0028] 1. Assessment: Calculate the current load using the registered load assessment function. The load evaluation function is registered as follows: in, Normalization of CPU usage for registered plugins; This is a normalization process for network latency (in seconds).

[0029] 2. Adjustment: Substitute into the dynamic frequency formula for calculation. The dynamic frequency adjustment formula is as follows: in, For the load threshold, (·) represents the hyperbolic tangent function, used to smooth the effects of load. This is the business cycle coefficient (e.g., 0.8 for peak production periods and 1.2 for off-peak periods).

[0030] 3. Smoothing: Obtained through exponential smoothing. The formula is as follows: in, The registration interval for the previous cycle, This is a smoothing factor (e.g., 0.3).

[0031] 4. Execution: In accordance with The registration interval is constrained to be within a reasonable range, and the trigger time for the next incremental data registration task is set. The specific constraint method is as follows: =max( ) For example: =0.5 hours (minimum interval) Hours (maximum interval).

[0032] Next, users registering for the first time or those making incremental registrations with adjustments to the database table structure will proceed to the semantic normalization module. Users can download the dictionary table template provided on this page, fill in the corresponding information, and upload it. This dictionary table will default to the field content of this user data registration. For users registering for the first time or whose database table structure has not been adjusted, the system will automatically register incremental data. The dictionary table template is as follows: Based on the dictionary table uploaded by the enterprise user, the system will generate a standardized dictionary table according to the Chinese names and meanings of the fields in the table. Finally, it will automatically add semantic dictionaries that are not present in its own semantic database. An example of the content of the dictionary table uploaded by the user is shown below: beautyco product_detail product_name Product Name varchar(255) Name of the product for sale The system-normalized dictionary table: product_name Product Name productName CPMC yes The system stores the complete normalized dictionary table in the user's information database. The normalized dictionary table will be automatically imported when the user registers data.

[0033] Finally, the system will register the corresponding data through the data registration module. Users can register data in a non-domain mode by downloading the registration plugin in the system, ensuring data security. The registration plugin is installed on the same server as the user's database. The plugin will search the corresponding database, table, and field names using the dictionary table previously filled in by the user and calculate the duplication rate (two data entries in a data table are considered duplicates if their contents are exactly the same) and the null value rate (a data entry is considered null if a column has an empty or NULL value).

[0034] The system will use the database, table, and field names from the dictionary as the registration content, rather than the user's data content, thus ensuring user data security from the source. The system will encrypt the registration content using the SM4 national cryptographic algorithm to generate 32-bit ciphertext. This ciphertext will serve as an identifier for the data registration and will be uploaded to the system.

[0035] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for incremental registration of industrial data based on identifiers, comprising the following steps: 1) The industrial data incremental registration system assigns a unique user ID to each registered user; 2) The industrial data incremental registration system determines whether a user is registering for the first time by using the user ID of the currently logged-in user. If it is the first time registering, the system will register all data in the user's database. If this is not the first registration, monitor the user's database and calculate the user's incremental data registration frequency based on data changes within a set period of N days in the database. Then register the frequency using incremental data. Register incremental data in the database; 3) Once a user's registration period exceeds a set duration, the registration frequency for the next N days will be predicted periodically based on the most recent incremental data in the database. Then, within the next cycle N days, the registration frequency will be used. Incremental data registration is performed on the database; the registration frequency for the next N days is predicted. The method is as follows: 31) Collect the number of new data entries D added to the database within the time window t. t The total time spent completing registration within the time window t. Normalized CPU utilization of the registration plugin Network latency normalization value ; Calculate the registered load And predict the data increment within the next time window (t+1). ;in, This is the moving average over the past m time windows. This is a seasonal factor calculated based on historical data from the same period. , , For the weighting coefficients, satisfying + + =1; 32) Calculate the dynamic registration frequency ;in, For the set load threshold, (·) represents the hyperbolic tangent function, used to smooth the effects of load. 33) The set business cycle coefficient; 33) The dynamic registration frequency Exponential smoothing is obtained ;in, The registration interval for the previous cycle, As a smoothing factor; 34) According to =max( Determine the registration frequency ;in, The minimum time interval is set. This is the maximum time interval set.

2. The method according to claim 1, characterized in that, Incremental data registration frequency ;in, This represents the number of incremental data entries on day i within a given period of N days. Let be the total registration duration on day i, where i = 1 to N.

3. The method according to claim 1, characterized in that, The cycle is set to 5 days.

4. The method according to claim 1, 2, or 3, characterized in that, If a user registration period exceeds 7 calendar days, the registration frequency for the next period (N days) is predicted based on the historical incremental data of this database. .

5. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 4.

Images