A Hive-based data updating method and device and readable storage medium

Abstract

The application discloses a Hive-based data updating method and device and a readable storage medium. After updating data records in a target data table, version information of the target data table is recorded. When the version information meets preset conditions, valid data records that are not deleted are acquired, and a unique identification value (UUID) and a latest operation version number of each row of the valid data records are inquired. Target data that needs to be reserved is further determined, and the operation version number is reset. A temporary data table is generated to replace historical data in the target data table, and the version information of the target data table is reset. Since the version information of the data table is monitored in real time after a data updating operation is performed, once the preset conditions are met, redundant data in the data table is automatically cleaned, and the version information of the data table is initialized. Therefore, in the case of long-term and continuous data updating, the calculation complexity can be ensured to be always within a reasonable range, so that data updating is realized while the data processing efficiency is ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a Hive-based data update method, apparatus, and readable storage medium. Background Technology

[0002] In the era of rapid development of big data, a data warehouse tool called Hive is often used to solve the problem of data statistics for massive amounts of structured logs because it can support large-scale data processing. Hive is a data warehouse analysis system built on Hadoop, which provides rich SQL query methods for analyzing data stored in the Hadoop Distributed File System. Because Hive is built on a cluster and can use MapReduce for parallel computing, it has significant advantages when processing large amounts of data.

[0003] However, because Hive is based on HDFS (Hadoop Distributed File System), which does not support modification of existing data files, Hive currently has poor support for data update operations such as update and delete. When modifying a single row of data, the entire table or partition must be deleted and the entire table regenerated, making data modification very costly. Current Hive data modification methods aim to address this cost issue and avoid deleting existing data by grouping each data entry in Hive and introducing an ingestion timestamp. Updates are achieved by calling the most recently ingested data within a group. However, after long-term operation and frequent data updates, redundant data in each group continuously increases, leading to increased computational complexity and decreased efficiency when calling the latest data. Summary of the Invention

[0004] In view of this, this application provides a data update method, apparatus and readable storage medium based on Hive, which is used to solve the problem that after continuous updates to data in Hive over a long period of time, the redundant data stored in Hive increases, leading to increased computational complexity and reduced computational efficiency.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] A Hive-based data update method includes:

[0007] Receive a data update operation instruction, determine the target data record to be updated in the target data table, wherein the data update operation instruction is a data modification instruction or a data deletion instruction;

[0008] The target data record is updated according to the data update operation instruction, and the version information of the target data table is updated. The version information is used to record the number of updates or update information of the target data table.

[0009] Determine whether the updated version information meets the preset conditions;

[0010] If the updated version information meets the preset conditions, then query the UUID and latest operation version number of each valid data record in the target data table. The UUID is used to uniquely identify a valid data record, and the latest operation version number is used to identify the number of times the valid data record has been updated. The valid data record is a data record for which the data deletion instruction has not been executed.

[0011] Obtain the target data corresponding to the latest operation version number of the valid data record, set an initial operation version number for the target data, and generate a temporary data table. The temporary data table contains the target data, the UUID, and the initial operation version number.

[0012] Clear the target data table, insert the temporary data table into the target data table, and initialize the version information of the target data table.

[0013] Optionally, before determining whether the updated version information meets the preset conditions, the method further includes:

[0014] Receive the command to add data and retrieve the new data record;

[0015] Determine the UUID value and initial operation version number of the newly added data record;

[0016] Execute the new data instruction, record the new data record, the UUID value, and the initial operation version number in the target data table, and update the version information of the target data table.

[0017] Optionally, it also includes:

[0018] Receive a data query instruction, determine a valid data set, the valid data set contains data information corresponding to the latest operation version number of each record in the target data table, and the data query instruction includes query conditions;

[0019] Execute the query data instruction to search for data information that meets the query conditions in the valid data set, obtain the target set and output it.

[0020] Optionally, updating the target data record according to the data update operation instruction includes:

[0021] When the data update operation instruction is a data modification instruction, the data modification instruction is executed to obtain the modified updated data record;

[0022] Record the updated data in the target data table;

[0023] Determine and record the UUID and operation version number of the updated data record;

[0024] or,

[0025] When the data update operation instruction is a data deletion instruction, the operation version number of the target data record is set to a preset value.

[0026] Optionally, it also includes:

[0027] Check whether the preset value is included in the first operation version number, where the first operation version number is the historical operation version number of the data record in the target data table after the update operation is completed;

[0028] If the first operation version number does not contain the preset value, then the data record is determined as the valid data record.

[0029] Optionally, determining whether the updated version information meets preset conditions includes:

[0030] Determine whether the number of update operations completed in the target data table has reached a first preset threshold;

[0031] And / or,

[0032] Determine whether the total number of data records in the target data table reaches a second preset threshold;

[0033] And / or,

[0034] Determine whether the time recorded at the current moment has reached the preset time, where the time recorded at the current moment is the time when the target data table version information was last initialized.

[0035] Optionally, it also includes:

[0036] The version information of the target data table is recorded in the management record table, which is used to manage the version of the data tables in the Hive library.

[0037] A Hive-based data update device includes:

[0038] The data record query unit is used to receive data update operation instructions and determine the target data record to be updated in the target data table. The data update operation instructions are either data modification instructions or data deletion instructions.

[0039] The data record update unit is used to update the target data record according to the data update operation instruction, and update the version information of the target data table. The version information is used to record the number of updates or update information of the target data table.

[0040] The first judgment unit is used to determine whether the updated version information meets the preset conditions;

[0041] The update information query unit is used to query the UUID and latest operation version number of each valid data record in the target data table when the updated version information meets the preset conditions. The UUID is used to uniquely identify a valid data record, and the latest operation version number is used to identify the number of times the valid data record has been updated. The valid data record is a data record for which the data deletion instruction has not been executed.

[0042] A temporary data table generation unit is used to obtain the target data corresponding to the latest operation version number of the valid data record, set an initial operation version number for the target data, and generate a temporary data table, wherein the temporary data table contains the target data, the UUID and the initial operation version number;

[0043] The target data table reset unit is used to clear the target data table, insert the temporary data table into the target data table, and initialize the version information of the target data table.

[0044] Optionally, before the first determination unit, the system further includes:

[0045] The data record acquisition unit is used to receive new data instructions and acquire new data records.

[0046] The update identifier setting unit is used to determine the UUID value and initial operation version number of the newly added data record;

[0047] The data recording and input unit is used to execute the new data instruction, record the new data record, the UUID value and the initial operation version number in the target data table, and update the version information of the target data table.

[0048] Optionally, it also includes:

[0049] The valid data query unit is used to receive a query data instruction and determine a valid data set, wherein the valid data set contains data information corresponding to the latest operation version number of each record in the target data table, and the query data instruction includes query conditions.

[0050] The target set query unit is used to execute the query data instruction, search for data information that meets the query conditions in the valid data set, obtain the target set, and output it.

[0051] A readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned Hive-based data update method.

[0052] As can be seen from the above technical solutions, the Hive-based data update method provided in this application updates the target data records to be updated in the target data table according to the received data update operation instructions, and updates the version information that records the update count or update information of the target data table. When the updated version information meets the preset conditions, it obtains the valid data records that have not been deleted from the target data table, queries the unique identifier value (UUID) and the latest operation version number of each valid data record, further determines the target data to be retained in the target data table through the latest operation version number, sets the initial operation version number for the target data, generates a temporary data table, and finally clears the target data table and inserts it into the temporary data table, while initializing the version information of the target data table, thus completing the cleanup and reset of the target data table. After completing the data update operation of the data table, this solution monitors the version information of the data table in real time. Once the preset conditions are met, it automatically cleans up redundant data in the data table and initializes the version information of the data table. In the case of long-term continuous data updates, it can ensure that the computational complexity is always within a reasonable range, thereby ensuring data processing efficiency while realizing data updates. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 A flowchart of a Hive-based data update method provided in this application embodiment;

[0055] Figures 2-4 Examples of data update results for several Hive data tables are shown in the diagram.

[0056] Figure 5 This is a schematic diagram of a Hive-based data update device provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0058] Investigations revealed that Hive is currently the mainstream tool for processing and analyzing massive amounts of data. Because Hive is based on the HDFS file system, it inherently does not support data rewriting. Therefore, updating data requires modifying the entire data table, resulting in very high modification costs. Existing technology has disclosed a method for updating data by using grouping and entry timestamps to identify the update status of entered data. This method retrieves the latest modified data for a group of data by calling the most recently entered data, eliminating the need to delete and regenerate the entire data table, thus reducing modification costs.

[0059] The inventors discovered that after long-term operation and frequent data updates, existing technologies fail to clean up historical data in the data table due to the continuous input of new data. This results in a continuous increase in the amount of data in the data table and excessive redundant data in each group. Consequently, the computational complexity increases and the computational efficiency decreases during operation.

[0060] In view of this, the inventors proposed a data update method based on Hive. Figure 1 The following is a flowchart illustrating a Hive-based data update method according to an embodiment of this application: Figure 1 As shown, the process may include:

[0061] Step S10: Receive the data update operation instruction and determine the target data record to be updated in the target data table.

[0062] The data update operation command can be a data modification command or a data deletion command. The data modification command means to modify the data of the target data record in the target data table, and the data deletion command means to delete all historical data of the target data record in the target data table.

[0063] Specifically, the data update operation instruction can include filtering conditions for the target data record to be updated. When a data update operation instruction is received, the target data record can be determined by searching for data records in the target data table that meet the filtering conditions. Optionally, the set of data records to be updated can also be pre-filtered to obtain the set of data records as the target data record.

[0064] Step S11: Update the target data record according to the data update operation instruction, and update the version information of the target data table.

[0065] Optionally, when the received data update operation instruction is a data modification instruction, the data modification operation begins. Specifically, the data modification instruction may include a data modification command, such as incrementing the data in the target data record by 1. By executing the data modification instruction, the modified data record is obtained, and the modified data record and the update identifier of this update are recorded in the target data table, thus realizing the modification of the target data record in the target data table. When the received data update operation instruction is a data deletion instruction, the data deletion operation begins. After determining the target data record to be deleted, the update identifier of the target data record can be set to the deletion identifier by executing the data deletion instruction, thus completing the deletion of the target data record.

[0066] The update identifier of a data record may include a UUID and an operation version number. The UUID is used to uniquely identify a data record, and the operation version number is used to identify the number of times a data record has been updated or its update status.

[0067] In one optional scenario, for newly entered data records, their UUID value can be calculated and recorded in the data table when the data record is entered, and its initial operation version number can be a preset initial value, indicating that no update operation has been performed on the data record; for modified data records, their UUID value remains unchanged and can be used as the unique UUID value of the data record, and its operation version number can be the value obtained by adding 1 to the maximum operation version number of the data record, used to record the number of modifications to the data record; for deleted data records, their UUID value remains unchanged, and their operation version number can be a preset value, which can be used as a deletion identifier to indicate that the update status of the data record is deleted.

[0068] Optionally, the version information of the target data table can be the number of update operations already completed. After each update, the update operation count is incremented by 1. Alternatively, the version information can be the total number of historical records in the target data table. After each data update operation, the version information is updated by adding the number of target data records to the previous version information. The version information can also be the current time. Preferably, the time can be recorded from the last time the target data table version information was reset. These methods enable the updating of the target data table version information.

[0069] Step S12: Determine whether the updated version information meets the preset conditions.

[0070] Optionally, the preset condition can be one or more of the following update conditions:

[0071] (1) The number of times the target data table has been updated has reached the first preset threshold.

[0072] Specifically, step S11 above has already explained that after updating the data records in the target data table, the version information of the target data table is updated immediately. The version information may include the number of update operations of the target data table. Each time the target data table is updated, the update operation count is incremented by 1. Then it is determined whether the number of update operations has reached the first preset threshold. If it has, the preset condition is met.

[0073] Optionally, the first preset threshold can be configured according to cluster resources and data usage frequency. When the number of updates reaches the first preset threshold, it indicates that after this number of data update operations, due to the continuous storage of data, the runtime computation efficiency may not reach the ideal value. In this case, it is necessary to clean up the historical data of the target data table.

[0074] (2) The total number of data records in the target data table reaches the second preset threshold.

[0075] Specifically, if the version information contains the total number of all historical records in the target data table, after each update of the target data table is completed, the total number of data records entered in the target data table is updated, and it is determined whether the total number reaches the second preset threshold. If it does, the preset condition is met.

[0076] Optionally, the first preset threshold can be configured according to cluster resources and data usage frequency. When the total number of data records reaches the second preset threshold, it indicates that the amount of data stored in the target data table has exceeded a reasonable range, and that the proportion of redundant data is increasing with the number of updates in the historical update process. At this time, it is necessary to clean up the redundant data in the target data table.

[0077] (3) The time recorded at the current moment has reached the preset time.

[0078] The current recorded time can be the time obtained from the moment the target data table version information was initialized last time (starting from midnight). Specifically, if the version information includes a recorded time, the current time is recorded each time the target data table is updated, and it is determined whether the recorded time has reached a preset time. If it has, the preset condition is met.

[0079] Optionally, the preset time can be configured based on the data update frequency and data usage frequency of the target data table. Further, optionally, by monitoring the recorded time in real time, the data table is automatically cleaned once the recorded time reaches the preset time. By recording the time elapsed since the last initialization of the target data table version information, the relationship between the recorded time and the preset time is monitored, enabling the historical data of the target data table to be cleaned periodically according to the preset time.

[0080] Preferably, in order to ensure that the cleanup operation does not affect the data update process, the monitoring of the recording time can be started from the time the update ends. If the preset time is reached during the data update process, the cleanup operation will not be triggered. After the update is completed, it can be performed to determine whether the version information meets the preset conditions. By determining that the time recorded at the current moment has reached the preset time, the cleanup operation is triggered, so as to realize real-time monitoring of the target data table and cleanup of historical data during idle time.

[0081] Step S13: If the updated version information meets the preset conditions, query the UUID and latest operation version number of each valid data record in the target data table.

[0082] Among them, the valid data record in the target data table can be a data record for which no deletion instruction has been executed. Optionally, whether the data record is valid can be determined by whether there is a deletion identifier. The latest operation version number can be used to identify the number of times the valid data record has been modified.

[0083] When the updated version information meets the preset conditions, it means that the target data table meets the conditions for performing the cleanup operation, and data cleanup can begin.

[0084] First, identify the valid data records. Valid data records are those that need to be retained when cleaning the data table. Specifically, this can be done by checking if a preset value or deletion identifier exists in the historical operation version number of a data record. If it exists, the data record has been deleted and is invalid; if it does not exist, the data record is valid.

[0085] Next, query the UUID value of each valid data record. Optionally, since there is a one-to-one correspondence between data records and UUID values, and each data record corresponds to a UUID value, it is possible to determine whether the UUID value is a valid data record's UUID by checking if a preset value exists in the operation version number corresponding to the UUID value.

[0086] Finally, query the latest operation version number for each valid data record. The latest operation version number can be the maximum value among the operation version numbers corresponding to the UUID values ​​of the valid data record. Optionally, each valid data record's UUID can correspond to a latest operation version number, used to determine the latest data in that valid data record, which can be the most recently modified data or newly entered data.

[0087] Step S14: Obtain the target data corresponding to the latest operation version number of the valid data record, set the initial operation version number for the target data, and generate a temporary data table. The temporary data table contains the target data, UUID, and initial operation version number.

[0088] Specifically, each valid data record corresponds to a latest operation version number. Optionally, by determining the latest operation version number of each valid data record in step S13 above, and further querying the data corresponding to the latest operation version number, the latest data entered for each valid data record can be determined, and the target data to be retained can be obtained.

[0089] Optionally, since Hive only supports update and delete operations at the table level, to modify row-level data in the target table, delete redundant parts, and retain only the target data, the determined target data can be cached in a newly created table. An initial operation version number is set for each target data entry in the new table, and the UUID obtained in step S13 is recorded in the new table, generating a temporary table. Here, the target data corresponds one-to-one with the UUID value, and the initial operation version number can be a preset initial value used to indicate that the target data in the temporary table has not undergone an update operation. The target data, its corresponding UUID value, and the initial operation version number can all be recorded in the temporary table simultaneously, generating an initial table containing the target data.

[0090] Step S15: Clear the target data table, insert the temporary data table into the target data table, and initialize the version information of the target data table.

[0091] Specifically, in order to update the target data table, all contents of the current target data table can be deleted, and the generated temporary data table can be inserted into the cleared target data table. At this time, the data in the target data table has been changed to the target data, other historical data has been deleted, and the name of the target data table has not changed. It is equivalent to replacing the contents of the target data table with the contents of the temporary data table, which will not affect the data retrieval operation.

[0092] Optionally, when inserting a temporary data table, the version information of the target data table is initialized. Specifically, the update count of the target data table can be reset to the initial value to indicate that the target data table after replacement has not undergone an update operation. The number of data records in the target data table can also be reset to the target data record count. Furthermore, the current time can be reset to zero, starting the timer from the moment the temporary data table is inserted, and the timing can be monitored.

[0093] Alternatively, a management log table can be pre-created for version management of all data tables in Hive. By recording the version information of the target data table in the management log table, version information can be monitored, and the data recorded in the management log table can be updated in real time each time the version information is updated or initialized, thus enabling version management of the target data table through visualized data.

[0094] This application embodiment updates the target data records and version information to be updated in the target data table according to the received data update operation instruction. When the updated version information meets the preset conditions, it obtains the valid data records that have not been deleted from the target data table, queries the unique identifier value (UUID) and the latest operation version number of each valid data record, further determines the target data to be retained in the target data table through the latest operation version number, generates a temporary data table and sets the initial operation version number of the target data, clears the target data table, inserts into the temporary data table, and initializes the version information of the target data table, thus completing the cleanup and reset of the target data table. After completing the data update operation of the data table, this application solution monitors the version information of the data table in real time. Once the preset conditions are met, it automatically cleans up redundant data in the data table and initializes the version information of the data table. In the case of long-term continuous data updates, it can ensure that the computational complexity remains within a reasonable range, thereby ensuring data processing efficiency while achieving data updates.

[0095] In some embodiments of this application, considering that the data update process of the target data table may also include adding data operations, this application provides an optional implementation of the adding data operation, which may include the following steps:

[0096] S1. Receive the instruction to add data and obtain the new data record.

[0097] Specifically, since adding new data does not require modifying existing data records in the target data table, it is not necessary to determine the target data record, and steps S10 and S11 above do not need to be executed. When a new data instruction is received, the instruction may contain the data information to be entered. The data information to be entered is retrieved and determined to be the new data record. Optionally, the new data record can be pre-stored in an empty data table, and when a new data instruction is received, the new data record is retrieved from the empty data table.

[0098] S2. Determine the UUID value and initial operation version number of the newly added data record.

[0099] Optionally, before adding new data records to the target data table, a unique UUID is calculated for each new data record. Simultaneously, the operation version number of each new data record is set to an initial value to indicate that no data update operation has been performed on that record. The UUID can be a randomly generated set of numbers, or it can be calculated based on data such as the current time, a counter, and a hardware identifier, where the hardware identifier can be the MAC address of the wireless network card.

[0100] S3. Execute the new data instruction to record the new data record, UUID value and initial operation version number in the target data table, and update the version information of the target data table.

[0101] Optionally, execute the new data instruction to enter the new data record, its corresponding UUID, and the initial operation version number into the target data table.

[0102] Alternatively, since the addition of data increases the total number of data records in the target data table and changes the version information, the version information of the target data table can be updated after the above-mentioned data addition operation is completed. Specifically, the number of newly added data records can be added to the previous data volume, and the update count of the target data table can be incremented by 1 to obtain the updated version information, and then the operations of steps S12-S15 above can be further executed.

[0103] In this embodiment, by receiving a new data instruction, acquiring new data records and entering them into the target data table, and setting a corresponding UUID and initial operation version number for each new data record, and updating the version information of the target data table, it is possible to add data to the target data table, making the data update function more comprehensive and meeting more Hive-based data update requirements.

[0104] In some embodiments of this application, considering that the data update process of the target data table may also include data query operations, this application provides an optional implementation of the data query operation, which may include the following steps:

[0105] S1. Receive the query data instruction and determine the valid data set.

[0106] Optionally, the valid data set includes the most recently entered data for each record and excludes deleted data. Specifically, when a data query instruction is received, by determining the latest operation version number of each record in the target data table, the corresponding data information can be identified as the most recently entered data for each record, forming a valid data set. The method for determining the latest operation version number has been described in detail in the above embodiments, specifically in step S13, and will not be repeated here.

[0107] S2. Execute the query data command to find data information that meets the query conditions in the valid data set, obtain the target set and output it.

[0108] The query data command can include query conditions. By executing the query data command, data information that meets the query conditions is filtered out from the valid data set to form a target set, and the target set is output as the query result.

[0109] In this embodiment, a valid data set is determined by querying the latest operation version number of each record in the target data table. Data information that meets the query conditions is then identified within the valid data set, yielding the query results. This ensures that the data retrieved from the target data table represents the latest data information for each record, enabling the query of the latest data in Hive. Since modifying and deleting data requires first finding data to be updated that meets the update conditions, the implementation of the data query operation provides a foundation for data updates.

[0110] Next, combined Figure 2 , Figure 3 and Figure 4 The example data update results of several Hive data tables further illustrate a Hive-based data update method disclosed in this application.

[0111] See Figure 2 , Figure 2 This example demonstrates the result of performing an insert data operation on data table A. All three rows in data table A are newly added data records.

[0112] When a new data instruction is received, the target data table is determined to be data table A, and the names, genders, and ages of the three individuals are retrieved and entered into data table A as three new data entries. Next, the UUIDs of the three data records "Zhang San," "Li Si," and "Wang Wu" are determined and recorded in data table A. Finally, the initial operation version number of each record is set to 0, and the version information of data table A is updated.

[0113] See furtherFigure 3 , Figure 3 This example demonstrates the result of performing a data modification operation on data table A. Specifically, Figure 3 The updated results presented are in Figure 2 The data records in data table A are obtained by modifying existing records.

[0114] The received data update operation instruction is the following statement:

[0115] Update table A SET age = age + 1 WHERE gender = male

[0116] By using data lookup operations, the data records in table A with the gender "male" were selected as the target data records to be modified. The two data records corresponding to "Zhang San" and "Wang Wu" were found, and the above operation instructions were executed to modify the ages of "Zhang San" and "Wang Wu". The operation version number of the modified data records was then set to 1, resulting in the following: Figure 3 The last two rows of data shown are recorded in data table A, completing the process. Figure 2 Modify the data in data table A and update the version information of data table A.

[0117] Specifically, the process first processes all existing information items of the data to be updated based on the query conditions in the operation command. Then, after inserting, replacing, or deleting the fields that need to be updated, the update operation is converted into an insert operation for execution. The insert operation command is used to insert fields into the Hive data table.

[0118] See further Figure 4 , Figure 4 This example demonstrates the result of performing a data cleanup operation on table A. Specifically, Figure 4 The updated results presented are in Figure 3 The data records in data table A are the updated records obtained after performing the above steps S12-S15 on data table A.

[0119] In this embodiment, the preset value corresponding to the deletion identifier is set to -1, and the preset threshold for the number of updates in the version information is set to 1. Figure 2 The update count for data table A is set to an initial value of 0. Perform the following operations on data table A.

[0120] The data update operation is complete and the result is as follows: Figure 3 The update results shown are due to Figure 2 The version information for data table A is 0, while Figure 3Data table A has undergone a data update operation, updating its version information to 1. After the update operation is completed, the version information of data table A is checked. At this time, the version information of data table A has reached the preset threshold of 1, and the cleanup operation of data table A is started.

[0121] First, determine the valid data records in data table A. Since there are no data records in data table A with an operation version number of -1, all three data records in data table A are valid data records. Next, determine the UUID and latest operation version number corresponding to the valid data records. The latest operation version number for "Zhang San" is 1, for "Li Si" it is 0, and for "Wang Wu" it is 1. Then, determine these three rows of data as the target data, cache the target data in a temporary data table, record the corresponding UUID, and set the operation version number of all three rows to the initial value of 0. Finally, clear the data in data table A, insert the temporary data table into data table A, obtaining the final cleaned data table A. Reset the update count of the initial data table A to the initial value of 0, ultimately presenting the result as shown below. Figure 4 The update results are shown below.

[0122] This application's solution monitors the update status of each data record in the data table using UUID and operation version number. By recording the version information of the data table, it monitors the update status of the data table. Once the number of updates meets a preset threshold, it automatically cleans up redundant data in the data table and initializes the version information of the data table. This allows for the periodic cleanup of redundant data in the data table. When updating data continuously over a long period of time, it can keep the computational complexity within a reasonable range, thereby maintaining the data processing efficiency within a reasonable range.

[0123] The following describes a Hive-based data update device provided in an embodiment of this application. The Hive-based data update device described below and the Hive-based data update method described above can be referred to and correspond to each other.

[0124] Combination Figure 5 This paper introduces a Hive-based data update device, such as... Figure 5 As shown, the device may include:

[0125] The data record query unit 100 is used to receive a data update operation instruction and determine the target data record to be updated in the target data table. The data update operation instruction is a data modification instruction or a data deletion instruction.

[0126] The data record update unit 110 is used to update the target data record according to the data update operation instruction, and update the version information of the target data table. The version information is used to record the number of updates or update information of the target data table.

[0127] The first judgment unit 120 is used to determine whether the updated version information meets the preset conditions;

[0128] The update information query unit 130 is used to query the UUID and latest operation version number of each valid data record in the target data table when the updated version information meets the preset conditions. The UUID is used to uniquely identify a valid data record, and the latest operation version number is used to identify the number of times the valid data record has been updated. The valid data record is a data record for which the data deletion instruction has not been executed.

[0129] Temporary data table generation unit 140 is used to obtain the target data corresponding to the latest operation version number of the valid data record, set an initial operation version number for the target data, and generate a temporary data table, wherein the temporary data table includes the target data, the UUID and the initial operation version number.

[0130] The target data table reset unit 150 is used to clear the target data table, insert the temporary data table into the target data table, and initialize the version information of the target data table.

[0131] Optionally, the Hive-based data update device provided in this application may further include:

[0132] The data record acquisition unit is used to receive new data instructions and acquire new data records.

[0133] The update identifier setting unit is used to determine the UUID value and initial operation version number of the newly added data record;

[0134] The data recording and input unit is used to execute the new data instruction, record the new data record, the UUID value and the initial operation version number in the target data table, and update the version information of the target data table.

[0135] Optionally, the Hive-based data update device provided in this application may further include:

[0136] The valid data query unit is used to receive a query data instruction and determine a valid data set, wherein the valid data set contains data information corresponding to the latest operation version number of each record in the target data table, and the query data instruction includes query conditions.

[0137] The target set query unit is used to execute the query data instruction, search for data information that meets the query conditions in the valid data set, obtain the target set, and output it.

[0138] Optionally, the data record update unit 110 may include:

[0139] Data modification unit and data deletion unit.

[0140] Optionally, the data modification unit may include:

[0141] The data record acquisition unit is used to execute the data modification instruction when the data update operation instruction is a data modification instruction, and obtain the modified updated data record;

[0142] An update data record entry unit is used to record the update data record in the target data table;

[0143] The update identifier input unit is used to determine and record the UUID and operation version number of the update data record.

[0144] Optionally, the data deletion unit may include:

[0145] The operation version number setting unit is used to set the operation version number of the target data record to a preset value when the data update operation instruction is a data deletion instruction.

[0146] Optionally, the Hive-based data update device provided in this application may further include:

[0147] The second judgment unit is used to query whether the preset value is included in the first operation version number, where the first operation version number is the historical operation version number of the data record in the target data table after the update operation is completed.

[0148] A valid data record identification unit is used to identify the data record as the valid data record when the first operation version number does not contain the preset value.

[0149] Optionally, the first determination unit 120 may include:

[0150] The update operation count determination unit is used to determine whether the number of update operations completed by the target data table has reached a first preset threshold.

[0151] And / or,

[0152] A data record quantity determination unit is used to determine whether the total number of data records in the target data table reaches a second preset threshold.

[0153] And / or,

[0154] The recording time judgment unit is used to determine whether the time recorded at the current moment has reached the preset time, wherein the time recorded at the current moment is the time when the target data table version information was last initialized.

[0155] Optionally, the Hive-based data update device provided in this application may further include:

[0156] The version information recording unit is used to record the version information of the latest data table in the management record table, which is used to manage the version of the data tables in the Hive library.

[0157] This application also provides a readable storage medium that stores a program suitable for processor execution, the program being used to implement various processing flows in the aforementioned Hive-based data update scheme.

[0158] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0159] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0160] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data update method based on Hive, characterized in that, include: Receive a data update operation instruction, determine the target data record to be updated in the target data table, wherein the data update operation instruction is a data modification instruction or a data deletion instruction; The target data record is updated according to the data update operation instruction, and the version information of the target data table is updated. The version information is used to record the number of updates or update information of the target data table. Determining whether the updated version information meets preset conditions includes: determining whether the number of update operations completed by the target data table has reached a first preset threshold; and / or determining whether the total number of data records in the target data table has reached a second preset threshold; and / or determining whether the time recorded at the current moment has reached a preset time, wherein the time recorded at the current moment is the time when the target data table version information was last initialized. If the updated version information meets the preset conditions, then query the UUID and latest operation version number of each valid data record in the target data table. The UUID is used to uniquely identify a valid data record, and the latest operation version number is used to identify the number of times the valid data record has been updated. The valid data record is a data record for which the data deletion instruction has not been executed. Obtain the target data corresponding to the latest operation version number of the valid data record, set an initial operation version number for the target data, and generate a temporary data table. The temporary data table contains the target data, the UUID, and the initial operation version number. Clear the target data table, insert the temporary data table into the target data table, and initialize the version information of the target data table.

2. The method according to claim 1, characterized in that, Before determining whether the updated version information meets the preset conditions, the method further includes: Receive the command to add data and retrieve the new data record; Determine the UUID value and initial operation version number of the newly added data record; Execute the new data instruction, record the new data record, the UUID value, and the initial operation version number in the target data table, and update the version information of the target data table.

3. The method according to claim 1, characterized in that, Also includes: Receive a data query instruction, determine a valid data set, the valid data set contains data information corresponding to the latest operation version number of each record in the target data table, and the data query instruction includes query conditions; Execute the query data instruction to search for data information that meets the query conditions in the valid data set, obtain the target set and output it.

4. The method according to claim 1, characterized in that, The step of updating the target data record according to the data update operation instruction includes: When the data update operation instruction is a data modification instruction, the data modification instruction is executed to obtain the modified updated data record; Record the updated data in the target data table; Determine and record the UUID and operation version number of the updated data record; or, When the data update operation instruction is a data deletion instruction, the operation version number of the target data record is set to a preset value.

5. The method according to claim 4, characterized in that, Also includes: Check whether the preset value is included in the first operation version number, where the first operation version number is the historical operation version number of the data record in the target data table after the update operation is completed; If the first operation version number does not contain the preset value, then the data record is determined as the valid data record.

6. The method according to claim 1, characterized in that, Also includes: The version information of the target data table is recorded in the management record table, which is used to manage the version of the data tables in the Hive library.

7. A Hive-based data update device, characterized in that, include: The data record query unit is used to receive data update operation instructions and determine the target data record to be updated in the target data table. The data update operation instructions are either data modification instructions or data deletion instructions. The data record update unit is used to update the target data record according to the data update operation instruction, and update the version information of the target data table. The version information is used to record the number of updates or update information of the target data table. The first judgment unit is used to judge whether the updated version information meets the preset conditions, including: judging whether the number of update operations completed by the target data table has reached a first preset threshold; and / or judging whether the total number of data records in the target data table has reached a second preset threshold; and / or judging whether the time recorded at the current moment has reached a preset time, wherein the time recorded at the current moment is the time when the target data table version information was last initialized. The update information query unit is used to query the UUID and latest operation version number of each valid data record in the target data table when the updated version information meets the preset conditions. The UUID is used to uniquely identify a valid data record, and the latest operation version number is used to identify the number of times the valid data record has been updated. The valid data record is a data record for which the data deletion instruction has not been executed. A temporary data table generation unit is used to obtain the target data corresponding to the latest operation version number of the valid data record, set an initial operation version number for the target data, and generate a temporary data table, wherein the temporary data table contains the target data, the UUID and the initial operation version number; The target data table reset unit is used to clear the target data table, insert the temporary data table into the target data table, and initialize the version information of the target data table.

8. The apparatus according to claim 7, characterized in that, Before the first determination unit, it also includes: The data record acquisition unit is used to receive new data instructions and acquire new data records. An update identifier setting unit is used to determine the UUID value and initial operation version number of the newly added data record; The data recording and input unit is used to execute the new data instruction, record the new data record, the UUID value and the initial operation version number in the target data table, and update the version information of the target data table; The valid data query unit is used to receive a query data instruction and determine a valid data set, wherein the valid data set contains data information corresponding to the latest operation version number of each record in the target data table, and the query data instruction includes query conditions. The target set query unit is used to execute the query data instruction, search for data information that meets the query conditions in the valid data set, obtain the target set, and output it.

9. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of a Hive-based data update method as claimed in any one of claims 1-6.

Images