Batch data updating method and device, electronic equipment and readable storage medium

Abstract

The application discloses a batch data updating method and device, electronic equipment and a readable storage medium. The method comprises the following steps: selecting one piece of data from a batch of to-be-updated data, wherein each piece of data in the to-be-updated data comprises a unique key and data content; searching for a first data block storing the unique key according to the unique key of the one piece of data, and updating the data content in the one piece of data into a corresponding data block according to the unique key; obtaining other unique keys except the unique key of the one piece of data from the first data block, and searching for data comprising the other unique keys from the to-be-updated data; and updating the data content of the data comprising the other unique keys into a corresponding data block. The application solves the problem that the database performance is affected by slow updating in the prior art when batch data updating is performed on the database, thereby improving the updating efficiency of the database and improving the performance of the database to a certain extent.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to methods, apparatus, electronic devices, and readable storage media for batch data updating. Background Technology

[0002] Databases are typically used to store large amounts of data. During database use, batch data updates often occur, meaning a batch of data is updated to the database all at once. Although it's a batch update, during execution, it's still necessary to query the database for each piece of data in the batch. After obtaining the query results, the corresponding operation is performed to update the database with that data.

[0003] Since the database already stores a large amount of data, querying each piece of data in the database takes time. When performing batch data updates, these query times accumulate and become very large, making batch updates relatively slow. In addition, querying each piece of data in the database also affects the performance of the database. Summary of the Invention

[0004] This application provides a batch data update method, apparatus, electronic device, and readable storage medium to at least solve the problem of slow updates affecting database performance when performing batch data updates in the prior art.

[0005] According to one aspect of this application, a batch data update method is provided, comprising: selecting a data entry from a batch of data to be updated, wherein each data entry in the batch of data to be updated includes a unique key and data content, the unique key being used to identify the data entry; searching a first data block storing the unique key according to the unique key of the data entry, and updating the data content of the data entry to the corresponding data block according to the unique key; obtaining other unique keys besides the unique key of the data entry from the first data block, and searching for data including the other unique keys from the data to be updated; and updating the data content of the data including the other unique keys to the corresponding data block.

[0006] Furthermore, obtaining the other unique keys and searching for and updating the data content including the other unique keys in the data to be updated to the corresponding database includes: starting from the position of the unique key of the first data in the first data block, sequentially obtaining the unique key of the next data; if data including the unique key obtained this time is found in the data to be updated, updating the data content including the unique key obtained this time to the corresponding data block, until the end of the first data block is reached or the unique key obtained this time is not found in the data to be updated.

[0007] Furthermore, after updating the data content including the other unique keys to the corresponding data block, the remaining data in the data to be updated is taken as a new batch of data to be updated, and the above method is performed on the new batch of data to be updated until there is no remaining data in the data to be updated, wherein the remaining data is data in the existing data block where a unique key can be found and the data content has not been updated to the corresponding data block.

[0008] Furthermore, it also includes: obtaining at least one piece of data in the data to be updated that does not have a unique key found in the existing data block; inserting the at least one piece of data into a list; and if the number of data entries in the list exceeds a threshold or the current time meets the time requirement, inserting the data in the list into the corresponding data block.

[0009] Furthermore, after updating the data content including the other unique keys to the corresponding data block, or after updating the data content of all data in the data to be updated to the corresponding data block, the method further includes: updating the data block containing the data content to be updated from memory to disk.

[0010] Furthermore, searching for data including the other unique keys from the data to be updated includes: searching for data including the other unique keys from a binary tree, wherein the binary tree is pre-created based on all the unique keys of the data to be updated.

[0011] Furthermore, the data to be updated includes: the results obtained after analyzing multimedia files for a predetermined target, wherein if multiple data have the same unique key, the multiple data are: data including the results obtained after analyzing the same multimedia files for the same target using different analysis devices, algorithms and / or neural network models.

[0012] Furthermore, it also includes: when the data to be updated includes data content of multiple fields, obtaining configuration information, wherein the configuration information carries field names; searching for a field that matches the field name among the multiple fields; and using the field that matches the field name as the unique key of the data.

[0013] According to another aspect of this application, a batch data update apparatus is also provided, comprising: a selection module, configured to select one data entry from a batch of data to be updated, wherein each data entry in the data to be updated includes a unique key and data content, the unique key being used to identify the data entry; a first lookup module, configured to look up a first data block storing the unique key according to the unique key of the data entry, and update the data content of the data entry to the corresponding data block according to the unique key; a second lookup module, configured to obtain other unique keys besides the unique key of the data entry from the first data block, and look up data including the other unique keys from the data to be updated; and an update module, configured to update the data content of the data including the other unique keys to the corresponding data block.

[0014] Furthermore, the second search module is used to sequentially obtain the unique key of the next data in the first data block, starting from the position of the unique key of the data. If data including the unique key obtained this time is found in the data to be updated, the update module updates the data content including the unique key obtained this time to the corresponding data block until the end of the first data block is reached or the unique key obtained this time is not found in the data to be updated.

[0015] Furthermore, after updating the data content including the other unique keys to the corresponding data block, the remaining data in the data to be updated is taken as a new batch of data to be updated. The new batch of data to be updated is processed by the selection model, the first search module, the second search module and the update module until there is no remaining data in the data to be updated. The remaining data is data in the existing data block where a unique key can be found and the data content has not been updated to the corresponding data block.

[0016] Furthermore, the update module is also used to: obtain at least one piece of data in the data to be updated that has not been found to have a unique key in the existing data block; insert the at least one piece of data into a list; and, if the number of data entries in the list exceeds a threshold or the current time meets the time requirement, insert the data in the list into the corresponding data block.

[0017] Furthermore, it also includes: a persistence module, used to update the data block containing the data including the other unique keys from memory to disk after updating the data content to the corresponding data block, or after updating the data content of all data in the data to be updated to the corresponding data block.

[0018] Furthermore, the second search module is used to: search for data including the other unique keys from a binary tree, wherein the binary tree is pre-created based on all the unique keys of the data to be updated.

[0019] Furthermore, the data to be updated includes: the results obtained after analyzing multimedia files for a predetermined target, wherein if multiple data have the same unique key, the multiple data are: data including the results obtained after analyzing the same multimedia files for the same target using different analysis devices, algorithms and / or neural network models.

[0020] Furthermore, it also includes: a configuration module, used to obtain configuration information when the data to be updated includes data content of multiple fields, wherein the configuration information carries field names; to search for a field that matches the field name among the multiple fields; and to use the field that matches the field name as the unique key of the data.

[0021] According to another aspect of this application, an electronic device is also provided, including a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the above-described method steps.

[0022] According to another aspect of this application, a readable storage medium is also provided, on which computer instructions are stored, wherein the computer instructions, when executed by a processor, implement the above-described method steps.

[0023] In this embodiment, a method is employed to select one data entry from a batch of data to be updated. Each data entry includes a unique key and data content, with the unique key identifying the data entry. The method involves searching a first data block storing the unique key and updating the data content of the data entry to the corresponding data block based on the unique key. Other unique keys besides the unique key of the data entry are then obtained from the first data block, and data containing these other unique keys is searched from the data to be updated. Finally, the data content of the data containing these other unique keys is updated to the corresponding data block. This application solves the problem of slow updates affecting database performance during batch data updates in the prior art, thereby improving database update efficiency and, to a certain extent, improving database performance. Attached Figure Description

[0024] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0025] Figure 1 This is a flowchart of a batch data update method according to an embodiment of this application;

[0026] Figure 2 This is a schematic diagram of the process for batch updating and persisting data according to this implementation.

[0027] Figure 3 This is a schematic diagram of the process for reading the unique key configuration according to this embodiment;

[0028] Figure 4 This is a schematic diagram of the background asynchronous update process according to an embodiment of this application; and,

[0029] Figure 5 This is a flowchart of data insertion according to an embodiment of this application. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0032] When updating batch data, there are two scenarios: either the data already exists in the database (in which case the new data replaces the existing data), or the data doesn't exist in the database (in which case it's inserted as new data). Insertion and update are two of the most basic operations provided by databases. For batch updates, it's necessary to know which data exists in the database and which doesn't to determine whether to use an update or insert operation. Therefore, although it's a batch update, each data record still needs to be queried. It's important to note that the time consumed by insert and update operations is essentially the same; the majority of the time for batch updates is spent searching the database record by record.

[0033] To address this issue, asynchronous updates can be used. The main idea behind asynchronous updates is to mark the data that needs updating in the database. When the data is found in a query, the update is performed based on this mark. Essentially, this approach divides batch data updates into different time intervals; updates are performed only when the data is found, and not when it is not found. In this approach, each time marked data needs updating is retrieved, it must be updated before the retrieval results are returned. Therefore, asynchronous updates impact retrieval performance. Furthermore, asynchronous updates do not fundamentally solve the time-consuming and performance-impacting problem caused by querying every single piece of data during batch updates.

[0034] Considering that batch data updates in a database require querying each record, and queries are typically performed across the entire database, which usually contains a large amount of data, retrieval requires iterating through existing data, thus consuming time. While the query time for a single record is acceptable, querying multiple records in a batch consumes significant time. Furthermore, the continuous execution of query operations also impacts the database's responsiveness to other requests, thereby degrading database performance. The following implementation addresses the problem of batch database updates by shortening query time. By reducing query time, the update time for batch data will also be shortened, thus resolving the problems existing in the prior art.

[0035] The following implementation provides a batch data update method. Figure 1 This is a flowchart of a batch data update method according to an embodiment of this application, such as... Figure 1 As shown below, Figure 1 The steps involved are explained.

[0036] Step S102: Select one piece of data from a batch of data to be updated, wherein each piece of data to be updated includes a unique key and data content, and the unique key is used to identify the data.

[0037] A database typically contains data across multiple fields. Taking a database storing user data as an example, the data for one user might include fields such as user ID, user name, and user age. The user ID identifies the user record and is therefore the unique key for that record. The other fields, such as user name and age, constitute the data content of that record. This unique key is used to retrieve the data.

[0038] In this step, a single data point can be randomly selected from the batch of data to be updated, or a data point at a fixed position, such as the first or last data point in the batch, can be selected. Regardless of the data point chosen, the execution of the following steps will not be affected. In the following implementation, the first data point in the batch of data to be updated will be used.

[0039] Step S104: Locate the first data block storing the unique key according to the unique key of the data, and update the data content in the data to the corresponding data block according to the unique key.

[0040] Data in a database is stored in data blocks. Database data is generally stored in multiple data blocks, and each data block contains the corresponding data content. In this step, the unique key of a selected piece of data is retrieved from the data blocks. For ease of description, the data block storing the unique key of this piece of data is called the first data block. When the data block containing this piece of data is retrieved, the data content of this piece of data is updated in the data block.

[0041] Step S106: Obtain other unique keys besides the unique key of the data from the first data block, and search for data including the other unique keys from the data to be updated.

[0042] It should be noted that a data block typically stores multiple unique keys, not just a single data entry or a single unique key. Therefore, in this step, other unique keys are directly retrieved from the first data block, and then searched for in the data to be updated. In existing technologies, searches are performed on the database based on unique keys. In this step, unique keys are retrieved from the data block, and then searched in the data to be updated. In the following implementation, this search method is called reverse retrieval (or simply reverse lookup). Considering the large amount of data stored in the database, searching within the database can be time-consuming. However, querying within the batch of data to be updated is much faster and does not affect other database operations. Therefore, this reverse retrieval saves search time.

[0043] Step S108: Update the data content, including the data of the other unique keys, to the corresponding data block.

[0044] After performing a reverse lookup on the batch data to be updated, if data corresponding to the unique key in the first data block is found in the data to be updated, then the data content of that data in the data to be updated is updated to the corresponding data block. The data content of a single piece of data may be stored in one or more data blocks, and during the update, the data content of that single piece of data needs to be updated to the corresponding data blocks.

[0045] In the above steps, instead of querying the database for every single piece of data in the data to be updated, a reverse lookup is performed on the data to be updated after finding the data block containing the unique key of a piece of data. This reverse lookup, compared to querying the database directly, improves query speed due to the smaller data size and does not interfere with other database operations or performance. Therefore, these steps solve the problem of slow updates impacting database performance in existing technologies when performing batch data updates, thereby improving database update efficiency and, to some extent, enhancing database performance.

[0046] To further improve the speed of reverse data lookup in the data to be updated, a binary tree can be pre-created based on all the unique keys of the data to be updated, and then data including the other unique keys can be searched from the binary tree. Using binary trees can improve retrieval speed. A binary tree is a tree-like structure; a tree with at most two children is called a binary tree. Since each element in a binary tree can only have two children, they are usually named the left child and the right child. There are many types of binary trees. For example, a Binary Search Tree (BST) is a node-based binary tree data structure with the following characteristics: the left subtree of a node contains only nodes with values ​​less than the node's value, the right subtree contains only nodes with values ​​greater than the node's value, and both the left and right subtrees must also be binary search trees. Another example is a balanced binary tree, which is a self-balancing binary search tree (BST) where the height difference between the left and right subtrees of all nodes cannot exceed 1. Yet another example is a red-black tree, a self-balancing binary search tree where each node has an extra space to store the node's color (red or black). These colors are used to ensure the tree remains balanced during insertions and deletions. In a red-black tree, every node is either red or black; the root node is always black; all leaf nodes are null and black; no two adjacent red nodes (a red node cannot have a red parent or child); and the path from any node to every leaf node in its subtree contains the same number of black nodes. While the balance of a red-black tree is not perfect, it is sufficient to reduce search time. Using these types of binary trees can improve retrieval time, and red-black trees are recommended as a preferred implementation.

[0047] The above steps also involve searching for other unique keys stored in the first data block. There are many ways to obtain other unique keys from the first data block. For example, all other unique keys in the first data block can be found by traversing the data, and then a reverse lookup can be performed in the data to be updated. The data content of the reverse-lookup data can then be updated to the corresponding data block. To improve the speed of finding other unique keys in the first data block, the search can start from the unique key of the selected data and search for the unique key of the next data. This method of obtaining unique keys follows the storage order of the data in the first storage block, and the efficiency of obtaining other unique keys is relatively high. That is, in this optional embodiment, obtaining the other unique keys, searching for and updating the data content of the data including the other unique keys in the data to be updated to the corresponding database can include the following steps: starting from the position of the unique key of the first data in the first data block, the unique key of the next data is obtained sequentially. If data including the unique key obtained this time is found in the data to be updated, the data content of the data including the unique key obtained this time is updated to the corresponding data block, until the end of the first data block is reached or the unique key obtained this time is not found in the data to be updated. In this optional implementation, the next piece of data can be retrieved sequentially according to the storage order of the data in the first storage block until the end of the first data block is reached or the unique key retrieved this time is not found in the data to be updated. This allows for quick retrieval of data corresponding to other unique keys in the first storage block and updates, thus improving the data update speed.

[0048] In another optional embodiment, after updating the data content of the data including the other unique keys in the data to be updated to the corresponding data block, if there is still remaining data in the data to be updated, then the remaining data in the data to be updated is regarded as a new batch of data to be updated, and steps S102 to S108 are executed on the new batch of data to be updated until there is no remaining data in the data to be updated. The remaining data refers to data whose unique keys can be found in existing data blocks but whose data content has not been updated to the corresponding data blocks. It should be noted that there are two types of data in the data to be updated: one type is data that already exists in the database, for which a unique key can be found in the database's data blocks; the other type is data that does not exist in the database, for which data needs to be inserted into the database's data blocks. In this optional embodiment, the update operation is performed after finding all existing data in the data to be updated in the database. For data in the data to be updated that does not exist in the database, an insertion operation can be performed in the database, and insertion can be performed one record at a time. In one optional embodiment, to improve insertion efficiency, multiple records can be inserted simultaneously, which can improve data insertion efficiency. In this optional implementation, the following steps may also be included: obtaining at least one piece of data from the data to be updated that does not have a unique key found in the existing data block; inserting the at least one piece of data into a list; and inserting the data in the list into the corresponding data block if the number of data entries in the list exceeds a threshold or the current time meets the time requirement. For example, all the data to be inserted can be stored in a list, and then an insertion operation can be performed every 10 seconds or other predetermined intervals; or, for another example, an insertion operation can be performed when the number of data entries in the list exceeds 100 or other values, which can insert all the data in the list into the data block of the database, improving the efficiency of data insertion.

[0049] The above implementation methods update the data to be updated into the data blocks through update or insert methods. Each update involves copying the data to be updated into the data blocks. Typically, the data to be updated is cached in memory. To improve the copying speed from the data to the memory blocks, in an optional implementation, the database data blocks can be pre-read from the disk into memory; that is, in the optional implementation, the data blocks are all stored in memory. Data in memory is not permanent; for example, it will be lost during a power outage. Therefore, in this optional implementation, the data blocks to be updated need to be saved to disk. Data saved to disk will not be lost during a power outage; it can be persistent. Therefore, saving data blocks to disk can also be called persistence. The persistence process can be performed once every time a data block is updated. This method consumes a lot of disk resources. Therefore, multiple data blocks can be updated to disk simultaneously during each persistence process to improve persistence efficiency. For example, these data blocks can be persisted after updating the data content of the data to be updated corresponding to the other unique keys included in the first data block to the corresponding data blocks; or, alternatively, all these databases can be persisted after updating all the data to be updated to the corresponding data blocks. That is, the data blocks containing the data to be updated can be updated from memory to disk after either updating the data content of the data including the other unique keys to the corresponding data blocks, or after updating the data content of all the data to be updated to the corresponding data blocks. The timing of persistence in these two examples can be flexibly chosen in practical applications.

[0050] Figure 2 This is a flowchart illustrating the process of batch updating and persisting data according to this implementation. Figure 2 In the data to be updated, each data entry includes: (rowKeyi, colA-i, colB-i, ...), where i represents the data entry number, rowKey identifies the unique key, and colA and colB are the data content of that entry, with colA and colB representing different fields. Figure 2 The database shown is a columnar database, in which the data of each column (i.e., the data of each field) is stored in a data block. Figure 2 The diagram shows that rowKey is stored in one data block, while colA and colB are stored in separate data blocks. Figure 2Data updates are performed through database interfaces and application programming interfaces (APIs). Database interfaces are provided by database providers for accessing databases. The method described above can be used as an API, enabling the continuous data update method using the reverse lookup approach. Figure 2 As shown, the process for batch data updates may include the following steps:

[0051] Step one: Upon receiving a batch update request from a client, first save the data to be updated to the update cache, and then return a success message. In this step, the client is the one used to access the database. Since updates take some time, sending a success message to the client upon receiving a batch update request prevents the client from being in a waiting state indefinitely.

[0052] Step two: The background update thread retrieves the earliest data record from the update cache and locates the data block containing that unique key, along with its offset. For batch data updates, this step can be performed in the background using a dedicated thread, referred to as the data update thread. Each data block stores multiple unique keys, and each unique key has a corresponding position within the data block. This position can be represented by an offset relative to the beginning of the file. Therefore, this step requires locating the data block containing the unique key and its offset. Since different data entries in the same field have the same length, the offset of each data entry within a data block relative to the previous data entry is the same.

[0053] Step 3: After locating the data position, update the remaining non-empty fields of this data entry based on the data offset. In this step, empty fields in a data entry do not need to be updated; therefore, only the non-empty fields are updated. Each non-empty field is stored in a corresponding data block, such as... Figure 2 The image shows two data blocks that store the colA and colB fields.

[0054] Step four: After the data update in step three is complete, immediately retrieve the next unique key with the offset, and then look up this piece of data to be updated from the update cache. This operation saves the time required to traverse the data block to find the location of the unique key for each piece of data to be updated;

[0055] Step 5: After finding the data to be updated, repeat step 4 to integrate the remaining non-empty fields into the analysis results;

[0056] Step 6: When the iteration reaches the end of the data block or the unique key found in the reverse lookup does not exist in the cache to be updated, commit the operation and flush the modified data blocks to disk.

[0057] Step 7: Repeat steps 2, 3, 4, 5, and 6 until there is no data left in the updated cache.

[0058] Step 8: Persist the updated data blocks to disk at pre-configured time intervals.

[0059] The above implementation method can be applied to batch updates of various types of data. For example, in video image analysis scenarios, when multiple analyses are performed on captured videos and images, it is necessary to integrate the results of multiple analyses targeting the same objective. That is, the data to be updated includes: results obtained after analyzing multimedia files for a predetermined objective. If multiple data sets have the same unique key, then the multiple data sets include: data obtained after analyzing the same multimedia files for the same objective using different analysis devices, algorithms, and / or neural network models.

[0060] In one optional implementation, the unique key can be configured by the user. That is, when the data to be updated includes data content of multiple fields, configuration information is obtained, wherein the configuration information carries field names; the field matching the field name is searched among the multiple fields; and the field matching the field name is used as the unique key of the data. Figure 3 This is a schematic diagram of the process for reading the unique key configuration according to this embodiment, as follows: Figure 3As shown, the process can include the following steps: First, read the configuration information used to configure unique keys. This configuration information involves multiple strings, separated by special characters. Separating the strings read from the configuration information with special characters yields strings representing each field. The configuration information can include unique keys configured for multiple tables, with one unique key per table. Unique keys across multiple tables can be separated by special characters. For example, if table A uses string A as the unique key and table B uses string B as the unique key, the read configuration information would be "string A, string B". The comma "," is used to separate strings A and B, resulting in strings A and B. Strings A and B can then be saved to a configuration list. Next, treat strings A and B as a set of data and search the data to be updated for a field identical to string A. If found, configure string A as the unique key (mapping string A in the table to the unique key information). After configuring the unique key, delete the data from the configuration list. If no fields matching strings A and B are found in the data to be updated, the configuration set is discarded and an error is logged. The process of configuring a unique key ends when the configuration list is empty after the unique key has been configured.

[0061] With a unique key configured, data updates will be performed multiple times based on that key. This can significantly impact database update performance. The main time-consuming aspects are: 1) When inputting secondary or multiple analysis results into the database, the data from the first input needs to be retrieved. This requires searching for the storage location of the data result based on the unique key, which is time-consuming and becomes the performance bottleneck. 2) After integrating multiple analysis results, the corresponding data blocks need to be persisted to disk to ensure reliability. Each update requires flushing multiple data files to disk, further reducing update performance. To address these issues, asynchronous background updates can be implemented. Figure 4 This is a schematic diagram of the background asynchronous update process according to an embodiment of this application, as shown below. Figure 4As shown, the data to be updated is placed in a cache (called the update cache). The earliest data record is retrieved from the cache, and its unique key is obtained. The data block containing the unique key is traversed until the data block and offset corresponding to the unique key are found. Then, the remaining fields of that data are updated. After the first data record is updated, the process iterates to the next unique key stored in that data block, following the order of the data blocks containing the unique key. The data to be updated is then retrieved from the update cache. Once the unique key is found, the remaining fields of the data containing that unique key are updated to the corresponding data block. Because the update cache uses a red-black tree based on the unique key, the query performance is high, thus saving the time spent traversing data blocks to find the unique key. This process of retrieving and updating is repeated until the data to be updated with the corresponding unique key is no longer found in the update cache, or the end of the data block has been reached. At this point, the loop iteration is exited, and the batch of updated data is committed. All previously updated data blocks are then persisted to disk. If there is still data to be updated in the cache at this point, the entire process is repeated. Figure 4 The process shown in the image is sufficient.

[0062] After Figure 4 Following the initial process, for new data in the data to be updated (i.e., data that has not been saved in the database before), an insertion process needs to be performed to save it to the database. Figure 5 This is a flowchart of data insertion according to an embodiment of this application, such as... Figure 5 As shown, after searching for a unique key, if no data for that unique key is found during a previous insertion, the data is converted to an insertion process. When no data for a previous insertion is found and needs to be converted to an insertion, the data is written to the list to be inserted, and batch insertions are submitted periodically to reduce insertion time and also reduce the impact on update performance.

[0063] Through the above Figures 2 to 5The illustrated process allows for the configuration of unique keys, enabling users to configure unique keys to identify the results of multiple analyses of the same video or videos based on the same objective. Upon receiving a user's batch update request, the data to be updated is stored in the update cache. From the update cache, the earliest data record is retrieved, and its unique key storage location is found. Based on the found unique key location, the remaining non-null fields are updated to this data record. The process iterates backward through the data block containing the earliest data record's unique key to find the locations of other unique keys within that block, retrieving the unique key of the next data record in that block. Based on the retrieved unique key, the data to be updated is searched in the update cache. If it exists, this data record is retrieved, and the remaining non-null fields of the data to be updated are updated to the corresponding data block. These steps are repeated until the end of the data block is reached or the retrieved unique key is no longer found in the update cache. Finally, the data blocks modified in this batch update are persisted to disk. These steps optimize the bottlenecks in batch updates, improving efficiency and ultimately enhancing database performance.

[0064] In this embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the methods described in the above embodiments.

[0065] The aforementioned program can run on a processor or be stored in memory (or computer-readable medium). Computer-readable medium includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable medium does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0066] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented using different modules, and different steps can be implemented using different modules.

[0067] This embodiment provides such an apparatus, referred to as a batch data update apparatus, comprising: a selection module, configured to select one data entry from a batch of data to be updated, wherein each data entry includes a unique key and data content, the unique key being used to identify the data entry; a first lookup module, configured to look up a first data block storing the unique key based on the unique key of the data entry, and update the data content of the data entry to the corresponding data block based on the unique key; a second lookup module, configured to obtain other unique keys besides the unique key of the data entry from the first data block, and look up data including the other unique keys from the data to be updated; and an update module, configured to update the data content of the data including the other unique keys to the corresponding data block.

[0068] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.

[0069] Optionally, the second search module is used to sequentially obtain the unique key of the next data in the first data block, starting from the position of the unique key of the data. If data including the unique key obtained this time is found in the data to be updated, the update module updates the data content including the unique key obtained this time to the corresponding data block until the end of the first data block is reached or the unique key obtained this time is not found in the data to be updated.

[0070] Optionally, after updating the data content including the other unique keys to the corresponding data block, the remaining data in the data to be updated is taken as a new batch of data to be updated, and the new batch of data to be updated is processed by the selection model, the first search module, the second search module and the update module until there is no remaining data in the data to be updated, wherein the remaining data is data in the existing data block where a unique key can be found and the data content has not been updated to the corresponding data block.

[0071] Optionally, the update module is further configured to: obtain at least one piece of data in the data to be updated that has not been found to have a unique key in the existing data block; insert the at least one piece of data into a list; and, if the number of data entries in the list exceeds a threshold or the current time meets the time requirement, insert the data in the list into the corresponding data block.

[0072] Optionally, it also includes: a persistence module, used to update the data block containing the data including the other unique keys from memory to disk after the data content is updated to the corresponding data block, or after the data content of all the data to be updated is updated to the corresponding data block.

[0073] Optionally, the second search module is used to: search for data including the other unique keys in a binary tree, wherein the binary tree is pre-created based on all the unique keys of the data to be updated.

[0074] Optionally, the data to be updated includes: the results obtained after analyzing multimedia files for a predetermined target, wherein if multiple data have the same unique key, the multiple data are: data including the results obtained after analyzing the same multimedia files for the same target using different analysis devices, algorithms and / or neural network models.

[0075] Optionally, it further includes: a configuration module, configured to, when the data to be updated includes data content of multiple fields, obtain configuration information, wherein the configuration information carries field names; search for a field that matches the field name among the multiple fields; and use the field that matches the field name as the unique key of the data.

[0076] The above optional implementation methods solve the problem of slow updates affecting database performance when performing batch data updates in the prior art, thereby improving the database update efficiency and improving database performance to a certain extent.

[0077] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of bulk data update, characterized by, The method comprises: selecting a piece of data from a batch of to-be-updated data, wherein each piece of data in the to-be-updated data comprises a unique key and data content, and the unique key is used to identify the piece of data; finding a first data block storing the unique key according to the unique key of the piece of data, and updating the data content in the piece of data into a corresponding data block according to the unique key; obtaining other unique keys from the first data block except the unique key of the piece of data, and finding data comprising the other unique keys from the to-be-updated data; updating the data content of the data comprising the other unique keys into a corresponding data block.

2. The method of claim 1, wherein, The obtaining of the other unique keys, the finding of the data comprising the other unique keys from the to-be-updated data, and the updating of the data content of the data comprising the other unique keys into a corresponding database comprise: obtaining the unique keys of the next pieces of data from the first data block starting from the position of the unique key of the piece of data in sequence, and updating the data content of the data comprising the obtained unique keys into a corresponding data block in the case that the data comprising the obtained unique keys is found in the to-be-updated data, until the end of the first data block is reached.

3. The method of claim 1, wherein, after the data content of the data comprising the other unique keys is updated into a corresponding data block, the remaining data in the to-be-updated data is taken as a new batch of to-be-updated data, and the method in claim 1 is performed on the new batch of to-be-updated data until there is no remaining data in the to-be-updated data, wherein the remaining data is data whose unique key can be found in an existing data block and whose data content has not been updated into a corresponding data block.

4. The method of claim 3, wherein, The method further comprises: obtaining at least one piece of data in the to-be-updated data whose unique key cannot be found in an existing data block; inserting the at least one piece of data into a list; in the case that the number of data in the list exceeds a threshold or the current time meets a time requirement, inserting the data in the list into a corresponding data block.

5. The method according to any one of claims 1 to 4, characterized in that, After the data content of the data comprising the other unique keys is updated into a corresponding data block, or after the data content of all data in the to-be-updated data is updated into a corresponding data block, the method further comprises: updating the data block of the updated data content from a memory to a disk.

6. The method according to any one of claims 1 to 4, characterized in that, The finding of the data comprising the other unique keys from the to-be-updated data comprises: finding the data comprising the other unique keys from a binary tree, wherein the binary tree is pre-created according to all unique keys of the to-be-updated data.

7. The method according to any one of claims 1 to 4, characterized in that, The to-be-updated data comprises results obtained after a multimedia file is analyzed for a predetermined target, wherein if the unique keys of multiple pieces of data are the same, the multiple pieces of data are data comprising results obtained after the same multimedia file is analyzed by different analysis devices, algorithms, and / or neural network models for the same target.

8. The method of claim 7, wherein, The method further comprises: in the case that the to-be-updated data comprises data content of multiple fields, obtaining configuration information, wherein the configuration information carries field names. finding a field matching the field name from the plurality of fields; updating the data content of the data including the other unique keys into corresponding data blocks.

9. A batch data updating apparatus characterized by comprising: comprising: a selecting module, configured to select one piece of data from a batch of to-be-updated data, wherein each piece of data in the to-be-updated data comprises a unique key and data content, and the unique key is used to identify the piece of data; a first finding module, configured to find a first data block storing the unique key according to the unique key of the one piece of data, and update the data content of the one piece of data into a corresponding data block according to the unique key; a second finding module, configured to acquire, from the first data block, other unique keys except the unique key of the one piece of data, and find, from the to-be-updated data, data comprising the other unique keys; an updating module, configured to update the data content of the data comprising the other unique keys into corresponding data blocks.

10. The apparatus according to claim 9, wherein the second finding module is configured to acquire, from the first data block, the unique key of a next piece of data in sequence from a position where the unique key of the one piece of data is located, and the updating module is configured to update the data content of the data comprising the acquired unique key into a corresponding data block until the end of the first data block is reached, in a case where the data comprising the acquired unique key is found in the to-be-updated data; and / or after the data content of the data comprising the other unique keys is updated into corresponding data blocks, the remaining data in the to-be-updated data is taken as a new batch of to-be-updated data, and the new batch of to-be-updated data is processed by the selecting module, the first finding module, the second finding module and the updating module until there is no remaining data in the to-be-updated data, wherein the remaining data is data whose unique key can be found in an existing data block and whose data content has not been updated into a corresponding data block; and / or the updating module is further configured to acquire at least one piece of data in the to-be-updated data whose unique key cannot be found in an existing data block, insert the at least one piece of data into a list, and insert the data in the list into corresponding data blocks in a case where the number of data in the list exceeds a threshold or the current time meets a time requirement; and / or further comprising a persisting module, configured to update a data block of data whose data content is updated into corresponding data blocks from a memory to a disk after the data content of the data comprising the other unique keys is updated into corresponding data blocks, or after the data content of all data in the to-be-updated data is updated into corresponding data blocks; and / or the second finding module is configured to find the data comprising the other unique keys from a binary tree, wherein the binary tree is pre-created according to all unique keys of the to-be-updated data; and / or The data to be updated comprises results obtained by analyzing multimedia files for predetermined targets, wherein if a plurality of data have the same unique key, the plurality of data are data comprising results obtained by analyzing the same multimedia files for the same target using different analysis devices, algorithms and / or neural network models, and / or The configuration module is further configured to, in a case where the data to be updated comprises data content of a plurality of fields, acquire configuration information, wherein the configuration information carries a field name; search for a field matching the field name from the plurality of fields; and take the field matching the field name as a unique key of the data.

11. An electronic device comprising a memory and a processor; wherein, The memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method steps of any one of claims 1 to 8.

12. A readable storage medium having computer instructions stored thereon, wherein, The computer instructions, when executed by the processor, implement the method steps of any one of claims 1 to 8.

Images