Data updating method and device, terminal equipment and storage medium

Abstract

The application discloses a data updating method and device, a terminal equipment and a storage medium, relates to the field of database data updating, and the method comprises the steps of acquiring to-be-updated data; performing array path mapping processing on the to-be-updated data to obtain mapped to-be-updated data; extracting primary key information of the mapped to-be-updated data to obtain a to-be-queried primary key; performing local updating on an aggregated document stored in a transfer database according to the mapped to-be-updated data to obtain an updated aggregated document; and performing array restoration on the updated aggregated document according to the to-be-queried primary key to obtain a final document. The application improves data updating efficiency.

Description

Technical Field

[0001] This application relates to the field of database data updating, and more particularly to a data updating method, apparatus, terminal device, and storage medium. Background Technology

[0002] Multi-table joins are a common database query operation, typically used in business scenarios involving multiple tables. Specifically, to reduce the overhead on the business database, multi-table data is usually stored in an Elasticsearch database, utilizing Elasticsearch's statistical and query capabilities to meet the needs of the business scenario.

[0003] However, this data storage method of storing multi-table data in Elasticsearch suffers from low data update efficiency. Specifically, updating data in the Elasticsearch database requires first querying the corresponding data document in the database for the data to be updated, then aggregating several data documents to obtain an aggregate document, then updating the data to be updated and overwriting the aggregate document, and finally writing the updated aggregate document back to the Elasticsearch database. Understandably, this data update process is cumbersome and reduces efficiency. This is especially true for complex update methods involving aggregating sub-documents into arrays before storing them. Specifically, in this case, the complete aggregate document must first be retrieved from the Elasticsearch database, then the array must be traversed and the sub-documents located before the update operation can be performed, and finally the updated aggregate document is written back to the Elasticsearch database. This process involves multiple interactions with the Elasticsearch database due to the need to traverse the array and query several data documents to aggregate the document, resulting in low document update efficiency. Summary of the Invention

[0004] The main objective of this application is to provide a data update method, apparatus, terminal device, and storage medium, with the aim of improving data update efficiency.

[0005] To achieve the above objectives, this application provides a data update method, the data update method comprising:

[0006] Get the data to be updated;

[0007] Perform array path mapping on the data to be updated to obtain the mapped data to be updated;

[0008] Extract the primary key information of the mapped data to be updated to obtain the primary key to be queried;

[0009] Based on the mapped data to be updated, the aggregated document stored in the transit database is partially updated to obtain the updated aggregated document.

[0010] Based on the primary key to be queried, the updated aggregated documents are restored as an array to obtain the final document.

[0011] Optionally, the step of performing array path mapping processing on the data to be updated to obtain the mapped data to be updated includes:

[0012] Based on a preset configuration file, the data structure of the data to be updated is identified to obtain primary key information, array information, and key-value pairs to be updated;

[0013] Based on the configuration file, the primary key information and the array information are mapped to an array path to obtain path information;

[0014] The path information and the key value to be updated are combined to obtain the mapped data to be updated.

[0015] Optionally, the step of partially updating the aggregated document stored in the transit database based on the mapped data to be updated, to obtain the updated aggregated document, includes:

[0016] Based on the path information of the mapped data to be updated, locate the aggregated document to obtain the position to be updated;

[0017] The key value corresponding to the position to be updated in the aggregated document is updated to the key value to be updated, thus obtaining the updated aggregated document.

[0018] Optionally, before the step of performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain the path information, the method further includes:

[0019] Determine whether the data to be updated belongs to the main table;

[0020] The step of performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain the path information includes:

[0021] If the data to be updated does not belong to the main table, then based on the configuration file, the primary key information and the array information are mapped using an array path to obtain the path information.

[0022] Optionally, the step of obtaining the data to be updated includes:

[0023] Obtain the data to be updated through the preset update window;

[0024] The step of performing array path mapping on the data to be updated to obtain the mapped data to be updated includes:

[0025] Calculate the number of data items to be updated within the update window to obtain the number of data items to be updated.

[0026] When the number of data items to be updated meets the preset update number threshold or the queue time of the update window meets the preset update time threshold, the data items to be updated are deduplicated to obtain the deduplicated data items to be updated.

[0027] The deduplicated data to be updated is subjected to path mapping to obtain the mapped data to be updated.

[0028] Optionally, after the step of partially updating the aggregated document stored in the transit database based on the mapped data to be updated, to obtain the updated aggregated document, the method further includes:

[0029] Send the primary key to be queried to the restore window;

[0030] Calculate the number of primary keys to be queried within the restore window to obtain the number of primary keys to be queried;

[0031] The step of restoring the updated aggregated documents into an array based on the primary key to be queried to obtain the final document includes:

[0032] When the number of records of the primary key to be queried meets the preset query record number threshold condition or the queue time of the restoration window meets the preset restoration time threshold condition, the primary key to be queried is deduplicated to obtain the deduplicated primary key to be queried.

[0033] Based on the deduplicated primary key to be queried, the updated aggregated documents are restored as an array to obtain the final document.

[0034] Optionally, before the step of performing array path mapping processing on the data to be updated to obtain the mapped data to be updated, the method further includes:

[0035] Based on the data to be updated, the data table of the data to be updated is type-identified to obtain the data table type;

[0036] The configuration file is obtained by configuring the data table type.

[0037] This application also proposes a data updating device, which includes:

[0038] The data acquisition module is used to acquire the data to be updated.

[0039] The data mapping module is used to perform array path mapping processing on the data to be updated to obtain the mapped data to be updated.

[0040] The primary key extraction module is used to extract the primary key information of the mapped data to be updated to obtain the primary key to be queried.

[0041] The data update module is used to partially update the aggregated document stored in the transit database according to the mapped data to be updated, so as to obtain the updated aggregated document.

[0042] The data restoration module is used to restore the updated aggregated documents into an array based on the primary key to be queried, so as to obtain the final document.

[0043] This application also proposes a terminal device, which includes a memory, a processor, and a data update program stored in the memory and executable on the processor. When the data update program is executed by the processor, it implements the steps of the data update method described above.

[0044] This application also proposes a computer-readable storage medium storing a data update program, which, when executed by a processor, implements the steps of the data update method described above.

[0045] The data update method, apparatus, terminal device, and storage medium proposed in this application embodiment involve: acquiring data to be updated; performing array path mapping on the data to be updated to obtain mapped data to be updated; extracting the primary key information of the mapped data to obtain a primary key to be queried; performing partial updates on aggregated documents stored in a transit database based on the mapped data to be updated to obtain an updated aggregated document; and restoring the updated aggregated document into an array based on the primary key to be queried to obtain a final document. Specifically, by performing array path mapping on the data to be updated to obtain mapped data to be updated without an array structure, and a primary key to be queried corresponding to the mapped data to be updated; then performing partial updates on aggregated documents stored in a transit database based on the mapped data to be updated to obtain an updated aggregated document; finally, restoring the updated aggregated document into an array based on the primary key to be queried to obtain a final document and storing the final document in a final database. The transit database and the final database have different data structures; the transit database stores data in a flattened path format, while the final database stores data in an array format. The above process maps the complex array structure of the data to be updated to a simpler data structure. Because the data format of the mapped data is consistent with the aggregate document and the corresponding aggregate document is stored in advance in the intermediate database, the aggregate document in the intermediate database can be updated directly using a simple local update method, without having to traverse the complex array structure in the final database to retrieve the aggregate document. This reduces the frequency of data interaction with the final database, alleviates the database load, improves the database response speed and concurrency processing capabilities, and increases data update efficiency. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of the functional modules of the terminal equipment to which the data update device belongs in this application;

[0047] Figure 2 This is a flowchart illustrating a first exemplary embodiment of the data update method of this application;

[0048] Figure 3 This is a schematic diagram illustrating the overall process framework for data updating and restoration involved in this application;

[0049] Figure 4 This is a schematic diagram of the configuration file structure involved in this application;

[0050] Figure 5 This is a schematic diagram of the data mapping process to be updated involved in this application;

[0051] Figure 6 This is a schematic diagram of the aggregated document update process involved in this application;

[0052] Figure 7 This is a schematic diagram of the node structure of the aggregated document involved in this application.

[0053] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0055] The main solution of this application embodiment is as follows: First, obtain the data to be updated; second, perform array path mapping on the data to be updated to obtain mapped data to be updated; third, extract the primary key information of the mapped data to be updated to obtain the primary key to be queried; fourth, perform partial updates on the aggregated documents stored in the transit database based on the mapped data to be updated to obtain updated aggregated documents; fifth, perform array restoration on the updated aggregated documents based on the primary key to be queried to obtain the final document. Specifically, by performing array path mapping on the data to be updated, mapped data to be updated without an array structure and the primary key to be queried corresponding to the mapped data to be updated are obtained; then, based on the mapped data to be updated, the aggregated documents stored in the transit database are partially updated to obtain updated aggregated documents; finally, the updated aggregated documents are restored to an array based on the primary key to be queried to obtain the final document, and the final document is stored in the final database. The transit database and the final database have different data structures; the transit database stores data in a flattened path format, while the final database stores data in an array format. The above process maps the complex array structure of the data to be updated to a simpler data structure. Because the data format of the mapped data is consistent with the aggregate document and the corresponding aggregate document is stored in advance in the intermediate database, the aggregate document in the intermediate database can be updated directly using a simple local update method, without having to traverse the complex array structure in the final database to retrieve the aggregate document. This reduces the frequency of data interaction with the final database, alleviates the database load, improves the database response speed and concurrency processing capabilities, and increases data update efficiency.

[0056] This application's embodiments take into account that multi-table join queries are a common database query operation, typically used in business scenarios involving multiple tables. Specifically, in business scenarios involving multi-table join queries, to reduce the consumption of the business database, the data from multiple tables is usually stored in an Elasticsearch (search server) database, using the statistical and query capabilities of the Elasticsearch database to meet the needs of the business scenario.

[0057] However, this data storage method of storing multi-table data in Elasticsearch suffers from low data update efficiency. Specifically, updating data in the Elasticsearch database requires first querying the corresponding data document in the database for the data to be updated, then aggregating several data documents to obtain an aggregate document, then updating the data to be updated and overwriting the aggregate document, and finally writing the updated aggregate document back to the Elasticsearch database. Understandably, this data update process is cumbersome and reduces efficiency. This is especially true for complex update methods involving aggregating sub-documents into arrays before storing them. Specifically, in this case, the complete aggregate document must first be retrieved from the Elasticsearch database, then the array must be traversed and the sub-documents located before the update operation can be performed, and finally the updated aggregate document is written back to the Elasticsearch database. This process involves multiple interactions with the Elasticsearch database due to the need to traverse the array and query several data documents to aggregate the document, resulting in low document update efficiency.

[0058] Based on this, this application proposes a solution: First, the data to be updated is mapped using an array path to obtain mapped data without an array structure, along with the corresponding primary key to be queried. Then, based on the mapped data, a partial update is performed on the aggregated document stored in the intermediary database to obtain the updated aggregated document. Finally, the updated aggregated document is restored using the primary key to be queried to obtain the final document, which is then stored in the final database. The intermediary database and the final database have different data structures; the intermediary database stores data in a flattened path format, while the final database stores data in an array format. This process maps the complex array structure of the data to be updated to a simpler, mapped data structure. Because the format of the mapped data is consistent with the aggregated document and the corresponding aggregated document is pre-stored in the intermediary database, the aggregated document in the intermediary database can be updated directly using a simple partial update method, without needing to traverse and query the complex array structure in the final database. This reduces the frequency of data interaction with the final database, alleviates the database load, and improves the database's response speed and concurrency processing capabilities.

[0059] Specifically, refer to Figure 1 , Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the data update device belongs in this application. The data update device can be a device that performs data processing independently of the terminal device, or it can be carried on the terminal device in the form of hardware or software.

[0060] In this embodiment, the terminal device to which the data update device belongs includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0061] The memory 130 stores the operating system and data update program to obtain the data to be updated;

[0062] The data to be updated is processed by array path mapping to obtain mapped data to be updated; the primary key information of the mapped data to be updated is extracted to obtain the primary key to be queried; based on the mapped data to be updated, the aggregated document stored in the transit database is partially updated to obtain the updated aggregated document; based on the primary key to be queried, the updated aggregated document is array restored to obtain the final document and stored in the memory 130; the output module 110 may be a display screen, speaker, etc. The communication module 140 may include a WIFI module, a mobile communication module, and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0063] When the data update program in memory 130 is executed by the processor, it performs the following steps:

[0064] Obtain the data to be updated; perform array path mapping on the data to be updated to obtain the mapped data to be updated; extract the primary key information of the mapped data to be updated to obtain the primary key to be queried; perform partial updates on the aggregated documents stored in the transit database based on the mapped data to be updated to obtain the updated aggregated documents; perform array restoration on the updated aggregated documents based on the primary key to be queried to obtain the final document.

[0065] Furthermore, when the data update program in memory 130 is executed by the processor, it also performs the following steps:

[0066] Based on a preset configuration file, the data structure of the data to be updated is identified to obtain primary key information, array information, and key-value pairs to be updated. Based on the configuration file, the primary key information and the array information are mapped using an array path to obtain path information. The path information and the key-value pairs to be updated are then combined to obtain the mapped data to be updated.

[0067] Furthermore, when the data update program in memory 130 is executed by the processor, it also performs the following steps:

[0068] Based on the path information of the mapped data to be updated, the aggregated document is located to obtain the position to be updated; the key value corresponding to the position to be updated in the aggregated document is updated to the key value to be updated, and the updated aggregated document is obtained.

[0069] Furthermore, when the data update program in memory 130 is executed by the processor, it also performs the following steps:

[0070] The step of determining whether the data to be updated belongs to the main table; and the step of performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain the path information includes: if the data to be updated does not belong to the main table, then performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain the path information.

[0071] Furthermore, when the data update program in memory 130 is executed by the processor, it also performs the following steps:

[0072] The step of obtaining the data to be updated through a preset update window and performing array path mapping on the data to be updated to obtain the mapped data to be updated includes: calculating the number of data items to be updated within the update window to obtain the number of data items to be updated; when the number of data items to be updated meets a preset update item count threshold or the queue time of the update window meets a preset update time threshold, deduplicating the data to be updated to obtain the deduplicated data to be updated; and performing path mapping on the deduplicated data to be updated to obtain the mapped data to be updated.

[0073] Furthermore, when the data update program in memory 130 is executed by the processor, it also performs the following steps:

[0074] The steps of sending the primary key to be queried to the restore window; calculating the number of primary keys to be queried in the restore window to obtain the number of primary keys to be queried; and restoring the updated aggregated documents into an array based on the primary keys to obtain the final document include: when the number of primary keys to be queried meets a preset query count threshold or the queue time of the restore window meets a preset restore time threshold, deduplicating the primary keys to be queried to obtain deduplicated primary keys to be queried; and restoring the updated aggregated documents into an array based on the deduplicated primary keys to obtain the final document.

[0075] Furthermore, when the data update program in memory 130 is executed by the processor, it also performs the following steps:

[0076] Based on the data to be updated, the data table of the data to be updated is type-identified to obtain the data table type; and the configuration is performed according to the data table type to obtain the configuration file.

[0077] This embodiment, through the above scheme, obtains the data to be updated; performs array path mapping on the data to be updated to obtain mapped data to be updated; extracts the primary key information of the mapped data to be updated to obtain the primary key to be queried; performs partial updates on the aggregated documents stored in the transit database based on the mapped data to be updated to obtain updated aggregated documents; and restores the updated aggregated documents to the array based on the primary key to be queried to obtain the final document. By performing array path mapping on the data to be updated to obtain mapped data to be updated without an array structure, and the primary key to be queried corresponding to the mapped data to be updated; then, based on the mapped data to be updated, performs partial updates on the aggregated documents stored in the transit database to obtain updated aggregated documents; finally, restores the updated aggregated documents to the array based on the primary key to be queried to obtain the final document and stores the final document in the final database. The transit database and the final database have different data structures; the transit database stores data in a flat path format, while the final database stores data in an array format. The above process maps the complex array structure of the data to be updated to a simpler data structure. Because the data format of the mapped data is consistent with the aggregate document and the corresponding aggregate document is stored in advance in the intermediate database, the aggregate document in the intermediate database can be updated directly using a simple local update method, without having to traverse the complex array structure in the final database to retrieve the aggregate document. This reduces the frequency of data interaction with the final database, alleviates the database load, improves the database response speed and concurrency processing capabilities, and increases data update efficiency.

[0078] Based on, but not limited to, the terminal device architecture described above, this application proposes method embodiments.

[0079] Reference Figure 2 , Figure 2 This is a flowchart illustrating a first exemplary embodiment of the data update method of this application.

[0080] An embodiment of the present invention provides a data update method, the method comprising:

[0081] Step S10: Obtain the data to be updated;

[0082] Multi-table joins are a common database query operation, typically used in business scenarios involving multiple tables. Multi-table joins require joining two or more tables to retrieve related data, thereby providing more comprehensive information.

[0083] Multi-table joins are a common database query operation, typically used in business scenarios involving multiple tables. Specifically, to reduce the overhead on the business database, multi-table data is usually stored in an Elasticsearch database, utilizing Elasticsearch's statistical and query capabilities to meet the needs of the business scenario.

[0084] However, this data storage method of storing multi-table data in Elasticsearch suffers from low data update efficiency. Specifically, updating data in the Elasticsearch database requires first querying the corresponding data document in the database for the data to be updated, then aggregating several data documents to obtain an aggregate document, then updating the data to be updated and overwriting the aggregate document, and finally writing the updated aggregate document back to the Elasticsearch database. Understandably, this data update process is cumbersome and reduces efficiency. This is especially true for complex update methods involving aggregating sub-documents into arrays before storing them. Specifically, in this case, the complete aggregate document must first be retrieved from the Elasticsearch database, then the array must be traversed and the sub-documents located before the update operation can be performed, and finally the updated aggregate document is written back to the Elasticsearch database. This process involves multiple interactions with the Elasticsearch database due to the need to traverse the array and query several data documents to aggregate the document, resulting in low document update efficiency.

[0085] Multi-table joins are a common database query operation. Taking an e-commerce website as an example, it may need to access user information tables, order information tables, and product information tables simultaneously. These tables typically have different data structures and fields, but there are some kind of relationship between them. For example, the order table may contain product IDs and user IDs, while the user information table may contain user IDs and basic user information.

[0086] More complexly, data tables are often linked in the form of data. Taking an order table as an example, product IDs might be stored as an array. Specifically, an order may be associated with multiple products, and these product IDs would be stored as an array in the "ProductIDs" field of the order table, forming multiple sub-documents. Such multi-table joins can help implement more complex business logic, but when updates involve sub-documents, it is necessary to traverse the array and locate the sub-document before the data update and overwrite operation can be performed, resulting in low data update efficiency.

[0087] Therefore, this embodiment proposes to map the data to be updated using an array path to obtain the mapped data without an array structure, and the corresponding primary key to be queried for the mapped data. Then, based on the mapped data, a partial update is performed on the aggregated document stored in the transit database to obtain the updated aggregated document. Finally, the updated aggregated document is restored using the primary key to be queried to obtain the final document, which is then stored in the final database. The transit database and the final database have different data structures; the transit database stores data in a flattened path format, while the final database stores data in an array format. This process maps the complex array structure of the data to be updated to a simpler, mapped data structure. Because the format of the mapped data is consistent with the aggregated document and the corresponding aggregated document is pre-stored in the transit database, the aggregated document in the transit database can be updated directly using a simple partial update method, without needing to traverse and query the complex array structure in the final database to retrieve the aggregated document. This reduces the frequency of data interaction with the final database, alleviates the database load, improves the database's response speed and concurrency processing capabilities, and thus improves data update efficiency.

[0088] Specifically, the data to be updated can be generated by business operations. Based on specific business needs and query conditions, SQL statements can be used to perform multi-table join queries to obtain the data that needs to be updated.

[0089] Taking an e-commerce website as an example, the data to be updated could be product information updates, which are stored in multiple tables, including product tables, category tables, brand tables, etc.; or order information updates, which include order tables, user tables, product tables, etc.; or user information updates, which include user tables, shipping address tables, shopping cart tables, etc.

[0090] Step S20: Perform array path mapping processing on the data to be updated to obtain the mapped data to be updated;

[0091] Specifically, in order to transform the complex array structure of the data to be updated into a simple structure, array path mapping can be performed on the data to be updated, which transforms the complex array structure into simple path information.

[0092] Specifically, by using array path mapping, the array information of the data to be updated can be transformed into more flexible and general path information. More specifically, configuration files can be used to identify the structure information (including primary keys and arrays) of the data to be updated (the table to be updated), and the structure information can be mapped to path information.

[0093] Among them, path mapping can adapt to different data table structures and field naming conventions, making the code more versatile and reusable in multiple business scenarios.

[0094] Step S30: Extract the primary key information of the mapped data to be updated to obtain the primary key to be queried;

[0095] To ensure that the final document restoration order is consistent with the original data structure, the primary key to be queried can be obtained by extracting the primary key information of the mapped data to be updated. The primary key to be queried is used for the array restoration of the final document, because the final document needs to be restored according to the original data structure, thereby avoiding data restoration errors and improving the reliability and stability of the system.

[0096] In addition, the transit database is also used to temporarily store data, that is, it can accumulate the amount of aggregated document updates over a certain period of time. After a certain period of time, the aggregated document updates are restored and overwritten into the final documents in the final database. In other words, the amount of aggregated document updates can be accumulated over a certain period of time by accumulating the primary keys to be queried.

[0097] Step S40: Based on the mapped data to be updated, perform a partial update on the aggregated document stored in the transit database to obtain the updated aggregated document;

[0098] To avoid having to query the final database before aggregating documents for each data update, the aggregated documents can be stored in an intermediate database first. Then, based on the mapped data to be updated, the aggregated documents stored in the intermediate database can be partially updated.

[0099] Specifically, the intermediary database can be MongoDB. MongoDB is a document-oriented NoSQL database that stores data using a flexible data model. Compared to traditional relational databases, MongoDB does not require strict definition of table structures and field types, allowing it to freely store various forms of aggregated document data and adapt to changing business needs.

[0100] Meanwhile, MongoDB has high-performance read and write capabilities, which can quickly process large amounts of data and is suitable for high-concurrency scenarios. Through reasonable index design and sharding technology, read and write performance can be further improved to meet the real-time requirements of data update.

[0101] MongoDB supports horizontal scaling, allowing storage capacity and processing power to be expanded by adding more nodes and shards. This enables MongoDB to act as a relay database to meet the needs of large-scale data aggregation and processing, ensuring the system's scalability and performance advantages.

[0102] In summary, using MongoDB as an intermediary database to store aggregated documents with simple data structures can fully leverage MongoDB's flexibility, high performance, scalability, and rich query capabilities, improving data update efficiency and system stability, while also providing support for data backup and recovery.

[0103] For transit databases (MongoDB), basic partial update statements can be used to update aggregate documents, which is more efficient.

[0104] Understandably, since the data format of the mapped data to be updated is consistent with that of the aggregate document, the aggregate document in the transit database can be updated directly using a simple partial update method.

[0105] Specifically, partial updates only require modification of the fields or data that need to be updated, without replacing or rewriting the entire document. Unlike existing technologies, which require querying the complete aggregate document before updating the final database, overwriting the data to be updated into the aggregate document, and then writing the updated aggregate document back to the ElasticSearch database, partial updates can greatly reduce the amount of data interaction and network transmission overhead, thereby improving the update speed.

[0106] Furthermore, partial updates only modify the fields or data that need updating, without triggering read and write operations on the entire document. This reduces database I / O operations and computational overhead, lowers the database load, and improves the database's response speed and concurrency processing capabilities.

[0107] Furthermore, the introduction of a transit database allows aggregated documents to be stored in advance, eliminating the need to retrieve the complete aggregated document for each data update, as is the case with existing technologies, thus significantly accelerating the efficiency of data updates.

[0108] Furthermore, the transit database can also serve as a caching layer, storing frequently queried data to improve query performance and response speed. Simultaneously, the transit database can optimize query results through operations such as aggregation and indexing, further enhancing query efficiency.

[0109] In summary, the introduction of a transit database can reduce the amount of data interaction with the final database, alleviate the database load, and improve data update efficiency. At the same time, the transit database can also provide caching and query optimization functions, protect the integrity of the final database, thereby improving the performance and stability of the entire system.

[0110] Step S50: Based on the primary key to be queried, restore the updated aggregated documents into an array to obtain the final document.

[0111] Specifically, the final database can be an ElasticSearch database.

[0112] After the update is successfully written to the intermediary database (MongoDB) and the updated aggregated document is obtained, the primary key to be queried will be pushed. Specifically, such as... Figure 6 As shown, the complete updated aggregate document can be retrieved from the Mongo database based on the primary key to be queried. At this time, the complete updated aggregate document retrieved is a single-level document. Then, the updated aggregate document is restored as an array to obtain the final document containing the array structure. The final document will then be written to the final database (ElasticSearch database).

[0113] Specifically, such as Figure 7 As shown, in the updated aggregated document retrieved, each key can be viewed as a path for that key-value pair within the document, and each array name or actual key name within the path can be considered a node. Therefore, the updated aggregated document can be viewed as a document tree. Then, by performing a depth-first traversal of the entire document tree, when an array node is encountered, the document tree within the array is traversed and restored first. After the traversal is complete, the updated aggregated document can be restored to the final document.

[0114] Overall, this solution addresses the inefficient update method in existing technologies for multi-table aggregated documents, where each update to a single table requires first querying the aggregated document and then overwriting the existing data. This is achieved by introducing a MongoDB database as an intermediate storage for aggregated documents.

[0115] The data update method proposed in this application involves: acquiring data to be updated; performing array path mapping on the data to be updated to obtain mapped data to be updated; extracting the primary key information of the mapped data to be updated to obtain a primary key to be queried; performing partial updates on aggregated documents stored in a transit database based on the mapped data to be updated to obtain updated aggregated documents; and restoring the updated aggregated documents into an array based on the primary key to be queried to obtain the final document. Specifically, by performing array path mapping on the data to be updated to obtain mapped data to be updated without an array structure, and a primary key to be queried corresponding to the mapped data to be updated; then, performing partial updates on aggregated documents stored in a transit database based on the mapped data to be updated to obtain updated aggregated documents; finally, restoring the updated aggregated documents into an array based on the primary key to be queried to obtain the final document and storing the final document in a final database. The transit database and the final database have different data structures; the transit database stores data in a flattened path format, while the final database stores data in an array format. The above process maps the complex array structure of the data to be updated to a simpler data structure. Because the data format of the mapped data is consistent with the aggregate document and the corresponding aggregate document is stored in advance in the intermediate database, the aggregate document in the intermediate database can be updated directly using a simple local update method, without having to traverse the complex array structure in the final database to retrieve the aggregate document. This reduces the frequency of data interaction with the final database, alleviates the database load, improves the database response speed and concurrency processing capabilities, and increases data update efficiency.

[0116] Based on the first embodiment, a second embodiment of this application is proposed. The difference between the second embodiment and the first embodiment is as follows:

[0117] Step S20, which involves performing array path mapping on the data to be updated to obtain the mapped data to be updated, can be further refined. This step may include:

[0118] Step S21: Based on the preset configuration file, identify the data structure of the data to be updated to obtain primary key information, array information and key-value pairs to be updated;

[0119] Step S23: Based on the configuration file, perform array path mapping processing on the primary key information and the array information to obtain path information;

[0120] Step S24: Combine the path information and the key value to be updated to obtain the mapped data to be updated.

[0121] Specifically, a configuration file is used to identify the data structure corresponding to each piece of data to be updated in the aggregate document. The data structure includes a primary key, an array, and a key-value pair to be updated. The primary key information and the array information are mapped using an array path to obtain path information. Finally, the path information is added to the key of the data to be updated as a prefix, and the suffix is ​​the key-value pair to be updated. The prefix and suffix are combined to obtain the mapped data to be updated. The mapped aggregate document does not contain an array structure.

[0122] Specifically, each piece of data to be updated is stored in a corresponding data table. In this embodiment, it is proposed to map the data to be updated through a corresponding configuration file. Each type of data table should have a corresponding configuration file.

[0123] Specifically, such as Figure 4 As shown, the configuration file for each data table needs to include both the mapping between the primary key of the concatenated document and the table's hierarchical structure within the aggregated document. For data belonging to deeper levels, the hierarchy configuration needs to include the primary key of each level above that table, the array name of each level, and the field name that serves as its own primary key. This configuration method ensures universality across various aggregation structures.

[0124] Understandably, the mapping relationship between fields and paths can be customized according to the configuration file, flexibly adapting to different data structures and data sources.

[0125] Furthermore, when the data table structure changes, only the mapping relationship in the configuration file needs to be updated, without modifying the actual data processing logic, which improves the scalability and maintainability of the system.

[0126] To further illustrate, let's take three tables, A, B, and C, as an example. Table A serves as the main table, associated with multiple records in table B. Table B is associated with multiple records in table C. Ultimately, these are aggregated into an aggregate document consisting of table A, an array containing multiple records from table B, and an array containing multiple records from table C. The position of each table (each piece of data to be updated) within the aggregate document is represented by its corresponding configuration file. The configuration file needs to identify the primary key of the aggregate document, the hierarchy depth of the table (data to be updated) within the aggregate document, and the primary key corresponding to each level. Taking the configuration file for table B as an example... Figure 4 As shown, the table B is located at the following level in the aggregated document: 2; the parent primary key: A_primary key; the data group name to which table B belongs in the parent level: array B; and the primary key of the document containing table B: B_primary key, etc.

[0127] Furthermore, with Figure 5 For example, Figure 5 This includes the mapping process of the data to be updated and the process of partially updating the aggregated document. Figure 5In the diagram, the left image shows the data to be updated, the middle image shows the configuration file, and the right image shows the updated aggregated document. Specifically, there are four pieces of data to be updated. After mapping these four pieces of data, the aggregated document is partially updated based on the mapped data. One piece of data to be updated is located in table A, two pieces in table B, and one piece in table C. Table B is a sub-table of table A (table A contains array B), and table C is a sub-table of table B (table B contains array C). Array B contains two pieces of data.

[0128] Taking a piece of data to be updated in table B as an example, the data contains three key-value pairs (each key-value pair contains primary key information, array information, and the key value to be updated), namely A_primary key: A1, B_primary key: B1, and B_field1: BB1. It can be understood that the left side of the ":" is the primary key information and array information, and the right side of the ":" is the key value to be updated.

[0129] Through such Figure 5 The intermediate configuration file maps a piece of data to be updated in table B. Specifically, the path information is converted from three key-value pairs: array B / B1 / A_primary key, array B / B1 / B_primary key, and array B / B1 / B_field1. These are then combined with the corresponding key-value pairs to obtain B / B1 / A_primary key: A1, array B / B1 / B_primary key: B1, and array B / B1 / B_field1: BB1, thus obtaining the mapped data to be updated.

[0130] The data update method proposed in this application identifies the data structure of the data to be updated based on a preset configuration file, obtaining primary key information, array information, and key-value pairs to be updated. Based on the configuration file, the primary key information and the array information are mapped using an array path to obtain path information. The path information and the key-value pairs to be updated are combined to obtain the mapped data to be updated. The mapping relationship between fields and paths can be customized according to the configuration file, flexibly adapting to different data structures and data sources. The mapped data to be updated does not contain an array structure.

[0131] Based on the second embodiment, a third embodiment of this application is proposed. The difference between the third embodiment and the second embodiment is as follows:

[0132] Step S40, which involves partially updating the aggregated document stored in the transit database based on the mapped data to be updated, is further refined. The refined steps include:

[0133] Step S41: Based on the path information of the mapped data to be updated, locate the aggregated document to obtain the position to be updated;

[0134] Step S42: Update the key value corresponding to the position to be updated in the aggregated document to the key value to be updated, and obtain the updated aggregated document.

[0135] Understandably, since the mapped data to be updated is obtained through a configuration file, and the configuration file contains the correspondence between the primary key of the concatenated document and the table fields, as well as the hierarchical structure of the table within the aggregate document, the aggregate document can be located and the position to be updated can be obtained through the path information of the mapped data to be updated.

[0136] Then, the key value corresponding to the position to be updated in the aggregated document is updated to the key value to be updated, thus obtaining the updated aggregated document.

[0137] Specifically, the primary key and array of the data to be updated are extracted from the configuration file and converted into path information to obtain the mapped data to be updated. The mapped data to be updated contains path information. Based on the path information, the aggregated document can be located to obtain the position to be updated. Finally, the key value corresponding to the position to be updated in the aggregated document is updated to the key value to be updated, thus completing the update of the aggregated document and obtaining the updated aggregated document.

[0138] Understandably, path information allows for direct location of the precise location of the data to be updated, eliminating the need to traverse the entire aggregated document. This saves time and resources and improves the efficiency of update operations.

[0139] Regardless of whether the data to be updated is located in nested objects, arrays, or other complex data structures, it can be accurately located and updated using path information.

[0140] Most importantly, using path information to locate the position to be updated allows for partial updates, modifying only the fields or data that need updating without triggering a replacement or rewrite of the entire document. This reduces network traffic and database operations, improving update efficiency.

[0141] The data update method proposed in this application locates the aggregated document and obtains the position to be updated based on the path information of the mapped data to be updated; then, the key value corresponding to the position to be updated in the aggregated document is updated to the key value to be updated, resulting in the updated aggregated document. Using path information to locate the position to be updated enables partial updates, modifying only the fields or data that need updating without triggering a replacement or rewrite of the entire document. This reduces network transmission volume and database operations, improving update efficiency.

[0142] Based on the second embodiment, a fourth embodiment of this application is proposed. The difference between the fourth embodiment and the second embodiment is as follows:

[0143] For step S23, based on the configuration file, the primary key information and the array information are processed by array path mapping. Before obtaining the path information, supplementation is performed, wherein the supplementation steps include:

[0144] Step S22: Determine whether the data to be updated belongs to the main table;

[0145] In this embodiment, step S23, the step of performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain path information, includes:

[0146] Step S231: If the data to be updated does not belong to the main table, then based on the configuration file, the primary key information and the array information are mapped using an array path to obtain the path information.

[0147] Before writing to the intermediate database (MongoDB), different processing methods can be determined based on the data's position within the aggregate document. If the data to be updated belongs to the main table, it is directly updated to the MongoDB. If the data to be updated belongs to a child table, the primary key information and the array information are mapped using an array path before being updated to the MongoDB.

[0148] Understandably, directly updating the main table data reduces additional path mapping steps and improves the efficiency of update operations. Without complex path mapping, the main table data can be directly updated to the MongoDB database, reducing intermediate steps and speeding up the update process.

[0149] The data update method proposed in this application determines whether the data to be updated belongs to the main table. The step of performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain path information includes: if the data to be updated does not belong to the main table, then performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain the path information. Directly updating the main table data can reduce additional path mapping processing steps and improve the efficiency of the update operation. It eliminates the need for complex path mapping processing, allowing direct updates of the main table data to the Mongo database, reducing intermediate steps and accelerating the update speed.

[0150] Based on the first embodiment, a fifth embodiment of this application is proposed. The difference between the fifth embodiment and the first embodiment is as follows:

[0151] Step S10, the step of obtaining the data to be updated, can be further refined, and the refined steps may include:

[0152] Step S11: Obtain the data to be updated through a preset update window;

[0153] Considering the large amount of data to be updated in actual operations, for frequently updated data sources, triggering writes to the Elasticsearch or MongoDB database immediately after each update not only leads to additional network overhead due to the high frequency of update requests, but also fails to fully utilize the batch processing capabilities provided by the Elasticsearch database, thus reducing the overall throughput of the Elasticsearch database.

[0154] Therefore, this embodiment proposes that for high-frequency update scenarios, there is an update window with the maximum number of documents to be updated and the maximum window time as thresholds. Only when either threshold condition is triggered will the documents to be updated in the window be written to the Mongo database.

[0155] Specifically, the update window can act as a queue cache to temporarily store data awaiting updates. In a multi-threaded or distributed environment, when a large amount of data needs to be updated, placing the data to be updated into the update window's queue can reduce the load on the database. Simultaneously, the queue can also provide concurrent processing capabilities, allowing multiple threads to process the data to be updated simultaneously, improving system performance and throughput.

[0156] Further, in this embodiment, step S20, which involves performing array path mapping processing on the data to be updated to obtain the mapped data to be updated, includes:

[0157] Step S25: Calculate the number of data items to be updated within the update window to obtain the number of data items to be updated;

[0158] Step S26: When the number of data items to be updated meets the preset update number threshold condition or the queue time of the update window meets the preset update time threshold condition, the data items to be updated are deduplicated to obtain the deduplicated data items to be updated.

[0159] Step S27: Perform path mapping processing on the deduplicated data to be updated to obtain the mapped data to be updated.

[0160] By setting preset update time thresholds and preset update count thresholds, the frequency of update operations can be controlled. Deduplication and actual updates are only performed when one of these conditions is met, avoiding frequent update requests and reducing system load.

[0161] Understandably, deduplicating data before batch updating reduces frequent database accesses and lowers database load. This is especially beneficial in high-concurrency scenarios, as it reduces the number of individual update requests, thus improving system stability and performance.

[0162] Furthermore, batch updating large amounts of data can optimize network transmission efficiency and reduce network communication overhead. Merging multiple pieces of data to be updated into a single update request reduces network transmission overhead and improves the efficiency of data updates.

[0163] Specifically, the process of deduplicating data to be updated involves determining whether the data to be updated in the update window has the same primary key. If they do, the data to be updated will be deduplicated first, that is, the latest data to be updated will be used to update the aggregated document, reducing unnecessary update operations.

[0164] This embodiment proposes a near real-time data synchronization and updating scheme for an Elasticsearch database. It also addresses the issue of not being able to fully utilize the batch processing capabilities of Elasticsearch in scenarios with high-frequency data updates.

[0165] The data update method proposed in this application obtains data to be updated through a preset update window. The step of performing array path mapping processing on the data to be updated to obtain mapped data to be updated includes: calculating the number of data items to be updated within the update window to obtain the number of data items to be updated; when the number of data items to be updated meets a preset update item count threshold or the queue time of the update window meets a preset update time threshold, deduplicating the data to be updated to obtain deduplicated data to be updated; and performing path mapping processing on the deduplicated data to be updated to obtain the mapped data to be updated, thereby realizing a near real-time data synchronization update scheme for the Elasticsearch database. Simultaneously, it solves the problem that the batch processing characteristics of the Elasticsearch database cannot be fully utilized in high-frequency data update scenarios.

[0166] Based on the first embodiment, a sixth embodiment of this application is proposed. The difference between the sixth embodiment and the first embodiment is as follows:

[0167] Step S40, which involves partially updating the aggregated document stored in the transit database based on the mapped data to be updated, is supplemented after this step. The supplementary steps may include:

[0168] Step S43: Send the primary key to be queried to the restore window;

[0169] Step S44: Calculate the number of primary keys to be queried within the restore window to obtain the number of primary keys to be queried;

[0170] Considering the large number of primary keys to be queried in actual restoration, for frequently updated data sources, triggering writes to the Elasticsearch or MongoDB database immediately after each update would not only lead to additional network overhead due to the high frequency of update requests, but also prevent the full utilization of the batch processing features provided by the Elasticsearch database, thus reducing the overall throughput of the Elasticsearch database.

[0171] Therefore, in this embodiment, the primary key to be queried is sent to the restore window. The restore window, acting as a queue cache, effectively manages the data flow of the primary key to be queried, temporarily storing it in the queue for later processing. Through queue caching, a large number of primary keys to be queried can be processed smoothly, avoiding the impact of sudden requests on the system.

[0172] Specifically, after the mapped data to be updated is written to the intermediary database (MongoDB), the primary key to be queried is sent downstream (to the restore window). Before restoring the updated aggregated document based on the primary key to be queried, it is necessary to determine whether either the number of primary key records to be queried or the time window size is met.

[0173] If any condition is met, iterate through the primary keys to be queried in the restoration window and remove duplicates. Then, query the aggregated documents from the Mongo database in batches according to the primary keys to be queried, restore the aggregated documents to the final documents, and finally write them in batches to the final database (ElasticSearch database).

[0174] Further, in this embodiment, step S50, which involves restoring the updated aggregated documents into an array based on the primary key to be queried, to obtain the final document, includes:

[0175] Step S54: When the number of records of the primary key to be queried meets the preset query record number threshold condition or the queue time of the restoration window meets the preset restoration time threshold condition, the primary key to be queried is deduplicated to obtain the deduplicated primary key to be queried.

[0176] Step S55: Based on the deduplicated primary key to be queried, the updated aggregated documents are restored as an array to obtain the final document.

[0177] Specifically, setting preset query count thresholds and preset restore time thresholds can control the frequency of query operations. Only when one of these conditions is met will the primary key to be queried be deduplicated, and then the updated aggregated documents will be restored as an array. This avoids frequent query requests and reduces the system load.

[0178] By using queue caching and deduplication operations, data processing can be optimized, improving system response speed and concurrency capabilities. Properly managing the flow of primary keys to be queried and deduplication processes is beneficial for improving overall system performance.

[0179] The data update method proposed in this application involves sending the primary key to be queried to a restoration window; calculating the number of primary keys to be queried within the restoration window to obtain the number of records of the primary key to be queried; and performing array restoration on the updated aggregated documents based on the primary key to obtain the final document. The steps include: when the number of records of the primary key to be queried meets a preset query record threshold or the queue time of the restoration window meets a preset restoration time threshold, deduplicating the primary key to be queried to obtain a deduplicated primary key; and performing array restoration on the updated aggregated documents based on the deduplicated primary key to obtain the final document. Through queue caching and deduplication operations, the data processing flow can be optimized, improving the system's response speed and concurrent processing capabilities. Reasonable management of the flow of primary keys to be queried and deduplication processing is beneficial to improving the overall performance of the system.

[0180] Based on the second embodiment, a seventh embodiment of this application is proposed. The difference between the seventh embodiment and the second embodiment is as follows:

[0181] Before step S20, which involves performing array path mapping on the data to be updated to obtain the mapped data to be updated, the following supplementary steps are provided:

[0182] Step S01: Based on the data to be updated, perform type identification on the data table of the data to be updated to obtain the data table type;

[0183] Step S02: Configure the data table according to the data table type to obtain the configuration file.

[0184] Specifically, the data type is first identified based on the data table containing the data to be updated, and then the configuration file is obtained through the data table type.

[0185] By creating appropriate configuration files, the mapping operation can be ensured to correctly parse and process the data to be updated when matching the data to be updated with the structure of the data table. The configuration file can include the field names, data types, and other constraints of the data table, enabling the mapping operation to accurately identify and process the data to be updated.

[0186] Different types of data tables may have different structures and fields. The structure of the data table can be determined by the data table type, thereby generating a matching configuration file. This ensures that the configuration file can accurately describe the fields, data types, constraints, and other information of the data table.

[0187] Furthermore, by generating configuration files according to data table types, unified management of configuration information can be achieved. Data tables of the same type can share similar configuration files, facilitating unified maintenance and updates, and avoiding problems of duplication of effort and inconsistency.

[0188] Understandably, since configuration files are created using data table types, they can be adapted to different data tables that need to be updated, thus defining the mapping relationship between fields and paths, and flexibly adapting to different data structures and data sources.

[0189] The data update method proposed in this application involves identifying the type of the data table to be updated based on the data to be updated, thereby obtaining the data table type; and configuring the configuration file according to the data table type. Since the configuration file is created based on the data table type, the configuration file can adapt to different data tables to be updated, thereby defining the mapping relationship between fields and paths, and flexibly adapting to different data structures and data sources.

[0190] Finally, all embodiments of this application can be combined to obtain, as follows: Figure 3 The data update scheme is shown below. Specifically, several data items to be updated are obtained through an update window. When the number of data items to be updated meets a preset update count threshold or the queue time of the update window meets a preset update time threshold, the data items to be updated within the window are traversed to remove duplicates. Then, a path is generated according to the configuration file and added as a prefix to the key of the document to be updated, resulting in the mapped data to be updated (data to be updated with the path prefix). The aggregated document in the Mon database is updated based on the mapped data to be updated, resulting in the updated aggregated document, which is then stored in the Mongo database.

[0191] After obtaining the updated aggregated documents, the primary key to be queried is sent downstream. When the number of records of the primary key to be queried meets a preset query record threshold or the queue time of the restore window meets a preset restore time threshold, the primary keys to be queried in the restore window are traversed to remove duplicates. The updated aggregated documents are then retrieved in batches from the Mongo database based on the primary key to be queried, and the updated aggregated documents are restored to the final documents. Finally, they are written in batches to the ElasticSearch database.

[0192] Furthermore, embodiments of this application also propose a data updating device, the data updating device comprising:

[0193] The data acquisition module is used to acquire the data to be updated.

[0194] The data mapping module is used to perform array path mapping processing on the data to be updated to obtain the mapped data to be updated.

[0195] The primary key extraction module is used to extract the primary key information of the mapped data to be updated to obtain the primary key to be queried.

[0196] The data update module is used to partially update the aggregated document stored in the transit database according to the mapped data to be updated, so as to obtain the updated aggregated document.

[0197] The data restoration module is used to restore the updated aggregated documents into an array based on the primary key to be queried, so as to obtain the final document.

[0198] The principle and implementation process of data updating in this embodiment are explained in the above embodiments and will not be repeated here.

[0199] Furthermore, this application also proposes a terminal device, which includes a memory, a processor, and a data update program stored in the memory and executable on the processor. When the data update program is executed by the processor, it implements the steps of the data update method described above.

[0200] Since this data update program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0201] Furthermore, embodiments of this application also propose a computer-readable storage medium storing a data update program, which, when executed by a processor, implements the steps of the data update method described above.

[0202] Since this data update program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0203] Compared to existing technologies, the data update method, apparatus, terminal device, and storage medium proposed in this application involve: acquiring data to be updated; performing array path mapping on the data to be updated to obtain mapped data to be updated; extracting the primary key information of the mapped data to obtain a primary key to be queried; performing partial updates on aggregated documents stored in a transit database based on the mapped data to be updated to obtain an updated aggregated document; and restoring the updated aggregated document into an array based on the primary key to be queried to obtain a final document. Specifically, by performing array path mapping on the data to be updated to obtain mapped data to be updated without an array structure, and a primary key to be queried corresponding to the mapped data to be updated; then performing partial updates on aggregated documents stored in a transit database based on the mapped data to be updated to obtain an updated aggregated document; finally, restoring the updated aggregated document into an array based on the primary key to be queried to obtain a final document and storing the final document in a final database. The transit database and the final database have different data structures; the transit database stores data in a flattened path format, while the final database stores data in an array format. The above process maps the complex array structure of the data to be updated to a simpler data structure. Because the data format of the mapped data is consistent with the aggregate document and the corresponding aggregate document is stored in advance in the intermediate database, the aggregate document in the intermediate database can be updated directly using a simple local update method, without having to traverse the complex array structure in the final database to retrieve the aggregate document. This reduces the frequency of data interaction with the final database, alleviates the database load, improves the database response speed and concurrency processing capabilities, and increases data update efficiency.

[0204] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0205] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0207] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A data update method, characterized in that, The data update method includes: Get the data to be updated; Perform array path mapping on the data to be updated to obtain the mapped data to be updated; Extract the primary key information of the mapped data to be updated to obtain the primary key to be queried; Based on the mapped data to be updated, the aggregated document stored in the transit database is partially updated to obtain the updated aggregated document. Based on the primary key to be queried, the updated aggregated documents are restored as an array to obtain the final document; The step of performing array path mapping on the data to be updated to obtain the mapped data to be updated includes: Based on a preset configuration file, the data structure of the data to be updated is identified to obtain primary key information, array information, and key-value pairs to be updated; Based on the configuration file, the primary key information and the array information are mapped to an array path to obtain path information; The path information and the key value to be updated are combined to obtain the mapped data to be updated; The step of partially updating the aggregated document stored in the transit database based on the mapped data to be updated, to obtain the updated aggregated document, includes: Based on the path information of the mapped data to be updated, locate the aggregated document to obtain the position to be updated; The key value corresponding to the position to be updated in the aggregated document is updated to the key value to be updated, thus obtaining the updated aggregated document.

2. The data update method according to claim 1, characterized in that, Before the step of performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain the path information, the method further includes: Determine whether the data to be updated belongs to the main table; The step of performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain the path information includes: If the data to be updated does not belong to the main table, then based on the configuration file, the primary key information and the array information are mapped using an array path to obtain the path information.

3. The data update method according to claim 1, characterized in that, The steps for obtaining the data to be updated include: Obtain the data to be updated through the preset update window; The step of performing array path mapping on the data to be updated to obtain the mapped data to be updated includes: Calculate the number of data items to be updated within the update window to obtain the number of data items to be updated. When the number of data items to be updated meets the preset update number threshold or the queue time of the update window meets the preset update time threshold, the data items to be updated are deduplicated to obtain the deduplicated data items to be updated. The deduplicated data to be updated is subjected to path mapping to obtain the mapped data to be updated.

4. The data update method according to claim 1, characterized in that, The step of partially updating the aggregated document stored in the transit database based on the mapped data to be updated, to obtain the updated aggregated document, further includes: Send the primary key to be queried to the restore window; Calculate the number of primary keys to be queried within the restore window to obtain the number of primary keys to be queried; The step of restoring the updated aggregated documents into an array based on the primary key to be queried to obtain the final document includes: When the number of records of the primary key to be queried meets the preset query record number threshold condition or the queue time of the restoration window meets the preset restoration time threshold condition, the primary key to be queried is deduplicated to obtain the deduplicated primary key to be queried. Based on the deduplicated primary key to be queried, the updated aggregated documents are restored as an array to obtain the final document.

5. The data update method according to claim 1, characterized in that, Before the step of performing array path mapping on the data to be updated to obtain the mapped data to be updated, the following steps are included: Based on the data to be updated, the data table of the data to be updated is type-identified to obtain the data table type; The configuration file is obtained by configuring the data table type.

6. A data update device, characterized in that, The data update device includes: The data acquisition module is used to acquire the data to be updated. The data mapping module is used to perform array path mapping processing on the data to be updated to obtain the mapped data to be updated. This includes: identifying the data structure of the data to be updated based on a preset configuration file to obtain primary key information, array information, and key-value pairs to be updated; performing array path mapping processing on the primary key information and the array information based on the configuration file to obtain path information; and combining the path information and the key-value pairs to be updated to obtain the mapped data to be updated. The primary key extraction module is used to extract the primary key information of the mapped data to be updated to obtain the primary key to be queried. The data update module is used to partially update the aggregated document stored in the transit database according to the mapped data to be updated, and obtain the updated aggregated document. The module includes: locating the aggregated document according to the path information of the mapped data to be updated, and obtaining the position to be updated; updating the key value corresponding to the position to be updated of the aggregated document to the key value to be updated, and obtaining the updated aggregated document. The data restoration module is used to restore the updated aggregated documents into an array based on the primary key to be queried, so as to obtain the final document.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a data update program stored in the memory and executable on the processor. When the data update program is executed by the processor, it implements the steps of the data update method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a data update program, which, when executed by a processor, implements the steps of the data update method as described in any one of claims 1-5.

Images