Database upgrading method and device based on data middleware, equipment and medium

Abstract

The application discloses a database upgrading method and device based on data middleware, equipment and medium, and relates to the technical field of database upgrading. The method comprises the following steps: a data middleware layer is constructed between the memory and the database; the current and target version structure definitions are maintained in response to an upgrading command; the data middleware layer intercepts business read-write requests; for a read request, data is obtained according to the target version structure definition, and if the data is missing, the data is obtained according to the current version structure definition and filled with a default value; for a write request, data is written based on the current and target version structure definitions; the batch size is dynamically adjusted based on system load indicators; historical data in the database is migrated to the target version structure in batches; when the historical data migration is completed and the consistency check is passed, the routing is switched to the target version structure definition, and the current version structure definition is cleared after a preset time period. The application keeps the application layer business unaware of the change of the data structure during the database upgrading.

Description

Technical Field

[0001] This application relates to the field of database upgrade technology, specifically to a database upgrade method, apparatus, device, and medium based on data middleware. Background Technology

[0002] In database system operation and maintenance practices, data structure changes (such as adding fields or modifying relationships) are common evolution requirements. Traditional upgrade solutions typically employ a shutdown and table-locking approach: business operations are paused before the upgrade, database tables are locked, and business operations are resumed after the table structure is modified. This approach leads to business interruption, and for production systems that require continuous operation, downtime means direct economic losses.

[0003] To reduce downtime, the industry has introduced online data definition language (DDL) tools, which can reduce database-level table locking time to some extent. However, these tools only address database-level structural changes and are unaware of application-level data structure changes. When the database table structure changes, the application-level memory object model and cache structure still need to be updated synchronously; otherwise, inconsistencies between the memory model and the database structure will occur, leading to serialization or deserialization anomalies.

[0004] During the upgrade process, there was also the challenge of historical data compatibility. After adding a new field, a large number of existing historical records in the database did not contain the value of that field, requiring historical data backfilling. The backfilling operation itself is time-consuming, and during the backfilling period, there is a time window conflict between the newly written data and the historical data that has not yet been backfilled—business read requests may access historical records with missing fields before the backfilling is completed, resulting in incomplete data or business logic errors.

[0005] The common drawback of the existing solutions mentioned above is that they cannot simultaneously keep application-layer services unaware of data structure changes during database structure modifications. Specifically, this manifests as follows: data structure changes and business operations interfere with each other, leading to business interruptions or performance degradation during upgrades; the application-layer memory model and database structure updates are out of sync, causing data anomalies; during the historical data backfilling window, business read requests may retrieve incomplete data, affecting business correctness; online data structure changes and business cascading model updates are disconnected, resulting in application-layer caching being out of sync and historical data backfilling not being completed during the change window, causing services to read inconsistent or missing related data, leading to runtime anomalies. Summary of the Invention

[0006] This application provides a database upgrade method, apparatus, device, and medium based on data middleware, which can solve the technical problem in the prior art that it is impossible to keep the application layer business unaware of data structure changes during database structure changes.

[0007] In a first aspect, embodiments of this application provide a database upgrade method based on data middleware, characterized in that the database upgrade method based on data middleware includes: A data middleware layer is built between memory and the database. In response to upgrade commands, the data middleware layer maintains both the current version structure definition and the target version structure definition. The data middleware layer intercepts business read and write requests and performs route arbitration: For read requests, data is obtained according to the target version structure definition. If the data is missing, it is obtained according to the current version structure definition and the default value is filled in. For write requests, data is written based on both the current version structure definition and the target version structure definition. Start an asynchronous background upgrade task, dynamically adjust the batch size based on system load indicators, and migrate historical data in the database from the current version structure to the target version structure in batches; Once all historical data migration is complete and consistency verification passes, the read / write route will be switched to the target version structure definition, and the current version structure definition will be retained for a preset duration before being cleared.

[0008] In conjunction with the first aspect, in one implementation: both the current version structure definition and the target version structure definition include a structure version identifier, a field dependency graph, field default value derivation rules, cross-entity cascading update relationships for field changes, and the current upgrade stage status; the data middleware layer adopts a two-layer cache layout, with the first layer being an in-heap cache and the second layer being a distributed cache. The current version structure definition and the target version structure definition are preferentially loaded into the in-heap cache. If the in-heap cache misses, it is retrieved from the distributed cache. If the distributed cache misses, it is loaded from the database and backfilled.

[0009] In conjunction with the first aspect, in one implementation, the step of obtaining data according to the target version structure definition for a read request, and obtaining and filling in default values ​​according to the current version structure definition if data is missing, includes: For read requests, retrieve the requested data record from the heap cache according to the target version structure definition, and check if the data has been retrieved from the heap cache: If so, the retrieved data will be used as the response data for the read request; If not, query the distributed cache according to the target version structure definition to check if the distributed cache has retrieved the data: - If so, the retrieved data will be used as the response data for the read request; - If not, query the database according to the target version structure definition to check whether the target field of the requested data record is complete in the database. If yes, retrieve the requested data record according to the target version structure definition as the data to respond to the read request, and backfill the data record to the heap cache and distributed cache in sequence. If not, retrieve the existing field data of the requested data record according to the current version structure definition, and fill the missing fields with default values ​​according to the field default value inference rules.

[0010] In conjunction with the first aspect, in one implementation, the step of filling missing fields with default values ​​according to the field default value derivation rules includes: Check the configuration value of the missing field in the field default value inference rule: If the configuration value is a specific value, then that specific value will be used. If the configuration value is context-derived, the derived value is obtained from the current request context; if the context is unavailable, it is filled with a zero value of the field type. Specifically, when performing default value filling on the missing fields of a read request, the data middleware layer marks the requested data record as pending backfilling and adds the marked record to the backfilling set of the background asynchronous upgrade task.

[0011] In conjunction with the first aspect, in one implementation, the initiation of the background asynchronous upgrade task, dynamically adjusting the batch size based on system load indicators, and migrating historical data in the database from the current version structure to the target version structure in batches, includes: Initiate a background asynchronous upgrade task to obtain the current CPU utilization and current lock wait time percentage of the database server. Based on the CPU utilization and lock wait time percentage, calculate the number of rows to be processed in the current batch, specifically:

[0012] in, This indicates the number of rows processed in the current batch. Represents the amplitude limiting function. Indicates the minimum allowed batch size. This represents the floor function. Indicates the base batch size. This represents the CPU load sensitivity coefficient. This indicates the current CPU utilization rate. This represents the lock wait sensitivity coefficient. This indicates the current percentage of lock waiting time. Indicates the maximum allowed batch size; Adjust the number of processing rows in the current batch based on system load indicators so that the number of processing rows in the current batch decreases as system load increases and increases as system load decreases. The historical data in the database is migrated from the current version structure to the target version structure in batches until all historical data in the database has been migrated.

[0013] In conjunction with the first aspect, in one implementation method: The data middleware layer performs a breadth-first traversal on the field dependency graph, starting from the field involved in this version change. The system performs a forward traversal along the cross-entity cascading update relationship, collecting the fields that need to be backfilled at each level of the dependency chain to form a set of fields to be backfilled. Traverse in reverse along the cross-entity cascading update relationship to collect all business orchestration units that depend on the field to be backfilled, forming an affected orchestration set; The background asynchronous upgrade task determines the range of historical data nodes to be migrated based on the set of fields to be filled, and performs the migration of historical data within the node range in batches according to the batch size.

[0014] In conjunction with the first aspect, in one implementation method: Compare the CPU utilization or service request latency with a preset threshold: When the CPU utilization reaches or exceeds the first preset threshold or the business request delay exceeds the second preset threshold, the background asynchronous upgrade task is paused. When the CPU utilization rate is lower than the third preset threshold and the service request latency is lower than the fourth preset threshold, the background asynchronous upgrade task will resume execution.

[0015] In conjunction with the first aspect, in one implementation method, the consistency check specifically includes: Based on the required fields defined in the target version structure, the filling status of each field value of the migrated record is checked field by field. When all required fields are not empty, the migration record is deemed to have passed the verification. When the cumulative verification pass rate of all migration records reaches the preset condition, the consistency verification is deemed to have passed.

[0016] In conjunction with the first aspect, in one implementation: the data middleware layer maintenance and upgrade phase state machine; The state machine includes at least: a dual-write phase, a dual-read phase, and a switch completion phase; The dual-write phase is that the write request is written simultaneously based on the current version structure definition and the target version structure definition. The dual-read phase involves the read request prioritizing the acquisition of data according to the target version structure definition, and initiating the background asynchronous upgrade task. The switching completion stage involves switching all read and write routes to the target version structure definition and clearing the current version structure definition after the preset time period.

[0017] In conjunction with the first aspect, in one implementation, the double-write phase includes: When writing to the memory cache based on the target version structure definition, a monotonically increasing logical timestamp is appended to each write record, which identifies the time when the write operation occurred; When multiple write versions of the same data record exist in memory due to concurrent write operations, the logical timestamps of each version are compared, the version with the larger timestamp is retained, and the version with the smaller timestamp is overwritten.

[0018] In conjunction with the first aspect, in one implementation, the database upgrade method based on data middleware further includes: If an upgrade anomaly is detected within the preset time period, the data middleware layer will perform a rollback operation based on the current upgrade stage: During the initial stage of a double-write or double-read phase, the target version structure definition in the cache is marked as invalid and the route is switched back to the current version structure definition. In the later stages of the dual-read phase, the broadcast clears the target version structure definition cache and forces a switch back to the current version structure definition route. After the switchover is complete, the rollback is achieved by pausing read and write operations, restoring the current version structure definition from the distributed cache within the freeze window, and then unfreezing.

[0019] In conjunction with the first aspect, in one implementation, switching the read / write route to the target version structure definition includes: Gradual switching based on tenant dimensions, with the data middleware layer maintaining a valid structure version identifier for each tenant independently in the heap cache to control the routing target; After the switch is completed, the current version structure definition is retained in the distributed cache for a preset time. During the retention period, any rollback triggered will directly restore the current version structure definition from the distributed cache. After the preset time expires, the data middleware layer removes the current version structure definition from the distributed cache, while the corresponding old fields in the database are retained and no field deletion operation is performed.

[0020] Secondly, embodiments of this application provide a database upgrade device based on data middleware, the database upgrade device based on data middleware comprising: A building module is used to build a data middleware layer between memory and the database, which responds to upgrade commands and maintains both the current version structure definition and the target version structure definition. The interception module is used by the data middleware layer to intercept business read and write requests and perform route arbitration: For read requests, data is obtained according to the target version structure definition. If the data is missing, it is obtained according to the current version structure definition and the default value is filled. For write requests, data is written based on both the current version structure definition and the target version structure definition. The migration module is used to start background asynchronous upgrade tasks, dynamically adjust the batch size based on system load indicators, and migrate historical data in the database from the current version structure to the target version structure in batches. The switching module is used to switch the read / write route to the target version structure definition after all historical data migration is completed and consistency verification is passed, and then clear it after the current version structure definition is retained for a preset period of time.

[0021] Thirdly, embodiments of this application provide a database upgrade device based on data middleware. The database upgrade device based on data middleware includes a processor, a memory, and a database upgrade program based on data middleware stored in the memory and executable by the processor. When the database upgrade program based on data middleware is executed by the processor, it implements the steps of the database upgrade method based on data middleware as described in any of the above embodiments.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing a database upgrade program based on data middleware, wherein when the database upgrade program based on data middleware is executed by a processor, it implements the steps of the database upgrade method based on data middleware described in any of the above embodiments.

[0023] The beneficial effects of the technical solutions provided in this application include: This application embodiment constructs a data middleware layer that simultaneously maintains the current version structure definition and the target version structure definition in memory. During database upgrades, both business read and write requests are intercepted and arbitrated by the middleware layer, no longer directly dependent on the database table structure. Therefore, there is no need to pause business operations or lock database tables. Business requests continue to be processed normally throughout the entire upgrade cycle. Compared to traditional shutdown and table locking solutions with business interruptions measured in minutes or even hours, this application embodiment can achieve uninterrupted business operations, ensuring the continuous operation of the production system. In this application embodiment, for read requests, the data middleware layer prioritizes obtaining data according to the target version structure definition. When a target field is missing, it automatically falls back to the current version structure definition to obtain existing data and fills the missing field with default values, ensuring that the data returned to the application layer is always structurally complete. For write requests, the data middleware layer performs dual writes simultaneously according to both the current version structure definition and the target version structure definition, ensuring data consistency between the old and new structures. During the window period before historical data is fully backfilled, there are a large number of historical records in the database that are missing target fields. In this embodiment of the application, the default values ​​are obtained and filled according to the current version structure definition, so that the business read request will not obtain incomplete data with missing fields at any time, thus ensuring the consistency and integrity of business data during the backfilling period. Attached Figure Description

[0024] Figure 1 A flowchart illustrating the database upgrade method based on data middleware provided in this application embodiment; Figure 2 A schematic diagram of the functional modules of a database upgrade device based on data middleware provided in an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0026] This application provides a database upgrade method, apparatus, device, and medium based on data middleware, which can solve the technical problem in the prior art that it is impossible to keep the application layer business unaware of data structure changes during database structure changes.

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0028] In a first aspect, embodiments of this application provide a database upgrade method based on data middleware.

[0029] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating the database upgrade method based on data middleware provided in an embodiment of this application. Figure 1 As shown, the database upgrade method based on data middleware specifically includes the following steps: Step S1: Build a data middleware layer between memory and database. In response to upgrade commands, the data middleware layer maintains both the current version structure definition and the target version structure definition.

[0030] The current version structure definition and the target version structure definition coexist in memory. The data middleware layer selects which structure definition to use to parse and assemble the data based on the current upgrade stage.

[0031] In the embodiments of this application, both the current version structure definition and the target version structure definition include a structure version identifier, a field dependency graph, field default value derivation rules, cross-entity cascading update relationships for field changes, and the current upgrade stage status.

[0032] It should be understood that the standard schema envelope of the data middleware layer contains the following fields: schemaVersion (structural version identifier), used to identify the currently effective schema version, using a segmented format of major version number, minor version number, and change sequence number, specifically "major version.minor version.change sequence number", for example, "2.3.7"; fieldGraph (field dependency graph), used to store the dependencies between fields in the form of an adjacency list, used to determine other fields that need to be cascaded updated when a field value changes; defaultRule (field default value inference rule), used to specify how to generate a default value for each field when the field value does not exist in the database, rules include setting to null, filling type, etc. Zero values, or values ​​derived from the request context, with the key being the field name and the value representing the default value inference rule. Here, NULL represents an empty value, TYPE_ZERO represents a zero value (INT→0, STRING→"", BOOL→false), and CONTEXT_DERIVE represents a value derived from the context (e.g., when a child node lacks a certain attribute, the attribute value is searched upwards along the parent node path and inherited). cascadeEdges (cross-entity cascading update relationships for field changes) describe the other data entities and their fields that need to be updated synchronously when a field changes, as well as the propagation rules followed by the updates. Each edge has the format {srcField, The `dstEntity`, `dstField`, and `propagateRule` classes describe the cross-entity cascading update relationship triggered by field changes. The `stage` (current upgrade stage state) identifies the current upgrade stage of the data middleware layer, including states such as Idle, Prepared, Dual Write, Dual Read, Cutover, Cleanup, and Rollback, which control the routing arbitration logic.

[0033] In this embodiment, the data middleware layer adopts a two-layer cache layout: the first layer is the in-heap cache and the second layer is the distributed cache. The current version structure definition and the target version structure definition are loaded into the in-heap cache first. If the in-heap cache is not hit, it is retrieved from the distributed cache. If the distributed cache is not hit, it is loaded from the database and backfilled.

[0034] In a specific implementation, the first layer is a local cache (L1) deployed in the Java Virtual Machine heap memory, offering low access latency; the second layer is a distributed cache (L2) used to store the full structure definition, supporting cross-node sharing. When a data middleware layer node needs to read a structure definition, it first retrieves it from the local heap cache; if the local cache misses the definition, it requests it from the distributed cache. If the distributed cache still lacks the required version, it loads it from the database and backfills it into the L2 and L1 caches. The fieldGraph, defaultRule, and cascadeEdges in the structure definition are serialized and hashed to form a version fingerprint. Each data middleware layer node compares the version fingerprint in its local cache with the fingerprint recorded in the registry. If an inconsistency is found, the structure definition refresh process is immediately triggered to ensure that all nodes use the same and up-to-date structure definition version.

[0035] This application embodiment constructs a data middleware layer that simultaneously maintains the current version structure definition and the target version structure definition in memory. During database upgrades, both business read and write requests are intercepted and arbitrated by the middleware layer, no longer directly dependent on the database table structure. Therefore, there is no need to suspend business or lock the database table. Business requests are continuously processed normally throughout the entire upgrade cycle. Compared with traditional shutdown and table locking schemes, which result in business interruptions measured in minutes or even hours, this application embodiment can achieve uninterrupted business operations, ensuring the continuous operation of the production system.

[0036] Step S2: The data middleware layer intercepts business read and write requests and performs route arbitration: For read requests, data is obtained according to the target version structure definition. If the data is missing, it is obtained according to the current version structure definition and the default value is filled in. For write requests, data is written based on both the current version structure definition and the target version structure definition.

[0037] In this embodiment of the application, step S2, for a read request, involves obtaining data according to the target version structure definition; if data is missing, it is obtained according to the current version structure definition and a default value is filled in. Specifically, this includes the following steps: Step S21: For the read request, retrieve the requested data record from the heap cache according to the target version structure definition, check whether the data is retrieved from the heap cache. If yes, proceed to step S22; otherwise, proceed to step S23. Step S22: Use the retrieved data as the response data for the read request; Step S23: Query the distributed cache according to the target version structure definition, and check whether the distributed cache has retrieved the data. If yes, go to step S22; otherwise, go to step S24. Step S24: Query the database according to the target version structure definition, and check whether the target field of the requested data record is complete in the database. If yes, proceed to step S25; otherwise, proceed to step S26. Step S25: Obtain the requested data record as the response data for the read request according to the target version structure definition, and then backfill the data record into the heap cache and the distributed cache in sequence; Step S26: Obtain the existing field data of the requested data record according to the current version structure definition, and fill the missing fields with default values ​​according to the field default value inference rules.

[0038] It should be understood that when a business system initiates a read request, the data middleware layer does not directly access the database, but attempts to retrieve data level by level according to a four-level priority order. The query target at each level is the data format defined by the target version structure definition.

[0039] Specifically, at the first level, the data middleware layer queries the requested data record from the local heap cache. The heap cache uses the version identifier defined by the target version structure as an index to store data organized according to the new structure. If the complete target version structure data is found in the heap cache, it is returned directly, resulting in the shortest response path. At the second level, when the heap cache misses, the data middleware layer queries the distributed cache. The distributed cache maintains the full target version structure data and can be shared across nodes. If the requested data exists in the distributed cache, it is returned to the requester and simultaneously populated back into the heap cache to improve the hit rate of subsequent identical requests. At the third level, when both the heap cache and the distributed cache miss, the data middleware layer sends the query to the database. The target of this query is the table in the database where the new field has been added according to the target version structure. If the record exists in the database and the target field is populated, the data is returned and then populated back into the heap cache and the distributed cache in sequence. In the fourth level, when the target field is also missing in the database, meaning the record is an unfilled historical record with an empty target field value, the data middleware layer no longer retrieves data based on the target version structure definition. Instead, it switches to the current version structure definition and retrieves the existing field data for the record from the database. For fields that exist in the target version structure but are missing in the database, the data middleware layer generates default values ​​for each missing field based on the field default value derivation rules in the structure definition. The completed data is then returned to the requester. At the same time, the data middleware layer marks the data record as pending filling and adds it to the backfill queue of the background asynchronous upgrade task so that the background task can complete the field filling of the record later.

[0040] This application's embodiments employ a multi-level cache fallback mechanism, prioritizing data retrieval from local and distributed caches for read requests. This reduces direct database query pressure. Simultaneously, a cache backfilling strategy gradually improves the hit rate of each cache level, lowering the average response latency of read requests. During the phase before historical data backfilling is complete, the system automatically switches to the current version's structure definition to retrieve existing data and fills in missing fields according to default value derivation rules. This ensures that business read requests always receive structurally complete data, avoiding null value anomalies or business logic errors caused by missing target fields in historical records.

[0041] In this embodiment of the application, step S26, which involves filling missing fields with default values ​​according to the field default value derivation rules, includes: Check the configuration value of the missing field in the field default value inference rule: If the configuration value is a specific value, then that specific value will be used. If the configuration value is context-derived, the derived value is obtained from the current request context; if the context is unavailable, it is filled with a zero value of the field type.

[0042] Specifically, when performing default value filling on missing fields of a read request, the data middleware layer marks the requested data record as pending backfilling and adds the marked record to the backfilling set of the background asynchronous upgrade task.

[0043] Specifically, when a new structure field is not found in the history during a read operation, a default value is derived according to the following three-level rules. Level 1: Check the configuration value of the missing field in the field default value derivation rules. If the configuration value is a specific value (including an explicitly specified null value or a zero value indicating the use of the field's data type), then directly fill the missing field with that specific value as the default value. If it is configured as CONTEXT_DERIVE, proceed to Level 2. Level 2: Derive the default value from the current request context (such as BOM parent node data, work order process data). For example, when a field is a priority field and a parent node exists, the priority value of the parent node can be reduced by one as the derived value. Level 3: If the required source data does not exist in the current request context, such as the absence of a parent node or the lack of relevant attributes in the context, derivation cannot be completed. The data middleware layer fills the field with the zero value of its data type. The zero value for integer fields is 0, for boolean fields it is false, and for string fields it is an empty string.

[0044] In this embodiment of the application, for read requests, the data middleware layer first obtains data according to the target version structure definition. When the target field is missing, it automatically falls back to the current version structure definition to obtain existing data and fills the missing field with default values ​​to ensure that the data returned to the application layer is always structurally complete.

[0045] Furthermore, during the default value population process described above, the data middleware layer synchronously marks the data record as pending backfilling and adds the record's identifier to the backfilling set of the background asynchronous upgrade task. This marking and enqueueing operation ensures that the record can be prioritized in subsequent background migration tasks, writing the missing field values ​​into the target version structure of the database.

[0046] In step S2, for write requests, data is written simultaneously based on both the current version structure definition and the target version structure definition. It should be understood that for business write requests, the data middleware layer does not directly send them to the database, but instead performs a dual-write operation. When writing to the memory cache according to the target version structure definition, a monotonically increasing logical timestamp is appended to each write record. This timestamp identifies the time the write operation occurred. When multiple write versions of the same data record exist in memory due to concurrent write operations, the data middleware layer compares the logical timestamps of each version, retaining the version with the larger timestamp and overwriting the version with the smaller timestamp, thus achieving priority visibility of later-written data. When writing to the old database structure according to the current version structure definition, the data middleware layer uses the field set in the current version structure definition as a basis to reassemble the same write data. New fields that exist in the target version structure definition but not in the current version structure definition are excluded and not written to the old database table. The data format written to the database is completely consistent with that before the upgrade, ensuring compatibility on the old database structure side. The data values ​​of newly added fields are filled in by subsequent asynchronous backfill tasks in the background, rather than being directly written to the database by this write request.

[0047] The dual-write mechanism ensures synchronized data updates under both the old and new structure definitions. Each write operation simultaneously generates two sets of data that conform to both the current version's structure definition and the target version's structure definition, avoiding data inconsistencies caused by single-structure writes during the upgrade process.

[0048] Step S3: Start the background asynchronous upgrade task, dynamically adjust the batch size based on system load indicators, and migrate historical data from the current version structure to the target version structure in batches.

[0049] In this embodiment of the application, step S3 specifically includes the following steps: Step S31: Start the background asynchronous upgrade task, obtain the current CPU utilization and current lock wait time percentage of the database server, and calculate the number of rows to be processed in the current batch based on the CPU utilization and lock wait time percentage. Specifically:

[0050] in, This indicates the number of rows processed in the current batch. Represents the amplitude limiting function. Indicates the minimum allowed batch size. This represents the floor function. Indicates the base batch size. This represents the CPU load sensitivity coefficient. This indicates the current CPU utilization rate. This represents the lock wait sensitivity coefficient. This indicates the current percentage of lock waiting time. Indicates the maximum allowed batch size.

[0051] Specifically, CPU utilization reflects the computing resource usage of the database server, ranging from 0 to 1. Lock wait time percentage reflects the current level of database lock contention, also ranging from 0 to 1. According to the above calculation formulas, when CPU utilization or lock wait time percentage increases, the calculated result decreases, and the batch size shrinks; conversely, when CPU utilization or lock wait time percentage decreases, the calculated result increases, and the batch size expands. This prevents batches from being too small, leading to low migration efficiency, or from being too large, causing instantaneous pressure on the database. In practical implementation, the typical baseline batch size is 1000 rows, the typical minimum allowed batch size is 100 rows, and the typical maximum allowed batch size is 5000 rows. The CPU load sensitivity coefficient is generally 0.8, and the lock wait time sensitivity coefficient is generally 1.2.

[0052] Step S32: Adjust the number of processing rows in the current batch based on the system load index so that the number of processing rows in the current batch decreases as the system load increases and increases as the system load decreases.

[0053] Specifically, based on the number of rows processed in the current batch calculated in step S31, the processing scale of this round of migration task is adjusted, thereby making the resource consumption of the migration task inversely related to the current database load, ensuring that the migration task always runs within the remaining processing capacity of the database.

[0054] Step S33: Migrate historical data from the current version structure to the target version structure in batches until all historical data in the database has been migrated.

[0055] Specifically, data migration operations are performed in batches. Each batch uses the primary key range of the database table as the shard unit, processing a left-closed, right-open interval. After each batch is completed, the currently processed primary key position is persisted to a checkpoint record in the metadata table. When the task restarts due to middleware restart or abnormal interruption, the previously completed primary key position is read from the checkpoint record, and subsequent batches continue execution from that breakpoint, avoiding duplicate processing of already filled records. The above process of batching, recording checkpoints, and resuming from breakpoints is executed cyclically until all historical data in the database has been migrated from the current version structure to the target version structure. The batching strategy based on primary key ranges and the checkpoint breakpoint resumption mechanism ensure the recoverability of the migration task under any interruption scenario, avoiding full rescanning and duplicate processing caused by restarts or abnormalities.

[0056] In the embodiments of this application: The data middleware layer performs a breadth-first traversal on the field dependency graph, starting from the field involved in this version change. The system performs a forward traversal along the cross-entity cascading update relationship, collecting the fields that need to be backfilled at each level of the dependency chain to form a set of fields to be backfilled. Traverse in reverse along the cross-entity cascading update relationship to collect all business orchestration units that depend on the field to be backfilled, forming an affected orchestration set; The background asynchronous upgrade task determines the range of historical data nodes to be migrated based on the set of fields to be filled, and performs the migration of historical data within the node range in batches according to the batch size.

[0057] It should be understood that the data middleware layer loads the affected orchestration set into the runtime routing strategy. During the execution of the background asynchronous upgrade task, for business read and write requests that hit the affected orchestration set, the synchronous cascading completion of the data records involved in the request is forcibly triggered. After all the fields to be filled in the record are filled, the response is returned or the downstream business logic continues to be executed. For requests that do not hit the affected orchestration set, they are handled normally according to the regular routing strategy.

[0058] In the embodiments of this application: Compare CPU utilization or business request latency with preset thresholds: When CPU utilization exceeds the first preset threshold or business request latency exceeds the second preset threshold, pause the background asynchronous upgrade task. When CPU utilization is below the third preset threshold and business request latency is below the fourth preset threshold, resume the execution of the background asynchronous upgrade task.

[0059] Specifically, the first preset threshold can be 0.7 ms, the second preset threshold can be 200 ms, the third preset threshold can be 0.5 ms, and the fourth preset threshold can be 100 ms. By setting an automatic pause mechanism during peak business periods, the database resources occupied by the background migration task can be released immediately when increased business pressure is detected, ensuring that the response latency of business requests is not continuously affected by the migration task, and realizing that the resource scheduling of business requests always has a higher priority than that of the migration task.

[0060] Step S4: After all historical data migration is completed and consistency verification is passed, switch the read / write route to the target version structure definition, and clear it after retaining the current version structure definition for a preset period of time.

[0061] In this embodiment of the application, the consistency verification in step S4 is specifically as follows: based on the required fields defined in the target version structure, the filling status of each field value of the migrated record is checked field by field. When all required fields are not empty, the migration record is determined to have passed the verification. When the cumulative verification pass rate of all migration records reaches the preset condition, the consistency verification is determined to have passed.

[0062] Specifically, after all batches of asynchronous background migration tasks are completed, the data middleware layer does not immediately switch read / write routes. Instead, it first performs structural integrity checks on all migrated data records. The preset condition is that the pass rate of the full migration record check is 100%. After the consistency check passes, the data middleware layer performs a route switching operation. At the same time, the current version structure definition is not immediately deleted but continues to be maintained for a preset period. After the preset period, the middleware layer removes the current version structure definition from the heap cache and distributed cache, retaining only the target version structure definition as the sole structural basis for subsequent business operations. In this embodiment, the upgrade is only allowed to proceed to the next stage when the full record check pass rate reaches 100%, avoiding business logic errors caused by switching routes prematurely when some data has not been migrated. In the embodiments of this application: the data middleware layer maintenance and upgrade phase state machine; the state machine includes at least: a dual-write phase, a dual-read phase, and a switch completion phase; the dual-write phase is when write requests are simultaneously written based on the current version structure definition and the target version structure definition; the dual-read phase is when read requests preferentially obtain data according to the target version structure definition, and start a background asynchronous upgrade task; the switch completion phase is when all read and write routes are switched to the target version structure definition, and the current version structure definition is cleared after a preset time.

[0063] It should be understood that the data middleware layer maintains an upgrade phase state machine, which contains seven states: idle phase, preparation phase, dual write phase, dual read phase, switchover completion phase, cleanup phase, and rollback phase. The preparation phase, dual write phase, dual read phase, switchover completion phase, and cleanup phase are sequentially progressive states, while the rollback phase is a non-sequential state that can be triggered from any of the aforementioned states.

[0064] During the preparation phase, the data middleware layer loads the target version structure definition into memory and performs pre-validation of the structure definition content. During the dual-write phase, all business write requests simultaneously perform dual-write operations based on both the current and target version structure definitions. Read paths are not switched initially to reduce risks introduced during the initial upgrade phase. During the dual-read phase, the routing strategy for read requests is switched from prioritizing data retrieval based on the current version structure definition to prioritizing data retrieval based on the target version structure definition. Write requests continue to perform dual-write operations, and the background asynchronous upgrade task is started in this state, beginning to migrate historical data from the current version structure to the target version structure in batches. During the switchover completion phase, all read and write routes are switched to the target version structure definition, the current version structure stops receiving new write operations, and the main upgrade work is completed. During the cleanup phase, the current version structure definition is not immediately deleted but is retained for a preset period. After the retention period expires, the middleware layer removes the current version structure definition from the heap cache and distributed cache. Old fields in the database are retained but are no longer accessed by the business through the data middleware layer. During the rollback phase, if an upgrade anomaly is detected in any phase, the state machine jumps to the rollback state, the read and write routes immediately switch back to the current version structure definition, and the retained old structure definition is used to shield the impact of the target version structure on the business. Subsequently, tiered rollback operations are performed according to the severity of the anomaly.

[0065] Pre-validation may include: field name validity check, type compatibility matrix check, default value and constraint conflict check, index and foreign key conflict check, and cascading node influence range scan.

[0066] The field name validity check specifically verifies whether the names of newly added fields in the target version structure definition conform to the database identifier naming conventions. For example, field names are only allowed to contain letters, numbers, and underscores; the first character of the field name cannot be a number; and the total length of the field name does not exceed the preset limit (e.g., 64 characters).

[0067] The type compatibility matrix check specifically checks whether there are implicit conversion conflicts between the new field type and the existing field type. For example, when a new field is used to establish a foreign key relationship, it checks whether the data type of the new field is consistent with the data type of the foreign key target field in the referenced target table.

[0068] The default value and constraint conflict check specifically verifies whether the default value configured for the newly added field in the field default value derivation rule satisfies the integrity constraints declared on the field. For example, whether the default value violates the non-null constraint, whether it meets the value conditions of the check constraint, and whether it duplicates the field value of an existing record under the uniqueness constraint.

[0069] The index and foreign key conflict check specifically checks whether the new field name is the same as the existing index name, and whether the foreign key relationship references a parent table or parent field that does not yet exist.

[0070] The cascading node impact range scan specifically involves estimating the impact range of cascading edges declared in cross-entity cascading update relationships for newly added fields. For example, starting with the newly added field, a breadth-first traversal is performed on the field dependency graph, collecting affected entity nodes layer by layer along the cascading edges. The scan results are used to count the total number of affected cascading nodes, such as the number of affected BOM nodes or process route nodes. If the total number of cascading nodes exceeds a preset threshold, the upgrade request is marked as a high-impact change, prompting operations personnel for confirmation before proceeding, preventing unexpected large-scale cascading propagation triggered by a single field addition. The preset threshold can be 100,000.

[0071] If any check fails, the pre-verification immediately terminates, the upgrade request is rejected, and the state machine remains idle, not entering a double-write state. This ensures that risky changes are intercepted at minimal cost in the early stages of the upgrade operation, preventing problems from propagating to subsequent states.

[0072] In this embodiment of the application, performing a graded rollback operation based on the severity of the anomaly specifically includes: If an upgrade anomaly is detected within a preset time period, the data middleware layer will perform a rollback operation based on the current upgrade stage: During the initial stage of a double-write or double-read phase, the target version structure definition in the cache is marked as invalid and the route is switched back to the current version structure definition. In the later stages of the dual-read phase, the broadcast clears the target version structure definition cache and forces a switch back to the current version structure definition route. After the switchover is complete, the rollback is achieved by pausing read and write operations, restoring the current version structure definition from the distributed cache within the freeze window, and then unfreezing.

[0073] It should be understood that during the initial stages of the dual-write or dual-read phase, the target version structure definition has been loaded into memory, but the business read path has not yet switched to the new structure. The impact of any anomalies is limited, therefore a soft rollback is performed. Specifically, the data middleware layer first switches the upgrade phase state from the current state back to the idle state. Then, it marks the target version structure definition already loaded in the heap cache as invalid, preventing it from participating in subsequent read / write route arbitration. The copy of the target version structure definition retained in the distributed cache is not deleted but used as a backup for reuse during subsequent upgrade attempts. Read / write routes are synchronously switched back to the current version structure definition path. For data already written to the new memory structure according to the target version structure definition during the dual-write phase, no rollback operation is performed because the same data has already been synchronously written to the old database structure according to the current version structure definition, and there is no risk of business data loss. The soft rollback time mainly consists of two operations: cache invalidation and route switching, typically taking 1 to 2 seconds.

[0074] In the later stages of the dual-read phase, the read path has switched to the new structure. Anomalies may directly impact business requests, therefore a hard rollback is performed. Specifically, the data middleware layer switches the upgrade phase to the rollback phase. The data middleware layer broadcasts a cache clearing command to all nodes in the cluster, and each node clears all target version structure definitions stored in its local heap cache. Read and write routes are forcibly switched back to the current version structure definition path. The background asynchronous upgrade task immediately stops executing, and the currently processed primary key position checkpoints are retained in the metadata table. After the cause of the anomaly is located and fixed, the migration task can be restarted from the breakpoint without reprocessing the already backfilled records. During this phase, the newly added fields of the target version structure already exist in the database, but because the data middleware layer has switched back to the old structure, business read and write requests no longer access these new fields. The new fields in the database are retained and not deleted to avoid triggering table locking at the data definition language level again due to field deletion. The hard rollback time mainly consists of broadcasting cache clearing and route switching, typically taking 2 to 5 seconds.

[0075] After the switchover is complete, all read and write routes have been switched to the target version's structure definition. The old structure has stopped receiving writes and cannot be rolled back through a simple route switch; therefore, a freeze operation is performed. Specifically, the data middleware layer suspends the processing of all business read and write requests and returns a system maintenance response to the requester. The freeze window has a timeout limit of 30 seconds. Within the freeze window, the data middleware layer reads a complete copy of the previously retained current version's structure definition from the distributed cache, restores it to the heap cache of each node, and reloads it as the effective structure definition. After restoration, the freeze is lifted, and business requests resume processing. At this point, read and write requests are executed according to the current version's structure definition. The freeze operation causes a short interruption to business requests; therefore, its triggering must be confirmed by operations personnel before execution.

[0076] The current version's structure definition retains the system for at least a configurable preset duration before cleanup. This preset duration can be 24 hours, during which hard rollback remains available. If potential issues are discovered within the retention period after the upgrade, a hard rollback can still be performed to restore the system to its pre-upgrade state. Each rollback operation is logged, including the rollback level, triggering reason, the upgrade stage at the time of triggering, rollback duration, and the number of records written to the new structure during the dual-write period. The number of records written to the new structure during the dual-write period is used to assess the rollback's impact and pinpoint the root cause of any anomalies. Rollback levels can include soft rollback, hard rollback, and freeze. Triggering reasons can include verification failure, timeout, and manual triggering.

[0077] The rollback mechanism matches rollback strategies of different intensities based on the escalation stage and scope of impact when the anomaly occurs. It uses a soft rollback with lower costs when the anomaly is controllable, a more thorough hard rollback when the impact of the anomaly expands, and provides freezing as a final recovery measure in extreme cases. This achieves a match between the rollback cost and the severity of the anomaly, avoiding over-response to minor anomalies.

[0078] Furthermore, during the process of switching the read / write route to the target version structure definition: The transition is gradual and phased down to the tenant level. The data middleware layer maintains a valid structure version identifier for each tenant independently in the heap cache to control the routing target. After the transition is completed, the current version structure definition is retained in the distributed cache for a preset period of time. During the retention period, any rollback triggered will directly restore the current version structure definition from the distributed cache. After the preset period expires, the data middleware layer deletes the current version structure definition from the distributed cache. The corresponding old fields in the database are retained and no field deletion operation is performed.

[0079] It should be understood that by using tenant-level canary rollouts, the risks introduced by the new version structure are controlled to a small extent, avoiding global failures caused by a full rollout. After the rollout is complete, the old version structure definition is retained in a distributed cache, which can be directly restored during rollback without reloading, shortening rollback time. After the retention period expires, only the cache is cleared without deleting old fields from the database, avoiding triggering database-level table locking operations. Afterward, the state machine enters the cleanup state, marking the completion of the entire upgrade process.

[0080] This application embodiment constructs a data middleware layer that simultaneously maintains the current version structure definition and the target version structure definition in memory. During database upgrades, both business read and write requests are intercepted and arbitrated by the middleware layer, no longer directly dependent on the database table structure. Therefore, there is no need to pause business operations or lock database tables. Business requests continue to be processed normally throughout the entire upgrade cycle. Compared to traditional shutdown and table locking solutions with business interruptions measured in minutes or even hours, this application embodiment can achieve uninterrupted business operations, ensuring the continuous operation of the production system. In this application embodiment, for read requests, the data middleware layer prioritizes obtaining data according to the target version structure definition. When a target field is missing, it automatically falls back to the current version structure definition to obtain existing data and fills the missing field with default values, ensuring that the data returned to the application layer is always structurally complete. For write requests, the data middleware layer performs dual writes simultaneously according to both the current version structure definition and the target version structure definition, ensuring data consistency between the old and new structures. During the window period before historical data is fully backfilled, there are a large number of historical records in the database that are missing target fields. In this embodiment of the application, the default values ​​are obtained and filled according to the current version structure definition, so that the business read request will not obtain incomplete data with missing fields at any time, thus ensuring the consistency and integrity of business data during the backfilling period.

[0081] In a specific embodiment, the upgrade method is illustrated by taking the addition of a priority field to the work order table in the mold manufacturing platform as an example.

[0082] First, operations personnel submit an upgrade request through the management interface, requesting the addition of a `priority` field to the work order table. Upon receiving the upgrade request, the data middleware layer sets the upgrade phase status to "ready" and loads the target version structure definition into a separate memory area. This target version structure definition adds complete descriptive information for the `priority` field to the existing current version structure definition, including the field name, data type, default value configuration, and value range constraints. After loading, the data middleware layer performs pre-validation. If all pre-validations pass, the upgrade request is accepted, and the state machine remains in the "ready" state, awaiting the formal switchover instruction. If any validation fails, the upgrade request is rejected, the state machine remains in the "idle" state, and the target version structure definition is cleared.

[0083] After receiving the formal switchover instruction, the operations and maintenance personnel initiate a dual-version coexistence mode in the data middleware layer: simultaneously maintaining two sets of structure definitions in memory—the current version (excluding the priority field) and the target version (including the priority field). New write operations simultaneously write data based on both sets of structure definitions: when assembling according to the target version structure definition, the priority field is filled with a default value of 0 and a logical timestamp is appended before being written to the memory cache; when assembling according to the current version structure definition, the priority field is not written, and the data is written to the old database structure. Read operations still prioritize retrieving data from the current version structure definition during this phase, ensuring the stability of the read / write path in the initial switchover period. Specifically, the upgrade process proceeds sequentially through five stages of a state machine: Phase 1 (Preparation, approximately 5 seconds): The target version structure definition is loaded into the heap cache and distributed cache. All pre-validations pass, and the state machine remains in the ready state. During this phase, business read and write operations continue to use the current version structure definition, while the target version structure definition remains in memory.

[0084] Phase Two (Dual Write, continuing until backfill progress reaches approximately 10%): The state machine switches to dual write mode. Write routing enables dual write: Write operations first assemble data according to the target version structure definition (filling the priority field with a default value of 0 and appending a logical timestamp), and write it to the storage area corresponding to the target version structure in the heap cache; simultaneously, data is assembled according to the current version structure definition (excluding the priority field), and asynchronously written to the old database structure. Read routing continues to use the current version structure definition to ensure stable read and write paths during the initial switchover. The pseudocode for dual write is: def write_with_dual(record, new_schema, old_schema): new_record = apply_schema(record, new_schema, fill_defaults=True) new_record.logicalTs = monotonic_clock() l1_cache.put(new_schema.version, new_record.id, new_record) async_write_db(record, old_schema) # Without new fields Phase 3 (Dual Read, continuing until backfilling is complete): The state machine switches to dual read state. Read routing switches to a four-level fallback strategy: prioritize retrieving data from the heap cache according to the target version structure definition; if a miss occurs, query the distributed cache and database sequentially according to the target version structure definition; if the target field is missing, retrieve existing field data according to the current version structure definition and fill the missing field with default values. Background backfilling tasks start synchronously, backfilling historical data in batches using an adaptive rate-limiting method.

[0085] Phase Four (Switch, approximately 1 second): After all historical data has been backfilled and consistency verification has passed, the state machine switches to the switch state. Write routes cease dual-write operations and only write according to the target version structure definition; read routes only retrieve data from the target version structure definition, and the database fields corresponding to the current version structure definition are no longer accessed through the middleware.

[0086] Phase 5 (Cleanup, retained for 24 hours): The current version structure definition is cleared from the heap cache and distributed cache, and the upgrade lifecycle is completely over.

[0087] The background asynchronous upgrade task performs database table structure changes with low priority. First, a `priority` field (default value 0) is added to the work order table, and then historical data is backfilled in batches. The backfilling task uses an adaptive rate-limiting algorithm: it dynamically calculates the number of rows processed in the current batch based on the database server's CPU utilization and lock wait time percentage, reducing the batch size as the load increases. The backfilling task uses the primary key range as the sharding unit. After each batch is processed, the currently processed primary key position is persisted to a checkpoint record. When the task is interrupted and restarted, the position is read from the checkpoint record and execution continues. When the CPU utilization exceeds the first preset threshold or the business request latency exceeds the second preset threshold, the backfilling task automatically pauses and waits for a preset time before re-detecting; execution resumes when the CPU utilization is below the third preset threshold and the business request latency is below the fourth preset threshold. The pseudocode for the backfilling task batch control algorithm is as follows: def backfill_task(table, field, default_val, meta_table): checkpoint = meta_table.get_checkpoint(table, field) id_start = checkpoint.last_id if checkpoint else 0 id_max = db.query("SELECT MAX(id) FROM " + table) while id_start <id_max: cpu_load = get_db_cpu_load() lock_wait = get_lock_wait_ratio() batch = clamp(B_min=100, round(B_0=1000 * (1 - 0.8*cpu_load - 1.2*lock_wait)), B_max=5000) if cpu_load>0.7 or get_p95_latency()>200: sleep(30) continue db.execute( "UPDATE " + table + " SET " + field + " = " + str(default_val) + " WHERE id>= " + str(id_start) + " AND id<" + str(id_start + batch) + " AND " + field + " IS NULL", priority="LOW" ) id_start += batch meta_table.save_checkpoint(table, field, id_start) In this embodiment, the work order table contains approximately one million records, with a baseline batch size of 1000 rows and a CPU load sensitivity factor of 0.8. When the baseline CPU utilization is 0.4, the batch size is approximately 680 rows. During peak business periods, when CPU utilization reaches or exceeds 0.7, the backfilling task is automatically paused. After the peak ends, when CPU utilization drops to 0.3, the batch size recovers to approximately 760 rows. Backfilling all historical data takes approximately 20 to 25 minutes. During this period, the middleware layer continuously masks structural differences, and business requests are unaware of them.

[0088] After the background asynchronous upgrade task completes the backfilling of all historical data, it checks whether the values ​​of each field in the migrated records have been filled, based on the required fields defined in the target version structure. When all required fields are not empty and the cumulative pass rate reaches the preset condition, the consistency check is considered successful. The middleware layer switches both read and write routes to the target version structure definition, while the current version structure definition continues to be retained for a preset period as a rollback backup.

[0089] Furthermore, the switchover can employ a canary rollout strategy: first, 10% of tenant traffic is switched to the target version's structure definition route; after observing for approximately 5 minutes without any anomalies, this is expanded to 50%, eventually achieving 100% traffic switching. Canary rollout is implemented by maintaining a valid structure version identifier for each tenant in the heap cache, without affecting the routing strategies of other tenants. After the switchover is complete, the current version's structure definition continues to be retained in the distributed cache for T_keep=24 hours (which can be extended to 72 hours depending on the business service level agreement). If a hard rollback is triggered during this retention period, the current version's structure definition can be directly restored from the distributed cache without reloading. After the retention period expires, the current version's structure definition is deleted from the distributed cache; old fields in the database are retained but not deleted to avoid triggering secondary data definition language-level table locking. The state machine switches to the cleanup state, and the upgrade lifecycle is complete.

[0090] If an anomaly occurs during the upgrade process, the data middleware layer will perform a graded rollback operation based on the current upgrade stage and the type of anomaly. The specific decisions are as follows: (1) Pre-verification failure (preparation stage): Stop early, do not trigger rollback, and keep the state machine in an idle state. Operation: Reject the upgrade request and return detailed reasons for the verification failure. (2) Memory new structure definition loading timeout (preparation stage): Perform a soft rollback. Operation: Clear the target version structure definition in the heap cache, restore the state machine to an idle state, and record the rollback log. (3) Dual-write database anomaly (dual-write stage, such as database connection interruption): Perform a soft rollback. Operation: Stop dual-write, clear the target version structure definition in the heap cache, switch the read / write route back to the current version structure definition path, and restore the state machine to an idle state. (4) Backfill task consecutive timeout (dual-read stage, timeout number exceeds 10): Perform a hard rollback. Operation: Broadcast to all nodes to clear the target version structure definition in the heap cache, the background backfill task stops immediately and retains the checkpoint, the read / write route is forcibly switched back to the current version structure definition path, and the state machine switches to the rollback state. (5) Business layer error (during the switching phase or dual-read phase, such as the MES work order parsing anomaly rate exceeding 1%): Perform a hard rollback. The operation is the same as item (4) above. (6) After the switch is completed, a serious data consistency anomaly is found: first trigger a freeze, and then perform a hard rollback after manual confirmation by the operation and maintenance personnel. Operation: Suspend the processing of business read and write requests for up to 30 seconds, restore the current version structure definition from the distributed cache within the freeze window, and unfreeze after restoration, and business requests will be processed normally.

[0091] The rollback operation is completed in memory and does not involve reverting database table structure changes. Newly added priority fields in the database are retained, as business reads and writes are routed through middleware and have switched back to the current version's structure definition; the business does not access the new fields. Each rollback operation is logged in an audit log with the following format: {"event": "ROLLBACK", "rollback_type": "SOFT|HARD|FREEZE", "trigger_reason": "...", "stage_at_trigger": "...", "rollback_duration_ms": ..., "records_written_to_new": ...,"operator": "auto|manual", "timestamp": "..."}.

[0092] Taking the newly added "cavity_count" field in the injection mold manufacturing platform as an example, this paper illustrates the specific implementation of the present invention in the scenario of industrial model cascade structure change.

[0093] Scenario Description: A certain injection mold manufacturing platform maintains approximately 80,000 BOM nodes (up to 6 levels of nesting) and approximately 5,000 process route nodes. Adding a new `cavity_count` field requires simultaneous modification of the BOM nodes containing mold sub-processes at the BOM level (updating the `bom_child_count` field for the number of child items), and simultaneous adjustment of the number of parallel operations (`parallel_op_count`) and equipment capacity allocation parameters (`capacity_alloc`) in the process route. This cascading effect affects approximately 28,000 nodes.

[0094] Cascading impact calculation: During the pre-validation phase, the middleware layer performs a breadth-first traversal on the field dependency graph, starting with the `cavity_count` field. It then traverses forward along the cross-entity cascading update relationships (`cavity_count` → `bom_child_count` → `parallel_op_count` → `capacity_alloc`), collecting a set of fields to be backfilled: `pendingFields` = {`cavity_count`, `bom_child_count`, `parallel_op_count`, `capacity_alloc`}. A reverse traversal collects a set of affected orchestrations: `affectedOrchestrations` = {`MES work order parsing orchestration`, `APS scheduling orchestration`}. The statistics show approximately 24,000 affected BOM nodes and approximately 4,000 affected process route nodes, totaling approximately 28,000 nodes requiring cascading backfill.

[0095] Cascading Marking and Dual Versioning: After the dual-write phase begins, the middleware layer marks all approximately 28,000 affected nodes as "cascading pending backfill" and writes them to the upgrade pending record table. Newly written BOM / process route records are automatically populated with cascading fields in the middleware layer: cavity_count is populated according to request parameters, bom_child_count is automatically calculated according to BOM hierarchy, and parallel_op_count is derived according to process route rules. Historical records are read by the middleware layer using default values ​​derived from field default values: cavity_count defaults to 1, bom_child_count and parallel_op_count retain their original values. The MES work order parser requires no modification and always obtains structurally complete data when reading BOM / process route data through the middleware route, without any "hierarchy mismatch" exceptions.

[0096] Batch-based cascading backfilling: The background backfilling task is executed in batches from leaf nodes to root nodes according to the BOM hierarchy to avoid inconsistencies in the calculation of the number of child nodes caused by parent nodes backfilling before child nodes. An adaptive rate-limiting algorithm controls the batch size, typically 300 to 800 nodes per batch, dynamically adjusted according to CPU load. The cascading backfilling of approximately 28,000 nodes was completed in about 15 minutes, with cascading marking taking about 90 seconds. During this period, the business system operated normally without any interruption.

[0097] Cascading Consistency Verification and Route Switching: After full backfilling, the middleware layer performs sampling verification on each orchestration in the affected orchestration set: randomly selects 100 BOM nodes and verifies the cascading consistency of cavity_count, bom_child_count, and parallel_op_count. Upon successful verification, a switchover state is triggered, and the read and write routes are fully switched to the target version's structure definition, completing the cascading upgrade.

[0098] In another specific embodiment, taking the upgrade of the work order form of the mold manufacturing platform as an example, the upgrade method and application effect are explained in detail.

[0099] A mold manufacturing company's industrial internet platform manages approximately 1 million work order records. The work order table is the platform's core data table, and the business systems (MES, APS, AGV scheduling) perform approximately 500 read / write operations on this table per second. The company needs to add a "priority" field to the work order table to support priority scheduling functionality. Using a traditional downtime upgrade solution, the estimated downtime is approximately 30 minutes, resulting in a production loss of approximately 100,000 yuan. In addition, the platform concurrently maintains approximately 500,000 BOM nodes (with a maximum nesting depth of 6 levels) and approximately 20,000 process route templates. During the platform's operation, an average of about 4 data structure changes occur per month, of which about 30% (approximately 1.2 times / month) involve changes to cascading fields in the BOM / process route. Using a traditional downtime upgrade solution, each cascading change upgrade results in an average downtime of approximately 45 minutes (including manual operation time for cascading field consistency alignment), resulting in a monthly production loss of approximately 540,000 yuan due to upgrade downtime.

[0100] After adopting the upgrade method provided in this application, the upgrade process is as follows: Loading and pre-verification of the target version structure definition takes approximately 5 seconds; after the dual-version coexistence mode is started, business read and write requests switch seamlessly; the background asynchronous upgrade task backfills 1 million rows of historical data in batches using an adaptive rate-limiting method, taking approximately 20 minutes (calculated at a rate of 700 rows per batch and 1 batch per second), during which the business system runs normally; switching the read and write route to the target version structure definition takes approximately 1 second. The entire upgrade process is uninterrupted, and the system response time remains at a normal level (average approximately 10ms).

[0101] Compared to traditional upgrade solutions that involve downtime (30 minutes of interruption, resulting in a loss of 100,000 yuan), this invention achieves upgrades with zero interruption and zero loss. During a 6-month operation period, the platform underwent 23 data structure changes (approximately 4 times per month on average), and the upgrade method provided in this application cumulatively avoided approximately 2.3 million yuan in downtime losses.

[0102] An industrial internet platform in an injection mold manufacturing workshop needs to add a "cavity count" field to the BOM table. This change will cascade and affect the number of child items in the BOM hierarchy (bom_child_count) and the number of parallel operations in the process route (parallel_op_count). This involves approximately 80,000 BOM nodes, approximately 5,000 process route nodes, and a total of approximately 28,000 nodes affected by the cascade.

[0103] After adopting the upgrade method provided in this application, the upgrade process is as follows: (1) In the pre-verification stage, the cascading impact range (approximately 28,000 nodes) is calculated by breadth-first traversal, which takes about 3 seconds. After confirmation by the maintenance personnel, the double-write stage is entered; (2) The cross-entity cascading update relationship cascading mark is written to the upgrade pending record table for approximately 28,000 affected nodes, which takes about 90 seconds; (3) The background batch backfill is executed in the BOM topology order from leaf node to root node, and the cascading backfill of approximately 28,000 nodes is completed with adaptive rate limiting (base batch size 800 nodes / batch, CPU utilization approximately 0.35), which takes about 15 minutes; (4) Before the switching stage, the cascading consistency sampling verification is performed (random 100 nodes, cavity_count→bom_child_count→parallel_op_count are all consistent). After passing, the route is switched and the upgrade is completed; (5) The MES work order parsing system has zero anomalies and the business system has zero interruptions throughout the entire process.

[0104] Compared to the traditional downtime upgrade solution (including manual cascading field alignment), which is expected to take about 45 minutes (about 5 minutes for database DDL and about 40 minutes for manual cascading alignment), the upgrade method provided in this application achieves zero downtime and zero anomaly upgrade, avoiding production stoppage losses of about 150,000 yuan (based on the company's estimated hourly production stoppage losses of about 200,000 yuan).

[0105] Secondly, embodiments of this application also provide a database upgrade device based on data middleware.

[0106] In one embodiment, reference is made to Figure 2 , Figure 2 This is a schematic diagram of the functional modules of a database upgrade device based on data middleware provided in an embodiment of this application. Figure 2 As shown, the database upgrade device based on data middleware includes: The building module is used to build a data middleware layer between memory and the database. In response to upgrade commands, the data middleware layer maintains both the current version structure definition and the target version structure definition. The interception module is used by the data middleware layer to intercept business read and write requests and perform route arbitration: For read requests, data is obtained according to the target version structure definition. If the data is missing, it is obtained according to the current version structure definition and the default value is filled. For write requests, data is written based on both the current version structure definition and the target version structure definition. The migration module is used to start background asynchronous upgrade tasks, dynamically adjust the batch size based on system load indicators, and migrate historical data in the database from the current version structure to the target version structure in batches. The switching module is used to switch the read / write route to the target version structure definition after all historical data migration is completed and consistency verification is passed, and then clear it after the current version structure definition is retained for a preset period of time.

[0107] The functions of each module in the aforementioned database upgrade device based on data middleware correspond to the steps in the aforementioned database upgrade method embodiment based on data middleware, and their functions and implementation processes will not be described in detail here.

[0108] Thirdly, embodiments of this application provide a database upgrade device based on data middleware. The database upgrade device based on data middleware can be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0109] In this embodiment of the application, the database upgrade device based on data middleware may include a processor, a memory, a communication interface, and a communication bus.

[0110] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0111] The communication interface includes input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the data middleware-based database upgrade device, as well as interfaces used for interconnecting the data middleware-based database upgrade device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0112] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0113] The processor can be a general-purpose processor, which can call a database upgrade program based on data middleware stored in memory and execute the database upgrade method based on data middleware provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the database upgrade program based on data middleware is called can be referred to in the various embodiments of the database upgrade method based on data middleware of this application, and will not be repeated here.

[0114] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0115] The present application provides a computer-readable storage medium storing a database upgrade program based on data middleware, wherein when the database upgrade program based on data middleware is executed by a processor, it implements the steps of the database upgrade method based on data middleware as described above.

[0116] The method implemented when the database upgrade program based on data middleware is executed can be referred to in various embodiments of the database upgrade method based on data middleware in this application, and will not be repeated here.

[0117] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0118] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0119] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0120] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0121] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.

[0123] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A database upgrade method based on data middleware, characterized in that, The database upgrade method based on data middleware includes: A data middleware layer is built between memory and the database. In response to upgrade commands, the data middleware layer maintains both the current version structure definition and the target version structure definition. The data middleware layer intercepts business read and write requests and performs route arbitration: For read requests, data is obtained according to the target version structure definition. If the data is missing, it is obtained according to the current version structure definition and the default value is filled in. For write requests, data is written based on both the current version structure definition and the target version structure definition. When writing to the memory cache according to the target version structure definition, a monotonically increasing logical timestamp is added to each write record. This timestamp identifies the time when the write operation occurs. When writing to the old database structure according to the current version structure definition, the data is reassembled based on the field set in the current version structure definition. New fields that exist in the target version structure definition but do not exist in the current version structure definition are excluded and not written to the old database table. Start an asynchronous background upgrade task, dynamically adjust the batch size based on system load indicators, and migrate historical data in the database from the current version structure to the target version structure in batches; Once all historical data migration is complete and consistency verification passes, the read / write route will be switched to the target version structure definition, and the current version structure definition will be retained for a preset duration before being cleared. The aforementioned initiation of the background asynchronous upgrade task dynamically adjusts the batch size based on system load metrics, migrating historical data from the current version structure to the target version structure in batches, including: Initiate a background asynchronous upgrade task to obtain the current CPU utilization and current lock wait time percentage of the database server. Based on the CPU utilization and lock wait time percentage, calculate the number of rows to be processed in the current batch, specifically: in, This indicates the number of rows processed in the current batch. Represents the amplitude limiting function. Indicates the minimum allowed batch size. This represents the floor function. Indicates the base batch size. This represents the CPU load sensitivity coefficient. This indicates the current CPU utilization rate. This represents the lock wait sensitivity coefficient. This indicates the current percentage of lock waiting time. Indicates the maximum allowed batch size; Adjust the number of processing rows in the current batch based on system load indicators so that the number of processing rows in the current batch decreases as system load increases and increases as system load decreases. The historical data in the database is migrated from the current version structure to the target version structure in batches until all historical data in the database has been migrated.

2. The database upgrade method based on data middleware according to claim 1, characterized in that: Both the current version structure definition and the target version structure definition include a structure version identifier, a field dependency graph, field default value derivation rules, cross-entity cascading update relationships for field changes, and the current upgrade stage status. The data middleware layer adopts a two-layer cache layout: the first layer is an in-heap cache, and the second layer is a distributed cache. The current version structure definition and the target version structure definition are first loaded into the in-heap cache. If the in-heap cache is not hit, it is retrieved from the distributed cache. If the distributed cache is not hit, it is loaded from the database and backfilled.

3. The database upgrade method based on data middleware according to claim 2, characterized in that, For read requests, data is retrieved according to the target version structure definition. If data is missing, it is retrieved according to the current version structure definition and filled with default values, including: For read requests, retrieve the requested data record from the heap cache according to the target version structure definition, and check if the data has been retrieved from the heap cache: If so, the retrieved data will be used as the response data for the read request; If not, query the distributed cache according to the target version structure definition to check if the distributed cache has retrieved the data: - If so, the retrieved data will be used as the response data for the read request; - If not, query the database according to the target version structure definition to check whether the target field of the requested data record is complete in the database. If yes, retrieve the requested data record according to the target version structure definition as the data to respond to the read request, and backfill the data record to the heap cache and distributed cache in sequence. If not, retrieve the existing field data of the requested data record according to the current version structure definition, and fill the missing fields with default values ​​according to the field default value inference rules.

4. The database upgrade method based on data middleware according to claim 3, characterized in that, The step of filling missing fields with default values ​​according to the field default value derivation rules includes: Check the configuration value of the missing field in the field default value inference rule: If the configuration value is a specific value, then that specific value will be used. If the configuration value is context-derived, the derived value is obtained from the current request context; if the context is unavailable, it is filled with a zero value of the field type. Specifically, when performing default value filling on the missing fields of a read request, the data middleware layer marks the requested data record as pending backfilling and adds the marked record to the backfilling set of the background asynchronous upgrade task.

5. The database upgrade method based on data middleware according to claim 2, characterized in that: The data middleware layer performs a breadth-first traversal on the field dependency graph, starting from the field involved in this version change. The system performs a forward traversal along the cross-entity cascading update relationship, collecting the fields that need to be backfilled at each level of the dependency chain to form a set of fields to be backfilled. Traverse in reverse along the cross-entity cascading update relationship to collect all business orchestration units that depend on the field to be backfilled, forming an affected orchestration set; The background asynchronous upgrade task determines the range of historical data nodes to be migrated based on the set of fields to be filled, and performs the migration of historical data within the node range in batches according to the batch size.

6. The database upgrade method based on data middleware according to claim 2, characterized in that: Compare the CPU utilization or service request latency with a preset threshold: When the CPU utilization reaches or exceeds the first preset threshold or the business request delay exceeds the second preset threshold, the background asynchronous upgrade task is paused. When the CPU utilization rate is lower than the third preset threshold and the service request latency is lower than the fourth preset threshold, the background asynchronous upgrade task will resume execution.

7. The database upgrade method based on data middleware according to claim 2, characterized in that, The consistency check specifically includes: Based on the required fields defined in the target version structure, the filling status of each field value of the migrated record is checked field by field. When all required fields are not empty, the migration record is deemed to have passed the verification. When the cumulative verification pass rate of all migration records reaches the preset condition, the consistency verification is deemed to have passed.

8. The database upgrade method based on data middleware according to claim 2, characterized in that: The state machine for the maintenance and upgrade phase of the data middleware layer; The state machine includes at least: a dual-write phase, a dual-read phase, and a switch completion phase; The dual-write phase is that the write request is written simultaneously based on the current version structure definition and the target version structure definition. The dual-read phase involves the read request prioritizing the acquisition of data according to the target version structure definition, and initiating the background asynchronous upgrade task. The switching completion stage involves switching all read and write routes to the target version structure definition and clearing the current version structure definition after the preset time period.

9. The database upgrade method based on data middleware according to claim 8, characterized in that, The double-write phase includes: When writing to the memory cache based on the target version structure definition, a monotonically increasing logical timestamp is appended to each write record, which identifies the time when the write operation occurred; When multiple write versions of the same data record exist in memory due to concurrent write operations, the logical timestamps of each version are compared, the version with the larger timestamp is retained, and the version with the smaller timestamp is overwritten.

10. The database upgrade method based on data middleware according to claim 8, characterized in that, The database upgrade method based on data middleware also includes: If an upgrade anomaly is detected within the preset time period, the data middleware layer will perform a rollback operation based on the current upgrade stage: During the initial stage of a double-write or double-read phase, the target version structure definition in the cache is marked as invalid and the route is switched back to the current version structure definition. In the later stages of the dual-read phase, the broadcast clears the target version structure definition cache and forces a switch back to the current version structure definition route. After the switchover is complete, the rollback is achieved by pausing read and write operations, restoring the current version structure definition from the distributed cache within the freeze window, and then unfreezing.

11. The database upgrade method based on data middleware according to claim 8, characterized in that, The step of switching the read / write route to the target version structure definition includes: Gradual switching based on tenant dimensions, with the data middleware layer maintaining a valid structure version identifier for each tenant independently in the heap cache to control the routing target; After the switch is completed, the current version structure definition is retained in the distributed cache for a preset time. During the retention period, any rollback triggered will directly restore the current version structure definition from the distributed cache. After the preset time expires, the data middleware layer removes the current version structure definition from the distributed cache, while the corresponding old fields in the database are retained and no field deletion operation is performed.

12. A database upgrade device based on data middleware, characterized in that, The database upgrade device based on data middleware includes: A building module is used to build a data middleware layer between memory and the database, which responds to upgrade commands and maintains both the current version structure definition and the target version structure definition. The interception module is used by the data middleware layer to intercept business read and write requests and perform route arbitration: For read requests, data is obtained according to the target version structure definition. If the data is missing, it is obtained according to the current version structure definition and the default value is filled in. For write requests, data is written based on both the current version structure definition and the target version structure definition. When writing to the memory cache according to the target version structure definition, a monotonically increasing logical timestamp is added to each write record. This timestamp identifies the time when the write operation occurs. When writing to the old database structure according to the current version structure definition, the data is reassembled based on the field set in the current version structure definition. New fields that exist in the target version structure definition but do not exist in the current version structure definition are excluded and not written to the old database table. The migration module is used to start background asynchronous upgrade tasks, dynamically adjust the batch size based on system load indicators, and migrate historical data in the database from the current version structure to the target version structure in batches. The switching module is used to switch the read / write route to the target version structure definition after all historical data migration is completed and consistency verification is passed, and then clear the current version structure definition after a preset duration. The aforementioned initiation of the background asynchronous upgrade task dynamically adjusts the batch size based on system load metrics, migrating historical data from the current version structure to the target version structure in batches, including: Initiate a background asynchronous upgrade task to obtain the current CPU utilization and current lock wait time percentage of the database server. Based on the CPU utilization and lock wait time percentage, calculate the number of rows to be processed in the current batch, specifically: in, This indicates the number of rows processed in the current batch. Represents the amplitude limiting function. Indicates the minimum allowed batch size. This represents the floor function. Indicates the base batch size. This represents the CPU load sensitivity coefficient. This indicates the current CPU utilization rate. This represents the lock wait sensitivity coefficient. This indicates the current percentage of lock waiting time. Indicates the maximum allowed batch size; Adjust the number of processing rows in the current batch based on system load indicators so that the number of processing rows in the current batch decreases as system load increases and increases as system load decreases. The historical data in the database is migrated from the current version structure to the target version structure in batches until all historical data in the database has been migrated.

13. A database upgrade device based on data middleware, characterized in that, The data middleware-based database upgrade device includes a processor, a memory, and a data middleware-based database upgrade program stored in the memory and executable by the processor, wherein when the data middleware-based database upgrade program is executed by the processor, it implements the steps of the data middleware-based database upgrade method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a database upgrade program based on data middleware, wherein when the database upgrade program based on data middleware is executed by a processor, it implements the steps of the database upgrade method based on data middleware as described in any one of claims 1 to 11.

Images