A low-code platform column arrangement method

Abstract

The application provides a low-code platform column arrangement method, relates to the technical field of electric digital data, and comprises the following steps: in response to a structure change operation of a target column by a user, generating a column metadata change instruction, creating a second metadata version corresponding to a currently effective first metadata version, intercepting all data access requests for the target column through a data access agent layer, selecting a target metadata version from the first metadata version and the second metadata version according to a preset routing strategy, and routing the data access request to a corresponding physical data storage area to perform a data operation based on the target metadata version; starting an asynchronous data migration task, gradually migrating inventory data stored in a physical data storage area corresponding to the first metadata version to a physical data storage area meeting a definition of the second metadata version, and making subsequent data access requests be routed based on the second metadata version by default after the asynchronous data migration task is completed.

Description

Technical Field

[0001] This invention belongs to the field of electronic digital data technology, specifically relating to a low-code platform column arrangement method. Background Technology

[0002] In low-code development platforms, users can quickly build business applications through visual drag-and-drop, significantly lowering the barrier to software development. Existing low-code platforms typically allow users to dynamically create and modify categories (i.e., data table structures) and achieve adaptive rendering of front-end pages through metadata configuration.

[0003] However, when the core data structure of enterprise applications changes (such as adding fields, modifying field types, or adjusting column levels for business tables that have been around for several years), existing technologies face a fundamental challenge: how to smoothly upgrade the physical structure of the database without interrupting online business, modifying business code, or losing existing data. Traditional solutions have the following drawbacks: Maintenance shutdown method: The system service is suspended, database table structure changes (DDL operations) are performed, and data is migrated. For core business systems that run 24 / 7, downtime is unacceptable to the business, and the larger the data volume, the longer the downtime.

[0004] Modifying only metadata (such as the invention patent with publication number CN119646082A): Stores form configurations as JSON without changing the physical table structure. While this method avoids downtime, the newly entered data is inconsistent with the existing data storage structure, leading to logical errors in data query, statistics, and export functions, and it cannot support complex data type conversions and field-level validation.

[0005] Code generation methods (such as the invention patent with publication number CN113849165A): Generate new code based on the design and redeploy it. This method requires manual intervention in code merging, database script writing, and system restart. The process is complex and error-prone, and cannot achieve a truly zero-code smooth upgrade.

[0006] Therefore, how to achieve smooth online upgrades of the core column structure in a low-code platform without interrupting business operations, modifying code, or losing data has become a pressing technical problem in this field. Summary of the Invention

[0007] In view of the above-mentioned defects or deficiencies in the existing technology, a low-code platform column arrangement method is provided, including the following steps: In response to a user's structural change operation on the target column in the visual interface, a column metadata change instruction is generated; wherein, the structural change operation includes modification of column level, field definition or field attribute; Based on the column metadata change instruction, a second metadata version corresponding to the currently effective first metadata version is created in the metadata repository, and the first metadata version is kept in an effective state, while the second metadata version is marked as a state to be migrated; The data access proxy layer intercepts all data access requests for the target column, selects the target metadata version from the first metadata version and the second metadata version according to the preset routing strategy, and routes the data access request to the corresponding physical data storage area to perform data operations based on the target metadata version. An asynchronous data migration task is initiated. Based on the structural differences between the first metadata version and the second metadata version, the existing data stored in the physical data storage area corresponding to the first metadata version is gradually migrated to the physical data storage area that conforms to the definition of the second metadata version. During the migration process, the data access proxy layer ensures the consistency and integrity of the data. Once the asynchronous data migration task is completed, the second metadata version is marked as active, and the routing policy of the data access proxy layer is updated so that subsequent data access requests are routed based on the second metadata version by default.

[0008] According to the technical solution provided in this application, the preset routing strategy is a dynamic routing strategy based on data state, including: In response to the data access request intercepted by the data access proxy layer, the target data record identifier corresponding to the data access request is parsed. Query the preset migration status record table to obtain the migration status corresponding to the target data record identifier; wherein, the migration status includes not migrated, migrating in progress, and migrated; Perform routing selection based on the migration status: When the migration status is not migrated, the data access request is routed to the first physical data storage area corresponding to the first metadata version; When the migration status is "migrated", the data access request is routed to the second physical data storage area corresponding to the second metadata version; When the migration status is in progress, the data access request is temporarily stored or routed according to a preset locking mechanism.

[0009] According to the technical solution provided in this application, the migration status record table is maintained in conjunction with the asynchronous data migration task, specifically including: When performing data migration, the asynchronous data migration task reads the existing data records in the first physical data storage area one by one, performs data format conversion according to the structure definition of the second metadata version, and writes them into the second physical data storage area. After each data record is successfully written to the second physical data storage area, the asynchronous data migration task triggers an atomic update operation to update the migration status of the data record in the migration status record table from not migrated or in migration to migrated; The atomic update operation and the data write operation are located in the same database transaction, ensuring the consistency between changes in the physical storage location and state of data records, so as to avoid inconsistent states where data has been migrated but the state has not been updated.

[0010] According to the technical solution provided in this application, after marking the second metadata version as effective, the method further includes the following steps: Start access monitoring of the first physical data storage area corresponding to the first metadata version; Record the number of data access requests received by the first physical data storage area within a preset statistical period; If the number of data access requests is zero for N consecutive preset statistical periods, it is determined that the first physical data storage area has no business access dependency, and a physical deletion operation is triggered on the first physical data storage area, and the first metadata version is removed from the metadata repository simultaneously.

[0011] According to the technical solution provided in this application, before triggering the physical deletion operation on the first physical data storage area, the following steps are also included: At least one batch of data records is extracted from the first physical data storage area and the second physical data storage area respectively as verification samples; For each verification sample, compare whether the values ​​of its key fields are consistent in the two storage areas; The step of triggering a physical deletion operation on the first physical data storage area includes the following steps: If the key field values ​​of each verification sample are consistent in both storage areas, a physical deletion operation is triggered on the first physical data storage area.

[0012] According to the technical solution provided in this application, the step of performing route selection based on the migration status further includes the following steps: When the intercepted data access request is a modification request and the migration status of the target data record is in migration, perform the following operations: Pause the ongoing migration task for this data record and record the migration interruption point; The modification request is allowed to directly affect the first physical data storage area, update the content of the data record, and update the version identifier of the data record to the new version number after modification. After the modification request is completed, the migration status of the data record in the migration status record table is rolled back to non-migration, and its migration task identifier is cleared. The identifier of the data record is added back to the migration queue, and subsequent migration tasks will re-migrate based on the latest modified data.

[0013] According to the technical solution provided in this application, the step of performing route selection based on the migration status further includes the following steps: When the intercepted data access request is a read request and the migration status of the target data record is in migration, perform the following operations: Query the processing progress of this data record in the current migration task, including whether the write to the second physical data storage area has been completed; If the write operation has been completed, the latest migrated version of the data record is read from the second physical data storage area; If the write operation is not completed, the latest version of the data record is read from the first physical data storage area; The retrieved data records are returned to the requester.

[0014] According to the technical solution provided in this application, the following steps are also included: When the target section has data dependencies with other sections, scan and identify the dependency chain before performing structural change operations; Based on the dependency chain, multiple related columns are grouped into migration groups, and a unified migration batch number is assigned to each migration group. The asynchronous data migration task is performed in units of migration groups to ensure that data from multiple columns with dependencies are migrated in the same batch, avoiding cross-column data reference errors caused by inconsistent migration progress. When the migration of any item in the migration group is interrupted, the progress of the entire migration group is rolled back to the previous checkpoint.

[0015] According to the technical solution provided in this application, the scanning and identification of dependency chains includes a static dependency resolution step: Parse the field definition of the target column, identify whether there is a foreign key constraint configuration in the field, and if so, record the column target pointed to by the foreign key as a static dependent column; Parse the form configuration and page configuration of the target column to identify whether it references data from other columns as data sources or drop-down options. If so, record the column corresponding to the data source as a static dependent column. Parse the business rules and linkage logic configured in the target column, identify other column fields involved in the rule expression, and record the involved column targets as static dependent columns; Use all the static dependency columns marked in the above steps as the static dependency list of the target column.

[0016] According to the technical solution provided in this application, the scanning and identification of dependency chains further includes dynamic dependency monitoring and dependency chain aggregation steps: During runtime, all data query requests for the target category are recorded through the data access proxy layer; Parse the SQL statement or API parameters in the query request and extract the information of other columns involved in the related query. Within a preset time window, count the frequency with which the target category and other categories appear in the same query request. Other categories whose co-occurrence frequency exceeds a preset threshold are recorded as dynamically dependent categories. The static dependency list and the dynamic dependency category are merged and deduplicated to construct the direct dependency list of the target category. Using each item in the direct dependency list as a new target, the static dependency resolution and dynamic dependency monitoring steps are recursively executed until no new dependency item can be found, thereby forming a dependency chain containing all levels of dependency relationships.

[0017] Compared with the prior art, the beneficial effects of this application are as follows: 1. Zero Business Interruption: By creating a second metadata version and keeping the first metadata version active, combined with dynamic routing in the data access proxy layer, we ensure that user business requests can always be processed normally throughout the entire data migration process without interrupting system services, thus meeting the high availability requirements of enterprise-level applications.

[0018] II. Lossless Migration of Existing Data: Initiate an asynchronous data migration task to gradually migrate existing data to the new physical storage area based on the structural differences between the first and second versions. The migration process is executed automatically in the background without blocking foreground business operations, and atomic operations ensure the integrity and consistency of each data migration, completely avoiding data loss or corruption.

[0019] 3. Zero modification to business code: All data access requests are automatically distributed to the correct physical storage area by the proxy layer according to the routing policy. The business code does not need to be aware of changes in the underlying data structure, nor does it need to modify any query statements or data access logic, realizing configuration as upgrade.

[0020] IV. Smooth Switching and Rollback Capabilities: After migration, the default effective version is switched to the second metadata version by updating the routing policy, achieving a smooth transition between the old and new versions. If any anomalies are found after the switch, the routing can be switched back to the first metadata version at any time, providing a natural and fast rollback mechanism that greatly reduces upgrade risks.

[0021] V. End-to-End Data Consistency Guarantee: During the migration process, the migration progress of each data entry is precisely tracked through a migration status log table, and atomic transactions ensure strong consistency between data writing and status updates. The data access proxy layer performs precise routing based on real-time status, ensuring that users always read complete and accurate data, fundamentally avoiding data chaos caused by structural changes. Attached Figure Description

[0022] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A flowchart illustrating the steps of the low-code platform column arrangement method provided in this application. Detailed Implementation

[0023] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] As mentioned in the background section, this application proposes a low-code platform column arrangement method, such as... Figure 1 As shown, it includes the following steps: S1. In response to the user's structural change operation on the target column in the visual interface, generate a column metadata change instruction; wherein, the structural change operation includes modification of column level, field definition or field attribute; S2. Based on the column metadata change instruction, create a second metadata version in the metadata repository that corresponds to the currently effective first metadata version, keep the first metadata version in an effective state, and mark the second metadata version as a state to be migrated; S3. Intercept all data access requests for the target column through the data access proxy layer, select the target metadata version from the first metadata version and the second metadata version according to the preset routing strategy, and route the data access request to the corresponding physical data storage area to perform data operation based on the target metadata version; S4. Start the asynchronous data migration task. Based on the structural differences between the first metadata version and the second metadata version, gradually migrate the existing data stored in the physical data storage area corresponding to the first metadata version to the physical data storage area that conforms to the definition of the second metadata version. During the migration process, the data access proxy layer ensures the consistency and integrity of the data. S5. When the asynchronous data migration task is completed, the second metadata version is marked as active, and the routing policy of the data access proxy layer is updated so that subsequent data access requests are routed based on the second metadata version by default.

[0026] Specifically, this solution will be explained using an enterprise-level application scenario, where a company uses a low-code platform to manage the "Employee Information" section. This embodiment is based on a common Web development technology stack, including a front-end visual interface, back-end business services, a data access proxy layer, and a relational database.

[0027] The low-code platform's front-end provides a visual interface for designing column layouts, receiving user requests for changes to the structure. The left side of the interface displays the component panel, the middle canvas area shows the field structure of the current column, and the right side is the attribute configuration panel. Currently, the "Employee Information" column is running, corresponding to the first metadata version. This version defines the following fields: Employee ID, Name, Employee Number, and Department. As business grows, the user needs to add an "Emergency Contact" field to the employee information and change the data type of the "Employee Number" field from integer to string to adapt to the new business system requirements.

[0028] The user clicks the "Add Field" button in the canvas area, drags a "Single-Line Text" component from the left component panel to the end of the field list, and sets the field name to "Emergency Contact," the field type to string, and the length to 50 in the right properties panel. Then, the user clicks the "Employee ID" field and changes the type from "Integer" to "String" in its properties panel. After making the changes, the user clicks the "Save" button.

[0029] The frontend encapsulates this change operation into a structure change command and sends it to the backend metadata management service via an API request. This command contains the following information: the target field is identified as "Employee Information", the current effective version is identified as V1, and the specific operation list—adding the field "Emergency Contact" and modifying the type of the field "Employee ID".

[0030] Creating a second metadata version based on a change instruction: After receiving the change instruction, the backend metadata management service first queries the metadata repository for the currently effective first metadata version. The metadata repository maintains a metadata version table, recording all historical versions of each category. Each version includes fields such as version number, JSON data of the category structure definition, and status. Based on the category identifier and the condition that the status is "effective," the service obtains the complete structure definition of version V1.

[0031] The service performs a complete copy of the V1 structure definition, generating an independent copy. This copying process ensures that any subsequent modifications will not affect the still-active V1 version. The service then applies the changes from the change instructions to the copy: appending a new field definition to the end of the field list, containing the field name "Emergency Contact," type "String," and length 50; and changing the type attribute of the employee ID field from "Integer" to "String."

[0032] After the modifications are complete, the service assigns a unique version identifier to the new structure, such as V2, and sets its status to "Pending Migration." Simultaneously, it records the metadata information for this change, including the version based on V1, creation time, and a description of the changes. Then, the structure definition and version information for version V2 are inserted into the metadata version table. At this point, version V1 remains in "Effective," while version V2 remains in "Pending Migration," and both versions coexist in the metadata repository.

[0033] All data access requests are intercepted by a data access proxy layer: The system deploys a data access proxy layer between the business application and the database. This proxy layer is implemented as middleware, and all data access requests for the "Employee Information" section, whether query, add, modify, or delete operations, are first processed by the proxy layer.

[0034] The proxy layer internally maintains a routing policy table, recording the currently active metadata version for each category and its corresponding physical data storage area. The physical data storage area corresponds to a real database table: the first metadata version V1 corresponds to the first physical data storage area, i.e., the old table "employee_v1"; the second metadata version V2 corresponds to the second physical data storage area, i.e., the new table "employee_v2". Initially, since V2 is not yet active and the data has not been migrated, the proxy layer's routing policy is configured to route all requests to the old table "employee_v1" by default.

[0035] When the proxy layer receives a request, it parses the field identifier and operation type in the request, then determines the target physical table based on its internal routing strategy, and forwards the request to that table to perform data operations. Throughout this process, the business application does not need to be aware of the changes in the underlying table structure.

[0036] Initiating the asynchronous data migration task: The system initiates an asynchronous data migration task in the background. This task typically runs as a daemon or scheduled job, responsible for migrating existing data from the old table to the new table. The migration task first retrieves the first metadata version V1 and the second metadata version V2 of the target field from the metadata repository, and calculates the structural differences between the two versions. In this embodiment, the differences include the addition of the field "Emergency Contact" and a change in the type of the field "Employee ID".

[0037] The migration task reads existing data from the old table "employee_v1" according to a preset batch size, such as processing 1000 records at a time. For each read record, the data format is converted based on structural differences: a default value is assigned to the newly added field "Emergency Contact," such as an empty string; the value of the employee ID field is converted from an integer to a string. After the conversion is complete, the processed record is inserted into the new table "employee_v2."

[0038] During the migration process, the data access proxy layer continues to operate. Since the default route still points to the old table, all business requests still read or write data from the old table, so the migration task has no impact on front-end business operations. The migration task records the range of data that has been successfully migrated and continues to execute until all existing data has been migrated.

[0039] After migration is complete, switch to the effective version: Once the asynchronous migration task detects that all existing data has been successfully written to the new table, it sends a completion notification to the metadata management service. The metadata management service updates the status of version V2 from "Pending Migration" to "Effective," while updating the status of version V1 to "Deprecated" or retaining it as an archive. Subsequently, the service notifies the data access proxy layer to update the routing policy: all subsequent requests for the "Employee Information" section will be routed to the new table "employee_v2" by default.

[0040] Thus, the entire column structure change process was completed smoothly without interrupting business operations. Users did not need to stop the application throughout the process, and no business code needed to be modified; all data access was automatically adapted by the proxy layer.

[0041] Technical Principle: The core principle of this step is to decompose the database structure change, which originally required downtime, into two stages by introducing dual metadata versioning and an asynchronous migration mechanism. The first stage only modifies the metadata definition, and the business continues to run on the old version; the second stage asynchronously migrates the existing data in the background, and switches the route after the migration is complete. The data access proxy layer, as a unified access point, shields the underlying physical storage changes, making the business layer completely unaware of them.

[0042] Technical Benefits: First, zero business interruption. The system remains continuously available throughout the entire structural change process, requiring no downtime for maintenance, meeting the high availability requirements of enterprise-level applications. Second, lossless data protection. Existing data is completely migrated to the new structure through asynchronous tasks, and new data is directly written to the correct storage area according to the routing strategy, eliminating data loss or corruption. Third, zero modification to business code. All data access is automatically routed by the proxy layer; business code does not need to be aware of the table structure changes, and no redeployment is required after the upgrade. Fourth, smooth switching and rollback capability. The effective version is switched immediately after migration, and if problems are found, the routing can be switched back to the old version at any time, greatly reducing upgrade risks.

[0043] In a preferred embodiment, the preset routing policy is a dynamic routing policy based on data state, including: In response to the data access request intercepted by the data access proxy layer, the target data record identifier corresponding to the data access request is parsed. Query the preset migration status record table to obtain the migration status corresponding to the target data record identifier; wherein, the migration status includes not migrated, migrating in progress, and migrated; Perform routing selection based on the migration status: When the migration status is not migrated, the data access request is routed to the first physical data storage area corresponding to the first metadata version; When the migration status is "migrated", the data access request is routed to the second physical data storage area corresponding to the second metadata version; When the migration status is in progress, the data access request is temporarily stored or routed according to a preset locking mechanism.

[0044] Specifically, to achieve dynamic routing based on data status, the system introduces a separate migration status record table. This table tracks the migration progress of each data record, and its core fields include: data record primary key, column code, migration status, task identifier for processing the record, version identifier, and last update time. The migration status values ​​are "not migrated," "migrable," and "migrated."

[0045] While creating the second metadata version V2, the system initializes the migration status record table for the "Employee Information" section. The initialization process is as follows: Query the primary key of all existing data records from the old table "employee_v1", insert a record for each record into the status table, and set the status to "Not Migrated". For subsequently added data, records are simultaneously inserted into the status table while writing to the old table, with the initial status being "Not Migrated".

[0046] The proxy layer intercepts requests and parses data record identifiers: After intercepting a data access request for the "Employee Information" section, the data access proxy layer first needs to parse the target data record identifier operated on by the request. For query requests, the proxy layer extracts the primary key value from the WHERE condition of the SQL statement or API request parameters. For example, if the request is to query employee information with ID equal to 1001, the proxy layer parses the target record identifier as 1001. For batch operations, such as querying employees with IDs in the range of 1001 to 1010, multiple identifiers are parsed. For new requests, since there is no record identifier yet, the proxy layer processes it according to the default route before updating the status table.

[0047] Query migration status and perform route selection: The proxy layer queries the migration status record table in batches based on the parsed target record identifier to obtain the current status corresponding to each record. Then, it performs route selection based on the status. If the status is "not migrated", it means that the record has not yet been migrated to the new table. The proxy layer will route the request to the first physical data storage area, i.e. the old table "employee_v1", and perform the corresponding data operations.

[0048] If the status is "Migrated", it means that the record has been successfully migrated to the new table, and the proxy layer will route the request to the second physical data storage area, namely the new table "employee_v2".

[0049] If the status is "Migration in progress", it means that the record is being processed by an asynchronous migration task. At this time, the proxy layer cannot route directly, but will process it according to the preset locking mechanism.

[0050] Handling requests in the "Migration in Progress" state: When the target record of a request is in the "Migration in Progress" state, the proxy layer uses a temporary storage and polling approach for lock handling. Specifically, the proxy layer places the request and its parameters into a memory queue or persistent queue, and starts a timer. The proxy layer periodically re-queries the migration status of the record, for example, every 100 milliseconds. Once the status changes to "Migrated" or "Not Migrated," the temporary request is immediately retrieved from the queue, and routing is re-executed based on the new status. To avoid infinite waiting, a timeout threshold is set, for example, 30 seconds. After the timeout, an error response is returned and logged, prompting the user "Data is being migrated, please try again later."

[0051] For batch requests, if the status of multiple records involved is inconsistent, the proxy layer needs to perform split routing. For example, if a batch query involves ten records, seven of which have been migrated and three have not, the proxy layer will query the new table for seven records and the old table for three records, then merge the results and return them to the client. This processing is completely transparent to the business layer.

[0052] Maintaining the real-time nature of the status table: The migration status record table needs to be synchronized with business operations and migration tasks. When a business request directly operates on the old table, if the operation involves adding or deleting records, the proxy layer needs to synchronously update the status table. For example, when adding a record to the old table, a record with a status of "not migrated" needs to be inserted into the status table simultaneously. When deleting a record, the corresponding record in the status table needs to be deleted synchronously. These synchronization operations are completed within the same database transaction as the business operations to ensure consistency.

[0053] Technical Principle: The core of this routing strategy is to bind the physical location of data to its migration status, achieving precise routing through a status table. The status table acts as an index for the data location, enabling the proxy layer to dynamically select the correct storage area based on real-time status. For data in migration, temporary storage and polling mechanisms prevent operations from being performed during periods of data inconsistency, ensuring data atomicity and consistency. Splitting the routing of batch requests ensures the integrity of complex operations.

[0054] Technical Benefits: First, high routing accuracy. Every data request reaches the correct physical location, avoiding data corruption caused by asynchronous migration progress. Second, completely transparent migration process. The business layer does not need to be aware of whether data has been migrated; all complexity is shielded by the proxy layer, ensuring a consistent user experience. Third, effective concurrency control. Locking mechanisms handle critical states, preventing data races at migration points. Fourth, robust support for batch operations. Intelligent route splitting ensures the correctness and integrity of batch requests.

[0055] In a preferred embodiment, the migration status record table is maintained in conjunction with the asynchronous data migration task, specifically including: When performing data migration, the asynchronous data migration task reads the existing data records in the first physical data storage area one by one, performs data format conversion according to the structure definition of the second metadata version, and writes them into the second physical data storage area. After each data record is successfully written to the second physical data storage area, the asynchronous data migration task triggers an atomic update operation to update the migration status of the data record in the migration status record table from not migrated or in migration to migrated; The atomic update operation and the data write operation are located in the same database transaction, ensuring the consistency between changes in the physical storage location and state of data records, so as to avoid inconsistent states where data has been migrated but the state has not been updated.

[0056] Specifically, asynchronous data migration tasks need to be closely linked to the migration status record table to ensure the atomicity of data physical location changes and status changes. The migration task is triggered by the scheduler, and each run executes the following process: First, the migration task selects a batch of data to be processed from the migration status record table. The selection criteria are that the status is "not migrated" and it is not locked by other tasks. To avoid multiple tasks processing the same record simultaneously, the migration task preempts records through atomic update operations: it executes the update statement to change the status of all records that meet the criteria to "migrifying," and simultaneously writes the task identifier of the current task, then obtains the number of rows affected by the update. Only records that are successfully updated are obtained and processed by the current task.

[0057] Migrate data record by record and ensure atomicity: For each record retrieved, the migration task performs the following operations, and these operations must be completed within the same database transaction: First, the complete raw data is read from the first physical data storage area, i.e., the old table "employee_v1", based on the primary key of the record. Then, the raw data is formatted according to the structure definition of the second metadata version V2. In this embodiment, the conversion includes assigning a default value of an empty string to the newly added field "Emergency Contact" and converting the value of the employee ID field from an integer to a string. After the conversion is complete, the processed data is written to the second physical data storage area, i.e., the new table "employee_v2". If a record with this primary key already exists in the new table, an update operation is performed.

[0058] Within the same database transaction, the migration task updates the status of the corresponding record in the migration status record table from "Migration in Progress" to "Migrated". All the above operations are encapsulated using database transactions. In databases that support transactions, such as MySQL, a transaction is started using the "Start Transaction" command, and after the data write and status update are completed, the transaction is committed. If any step fails, such as a unique key conflict or network timeout when writing to a new table, the transaction is rolled back. In this case, the write to the new table is undone, and the record in the status table remains in the "Migration in Progress" state. To facilitate subsequent processing, the migration task can explicitly update the status back to "Not Migrated" after rollback, or rely on subsequent timeout mechanisms for automatic recovery.

[0059] Handling migration failures: If an unrecoverable error occurs during the migration process, such as abnormal data format conversion or incompatibility with the new table structure, the migration task should roll back its status to "Not Migrated" and log the error. The rollback operation must also be performed within a transaction: first, delete the data already written to the new table, then update the status to "Not Migrated." This way, the record will be reprocessed by subsequent migration tasks. If the error is due to problems with the data itself, such as data in the old table not meeting the data type requirements of the new table, manual intervention is needed to repair the data or adjust the migration rules.

[0060] Timeout Mechanism for Zombie Tasks: To prevent records from remaining in the "migration" state for extended periods due to task process crashes, a timeout mechanism is implemented. A scheduled scanning task is set up to periodically check the migration status record table for records whose status is "migration" and whose last update time exceeds a preset threshold. The threshold can be set according to business needs, such as 30 minutes. For these timed-out records, their status is forcibly rolled back to "not migrated," and the task identifier is cleared so that other tasks can reprocess them. The timeout mechanism ensures system robustness and prevents data from being permanently locked.

[0061] Migration progress recording and monitoring: Migration tasks need to record the range of data processed and the processing progress so that they can resume from the breakpoint if the task is interrupted. The current progress of each migration task can be recorded in a separate migration progress table, such as the maximum primary key of the processed record. When a migration task restarts, it first queries the progress table to retrieve the last processed position and then continues execution from that position. Simultaneously, the migration status record table itself provides a real-time migration progress view, allowing operations personnel to monitor the overall progress by querying and counting the number of records marked "Not Migrated," "Migrable," and "Migrated."

[0062] Technical Principle: This linkage mechanism leverages the atomicity of database transactions, binding the data writing to a new table and the status update operations together, ensuring that they either succeed simultaneously or fail simultaneously. This fundamentally eliminates the risk of data inconsistency due to system failures. Mutual exclusion between migration tasks is achieved through the "migration in progress" status and task identifiers in the status table, preventing duplicate migrations. A timeout mechanism provides additional fault tolerance, preventing permanent data locking due to task crashes. Progress logging ensures the ability to resume interrupted tasks.

[0063] Technical Benefits: First, strong consistency guarantee. The migration of each data record is atomic, eliminating the possibility of a "half-migration" state and completely avoiding dirty data issues. Second, high reliability. Transaction rollback ensures data integrity and losslessness in the event of failure, and the system can automatically recover. Third, concurrency safety. Multiple migration tasks can be executed in parallel, achieving natural mutual exclusion through atomic state updates, eliminating the need for additional distributed locks. Fourth, monitorability. The status table reflects the migration progress in real time, facilitating viewing and management by operations personnel. Fifth, self-healing capability. A timeout mechanism automatically cleans up zombie tasks, requiring no manual intervention. Sixth, breakpoint resumption. Migration progress records ensure that tasks can resume from the breakpoint after interruption, avoiding redundant work.

[0064] In a preferred embodiment, after marking the second metadata version as active, the method further includes the following steps: Start access monitoring of the first physical data storage area corresponding to the first metadata version; Record the number of data access requests received by the first physical data storage area within a preset statistical period; If the number of data access requests is zero for N consecutive preset statistical periods, it is determined that the first physical data storage area has no business access dependency, and a physical deletion operation is triggered on the first physical data storage area, and the first metadata version is removed from the metadata repository simultaneously.

[0065] Specifically, once the asynchronous data migration task is completed and the second metadata version is marked as active, the system needs to securely clean up the no longer-used first metadata version and its corresponding first physical data storage area. This embodiment provides an automatic cleanup mechanism based on access monitoring to ensure that old versions are only deleted when they are completely undependent on any business application.

[0066] Activate the access monitoring module: After the system switches the second metadata version to active status, it automatically triggers the access monitoring module for the first physical data storage area corresponding to the first metadata version. This monitoring module can run as an independent background process or be integrated into the data access proxy layer. The core task of the monitoring module is to record all data access requests pointing to the old table. Although the default route of the proxy layer now points to the new table, theoretically, some legacy access paths may still exist, such as outdated caches, old table names directly configured by external systems, and manually executed SQL scripts. If these accesses are not detected before deleting the old table, it may lead to business anomalies.

[0067] The monitoring module sets up an interception point in the data access proxy layer to capture and count any requests attempting to access the first physical data storage area. To ensure comprehensive coverage, the system can also use triggers or audit logs at the database level to assist in monitoring, but this embodiment prefers to use the proxy layer for unified interception because the proxy layer is the unified entry point for all data access in the system.

[0068] Access counts are recorded in statistical periods: The monitoring module counts the number of data access requests received by the first physical data storage area within each preset statistical period. The statistical period can be configured according to business characteristics. For example, for core business systems with frequent access, a shorter period such as 12 hours or 24 hours can be set; for non-core systems with less access, a longer period such as one week can be set.

[0069] Statistical results can be stored in a monitoring log table. Each record includes fields such as the start time, end time, and number of accesses for the statistical period. The monitoring module generates a statistical record at the end of each period and resets the counter to begin the next period's statistics. If no requests access the old table during a period, the count for that period is zero.

[0070] Determining No Business Access Dependencies: The system sets a continuous period N, for example, 7 consecutive days. After each statistical period, the monitoring module checks the access count records for the past N consecutive periods. If the access counts for all N periods are zero, it is determined that the first physical data storage area has no business access dependencies.

[0071] The setting of this N value needs to comprehensively consider the importance of the business and the risk tolerance. For core business systems, a longer period, such as 30 days, can be set to ensure absolute security; for non-core systems, a shorter period, such as 3 days, can be set to release storage resources more quickly. The judgment logic is executed automatically by the monitoring module periodically, without manual intervention.

[0072] Triggering physical deletion operation: Once the conditions are met, the monitoring module automatically triggers a physical deletion operation on the first physical data storage area. The deletion operation includes two parts: first, executing a command to delete the table in the database to completely remove the old table and all its data; second, synchronously removing the records of the first metadata version from the metadata repository, or marking their status as "deleted" or "archived," depending on the system design.

[0073] To ensure the traceability of deletion operations, the system records a complete operation log before deletion, including information such as deletion time, operation type, table names involved, and version numbers. If the system is configured with a backup policy, a backup can also be automatically performed before deletion, exporting the old table data to a file and storing it on the backup server for future use.

[0074] Technical Benefits: First, it automates the cleanup process. The system can automatically clean up old versions without manual intervention, significantly reducing the operational burden and avoiding human error. Second, it ensures security and reliability. Through strict conditions of continuous zero access, it ensures that deletion operations will not affect any running services, posing extremely low risk. Third, it supports flexible configuration. The N-value and statistical period can be flexibly adjusted according to business needs, adapting to the requirements of systems with different levels of importance. Fourth, it releases resources promptly. Timely deletion of useless data frees up database storage space, optimizes system performance, and reduces storage costs.

[0075] In a preferred embodiment, before triggering the physical deletion operation on the first physical data storage area, the following steps are further included: At least one batch of data records is extracted from the first physical data storage area and the second physical data storage area respectively as verification samples; For each verification sample, compare whether the values ​​of its key fields are consistent in the two storage areas; The step of triggering a physical deletion operation on the first physical data storage area includes the following steps: If the key field values ​​of each verification sample are consistent in both storage areas, a physical deletion operation is triggered on the first physical data storage area.

[0076] Specifically, this embodiment further requires adding a data consistency verification step before triggering the physical deletion operation to ensure that the data in the old and new storage areas are completely consistent, avoiding data loss after deletion due to omissions or errors that may occur during the migration process. This verification step is the last line of defense for the deletion operation.

[0077] Sample Extraction: After determining that the zero-access condition has been met for N consecutive periods, the system does not immediately perform deletion, but first initiates a data consistency verification task. The verification task requires extracting at least one batch of data records from the first physical data storage area and the second physical data storage area as verification samples.

[0078] The sampling method can employ random sampling to ensure the representativeness of the sample. In the specific implementation, the verification task first obtains the total number of records in the first physical data storage area, and then calculates the sample size according to a preset sampling ratio, such as sampling one percent of the total data or a fixed number, such as ten thousand records. To reduce the impact on database performance, sampling can be implemented by randomly selecting a range of primary keys. For example, first query the minimum and maximum values ​​of the primary key, and then randomly generate multiple primary key values ​​within the range for querying. Simultaneously, the primary key identifiers of these sample records are recorded for later locating the corresponding records in the new table.

[0079] Compare key field values ​​one by one: For each extracted verification sample, the verification task needs to compare whether its key field values ​​are consistent in the two storage areas. Key fields typically include the business primary key and all core business fields; the range of fields to be compared can also be selected according to the actual situation. To ensure the rigor of the verification, this embodiment adopts a full field comparison method.

[0080] The comparison process is as follows: For each sample record, query the corresponding record in the new table based on its primary key. If the record with that primary key does not exist in the new table, it means that this data was missed during the migration process, and the verification fails. If it exists, compare the values ​​of the two records field by field. Since field type conversions may have been performed during the migration process, such as converting integers to strings, the comparison needs to be performed according to the converted format, or the original stored values ​​can be compared directly. To simplify the comparison operation, the checksum of each record can also be pre-calculated during the migration, and the data consistency can be quickly determined by comparing the checksums.

[0081] The system determines whether to delete based on the verification results: if all verification samples have completely consistent key field values ​​in both storage areas, the data consistency verification is deemed to have passed, and the system continues to perform the physical deletion operation, that is, delete the old table and remove the metadata version.

[0082] If any inconsistent samples are found, the system immediately terminates the deletion process and generates a detailed inconsistency report. The report includes information such as the inconsistent record primary key, inconsistent field names, field values ​​in the old table, and field values ​​in the new table. The system writes this information to the error log and sends an alert to the system administrator via email or system message. At this point, both the old and new tables are retained, requiring manual intervention to investigate the cause of the inconsistency. Possible causes include concurrent modifications during the migration process not being handled correctly, program defects in the migration task, or accidental modification of the new table by other applications. After manually fixing the problem, the verification process can be restarted until it passes before performing deletion.

[0083] Technical Benefits: First, it provides dual protection. Adding data consistency verification on top of access monitoring creates a more secure deletion decision-making mechanism, reducing risk. Second, it ensures controllable risk. Deletion is immediately terminated upon detection of data inconsistency, preventing catastrophic data loss and ensuring data asset security. Third, it supports problem traceability. The verification report records detailed inconsistency information, facilitating rapid problem identification and remediation by operations personnel. Fourth, it offers flexibility and adaptability. Sample size and comparison field ranges can be configured according to actual needs, achieving a balance between verification accuracy and execution efficiency.

[0084] In a preferred embodiment, performing route selection based on the migration state further includes the following steps: When the intercepted data access request is a modification request and the migration status of the target data record is in migration, perform the following operations: Pause the ongoing migration task for this data record and record the migration interruption point; The modification request is allowed to directly affect the first physical data storage area, update the content of the data record, and update the version identifier of the data record to the new version number after modification. After the modification request is completed, the migration status of the data record in the migration status record table is rolled back to non-migration, and its migration task identifier is cleared. The identifier of the data record is added back to the migration queue, and subsequent migration tasks will re-migrate based on the latest modified data.

[0085] Specifically, when a data record is being processed by an asynchronous migration task, if a user initiates a modification request for that record, a coordination mechanism is needed to balance migration and modification to ensure eventual data consistency. Specifically: Intercepting modification requests and identifying migration status: After the data access proxy layer intercepts a data modification request for a target section, it first parses out the identifier of the target data record to be operated on, and then queries the migration status record table to obtain the current status of the record. If the query result shows that the status of the record is "Migration in progress", then the special processing flow of this step is entered.

[0086] Modification requests include update and deletion operations. This embodiment uses the update operation as an example for explanation; the processing logic for the deletion operation is similar. After identifying the migration status, the proxy layer needs to pause the current migration task's processing of the record to avoid data conflicts caused by simultaneous migration and modification.

[0087] Pause the current migration task: The proxy layer needs to coordinate with the asynchronous migration task to pause the migration task for this data record. The specific implementation is as follows: The proxy layer first attempts to acquire a lightweight lock on the record, such as a database row lock or a distributed lock, to prevent the migration task from continuing. Simultaneously, the proxy layer sets a special flag for this record in the migration status record table, such as a "Pause Migration" flag, or notifies the migration task by updating the status to "Migration Paused".

[0088] Before processing each record, the migration task checks if a pause flag exists. If a pause flag is found, the migration task skips the record, puts it back in the processing queue, or waits for a period of time before retrying. This approach does not forcibly interrupt ongoing transactions; instead, the migration task proactively pauses processing of the record at the next checkpoint.

[0089] Recording Migration Interruption Points: To enable subsequent migration recovery, the system needs to record migration interruption points. A field can be added to the migration status record table to record the steps the migration task has completed, such as "Data read but not written to the new table." Alternatively, a simpler approach is for the system to directly clear the migration task identifier for that record and maintain the status as "Migration in progress," automatically resuming later through a timeout mechanism. This embodiment uses the interruption point recording method for more accurate migration progress recovery.

[0090] Modification requests can be applied directly to the first physical data storage area: After pausing migration, the proxy layer allows modification requests to be applied directly to the first physical data storage area, i.e., to update the record in the old table. Simultaneously with the modification, the record's version identifier needs to be updated to the new version number. The version identifier can be a timestamp or an auto-incrementing number, used for subsequent detection of data changes.

[0091] This step ensures that business modification requests take effect immediately and are not blocked by migration tasks. Users receive real-time responses and have a good experience. After the modifications are complete, the proxy layer returns the results to the client.

[0092] Rollback migration status and re-add to queue: After a modification request is successfully completed, the proxy layer needs to update the migration status record table, rolling back the migration status of the record from "Migration in Progress" to "Not Migrated," and clearing the migration task identifier and pause flag. Simultaneously, the record's identifier is re-added to the pending migration queue for subsequent migration tasks to process.

[0093] The reason for rolling back to unmigrated is that the data has been modified. The previous migration task may have read the old data but not yet written it to the new table, or it may have written it but the data is outdated. In this case, the previous migration progress must be abandoned, and a new migration task must be launched to re-migrate based on the latest modified data. Re-adding the data to the migration queue ensures that this data is not missed.

[0094] Subsequent migration tasks will reprocess this record: When scanning the migration queue, subsequent asynchronous migration tasks will retrieve this record with a status of "not migrated" and then reprocess it according to the normal migration process: read the latest data from the old table, convert the format, write it to the new table, and atomically update the status to "migrated". Since the data version has been updated at this time, the data in the new table after migration will be consistent with the business modifications.

[0095] If another modification request occurs during the migration process, the above steps will be repeated to ensure that each modification is processed correctly. For frequently modified "hot" data, this mechanism may cause the record to be repeatedly re-queued, slowing down the migration process. Therefore, the system can combine this with a hot data processing mechanism to adopt special strategies for frequently modified records, such as temporary modification storage or staggered migration.

[0096] Technical Benefits: First, business priority is guaranteed. Modification requests take effect immediately, without being blocked by migration, resulting in a good user experience and meeting the requirements of online systems. Second, eventual data consistency is ensured. Migration tasks are re-migrated based on the latest data, ensuring the accuracy of data in the new tables. Third, lock-free waiting design. Through pause and re-queue mechanisms, long lock waits are avoided, resulting in high system concurrency performance and good resource utilization. Fourth, strong fault tolerance. Even if the migration task terminates abnormally during modification, state rollback can ensure that no data is lost, and the system has self-healing capabilities. Fifth, wide applicability. This mechanism is applicable to various types of modification operations, including updates and deletions, covering a complete range of business scenarios.

[0097] In a preferred embodiment, performing route selection based on the migration state further includes the following steps: When the intercepted data access request is a read request and the migration status of the target data record is in migration, perform the following operations: Query the processing progress of this data record in the current migration task, including whether the write to the second physical data storage area has been completed; If the write operation has been completed, the latest migrated version of the data record is read from the second physical data storage area; If the write operation is not completed, the latest version of the data record is read from the first physical data storage area; The retrieved data records are returned to the requester.

[0098] Specifically, when a data record is being processed by an asynchronous migration task, if a user initiates a read request for that record, it is necessary to ensure that the user can obtain the correct and complete data without affecting the normal progress of the migration task.

[0099] Intercepting read requests and identifying migration status: After the data access proxy layer intercepts a data read request for the target column, it first parses the identifier of the target data record to be operated on, and then queries the migration status record table to obtain the current status of the record. Read requests include single-record queries and batch queries. This embodiment uses a single-record query as an example for explanation; the processing logic for batch queries is similar. If the query result shows that the status of the record is "Migration in Progress," then the special processing flow in this step is entered.

[0100] Unlike modification requests, read requests do not change the data content, so there is no need to pause the migration task; it is only necessary to ensure that the data read is correct and complete. The proxy layer needs to determine which storage area to read data from based on the processing progress of the migration task.

[0101] Querying migration task processing progress: The agent layer needs to obtain the processing progress of the data record in the current migration task. Processing progress information can be stored in the migration status record table or recorded through a separate migration progress table. This embodiment adds a field to the migration status record table to record the processing stage of the record in the migration task, such as "read but not written" or "written but not committed".

[0102] The specific implementation is as follows: When processing each record, the asynchronous migration task records the current processing stage in the status table. For example, when the migration task reads data from the old table, it updates the processing stage to "Data read"; when the data is written to the new table, it updates it to "Data written"; and when the transaction is committed, it updates it to "Migration complete". This stage information provides a basis for judging read requests.

[0103] Selecting the read source based on processing progress: The proxy layer performs route selection based on the queried processing progress. If the processing progress indicates that the record has been successfully written to the second physical data storage area, meaning the data has been successfully written to the new table, the agent layer reads the latest migrated version of the record from the second physical data storage area. In this case, the data in the new table is complete and up-to-date, and can be directly returned to the user.

[0104] If the processing progress indicates that the record has not yet been written to the second physical data storage area—for example, only the old data has been read but not written to the new table, or the write has not yet been committed—the agent layer reads the latest version of the record from the first physical data storage area. At this point, the data in the old table is up-to-date because any modification requests will take precedence over the old table.

[0105] Special attention is needed for cases where the processing progress is "data written but not committed". Since the transaction has not yet been committed, data in the new table may be rolled back, posing a risk when reading from the new table. This embodiment adopts a conservative strategy, still reading from the old table to ensure data stability and reliability.

[0106] Batch read request processing: For batch read requests involving multiple records, the proxy layer needs to obtain the processing progress of each record separately, and then execute the corresponding routing strategy according to the progress distribution. If all records have been written to the new table, then read from the new table; if all records have not been written, then read from the old table; if the record status is mixed, the proxy layer needs to split the request, read from the new and old tables respectively, and then merge the results.

[0107] For example, if a batch query retrieves ten records, seven of which have been written to a new table and three have not yet been written, the proxy layer will query the new table for the seven records and the old table for the three records, then merge the results in the order requested and return them to the client. This processing is completely transparent to the business layer, and the user is unaware of the underlying data distribution.

[0108] Returning the read results: The proxy layer returns the data records read from the corresponding storage area to the requester. The returned data structure needs to be consistent with the metadata version definition to ensure that the client can parse it correctly. For data read from an old table, its structure conforms to the first metadata version definition; for data read from a new table, its structure conforms to the second metadata version definition. Since the structures of the two versions may differ, for example, the new table may have added fields, the proxy layer needs to perform necessary format unification when returning data, such as adding default values ​​to the old table data, so that the returned results conform to the format expected by the client.

[0109] Technical Benefits: First, real-time response to read requests. Users can obtain results immediately upon initiating a read request at any time, without waiting for migration to complete, ensuring system availability. Second, guaranteed data integrity. Whether reading from the old or new table, the returned data is the latest and complete, without missing or incorrect data. Third, uninterrupted migration. Read requests do not affect the normal progress of the migration task; both can be processed in parallel, improving overall system efficiency. Fourth, robust support for batch queries. Intelligent routing ensures the correctness and integrity of batch read requests, completely transparent to the business layer.

[0110] In a preferred embodiment, the following steps are also included: When the target section has data dependencies with other sections, scan and identify the dependency chain before performing structural change operations; Based on the dependency chain, multiple related columns are grouped into migration groups, and a unified migration batch number is assigned to each migration group. The asynchronous data migration task is performed in units of migration groups to ensure that data from multiple columns with dependencies are migrated in the same batch, avoiding cross-column data reference errors caused by inconsistent migration progress. When the migration of any item in the migration group is interrupted, the progress of the entire migration group is rolled back to the previous checkpoint.

[0111] Specifically, in real-world enterprise applications, columns are often interconnected through foreign keys, data references, etc. For example, the department number field in the "Employee Information" column might reference the "Department Information" column. If these related columns are migrated at different times, data inconsistencies or referencing errors may occur when querying across columns. This embodiment provides a solution based on migration groups. Specifically: Step 1: Identify Data Dependencies Between Categories: Before performing structural changes on the target category, the system first needs to scan and identify the dependencies between this category and other categories. Dependencies can be identified from multiple dimensions, including foreign key constraints at the database level, data source configurations at the application level, and field references in business rules. For example, for the "Employee Information" category, the system analysis reveals that its "Department Number" field has a foreign key constraint pointing to the "Department Information" category; simultaneously, in the "Employee Information" page configuration, the dropdown options for the "Department" field originate from the "Department Information" category. This information indicates that the "Employee Information" category depends on the "Department Information" category.

[0112] Dependency identification is a complex process that requires a combination of static configuration analysis and runtime behavior monitoring.

[0113] Step 2: Forming migration groups based on dependency chains: Based on the identified dependencies, the system constructs a complete dependency chain for the target category. For example, if it is found that "Employee Information" depends on "Department Information," and "Department Information" depends on "Department Manager Information," then the dependency chain is "Employee Information → Department Information → Department Manager Information." The system groups all categories in the dependency chain into a migration group and assigns a unique migration batch number to this group.

[0114] The composition of migration groups ensures that multiple sections with dependencies are treated as a single entity for migration. During subsequent asynchronous data migrations, sections belonging to the same migration group will be migrated according to a unified scheduling strategy, avoiding cross-section data referencing issues caused by inconsistent migration progress.

[0115] Step 3: Execute data migration by migration group: When the system starts the asynchronous data migration task, it no longer schedules it on a per-category basis, but rather on a per-migration-group basis. The migration task first obtains all migration groups in the pending migration state, and then allocates migration resources to each migration group. For multiple categories within the same migration group, the migration task adopts a collaborative scheduling strategy to ensure that the data of these categories is migrated within similar time windows.

[0116] In practice, migration tasks can be executed sequentially according to the order of the categories, or they can be performed in parallel while maintaining synchronized progress. This embodiment adopts a batch synchronization strategy: the migration task assigns the same batch size and batch interval to each category within the migration group, for example, migrating 1000 records from each category each time, and then submitting them all at once. In this way, even if the data volume of multiple categories is different, the relative synchronization of migration progress can be guaranteed.

[0117] Step 4: Handling Intra-Group Rollback During Migration Interruptions: When the migration process for any category within a migration group is interrupted, for example, due to system failure or data errors causing migration failure, the progress of the entire migration group needs to be rolled back to the previous checkpoint. This is because if some categories migrate successfully while others fail, data inconsistencies will occur when querying across categories.

[0118] In practice, before starting to process a batch, the migration task creates a checkpoint for that batch, recording the current processing position of each category. If an error occurs in any category during processing, the migration task rolls back the entire batch, undoes all written data, and restores the processing position of each category to the checkpoint state. Then, the migration task can decide whether to retry the current batch or skip the problematic data based on the error type. This intra-group rollback mechanism ensures strong consistency of data across all categories within the migration group.

[0119] Step 5: Version Switching After Migration: Once all existing data for all categories within the migration group has been migrated, the system will uniformly switch the metadata versions of these categories from the pending migration state to the active state. Version switching operations should also be performed atomically on a group-by-group basis, ensuring that either all categories switch simultaneously or none switch at all. If a category's version switching fails, the switching operations for other categories need to be rolled back to maintain overall consistency.

[0120] Technical Benefits: First, it ensures data consistency. Multiple categories with dependencies complete migration within a similar time window, preventing the mixing of old and new data during cross-category queries. Second, it avoids referencing errors. A unified rollback mechanism ensures that if any category within a group fails, other categories will not continue migrating, preventing data misalignment. Third, it supports complex business scenarios. Enterprise applications often have complex cross-table dependencies, and this mechanism effectively handles this complexity. Fourth, it offers strong scalability. The concept of migration groups can be flexibly combined to adapt to dependency networks of different sizes.

[0121] In a preferred embodiment, the scanning and identification of dependency chains includes a static dependency resolution step: Parse the field definition of the target column, identify whether there is a foreign key constraint configuration in the field, and if so, record the column target pointed to by the foreign key as a static dependent column; Parse the form configuration and page configuration of the target column to identify whether it references data from other columns as data sources or drop-down options. If so, record the column corresponding to the data source as a static dependent column. Parse the business rules and linkage logic configured in the target column, identify other column fields involved in the rule expression, and record the involved column targets as static dependent columns; Use all the static dependency columns marked in the above steps as the static dependency list of the target column.

[0122] Specifically, static resolution refers to directly extracting dependencies from existing system configuration information and metadata definitions. These dependencies are explicitly configured and exist statically.

[0123] Step 1: Parsing Foreign Key Constraints in Field Definitions: The system first parses the field definitions of the target field to identify whether foreign key constraints are configured in the fields. In relational databases, foreign key constraints are the most direct representation of dependencies between tables. For example, in the field definition of the "Employee Information" field, the "Department Number" field may have a foreign key constraint, pointing to the "Department Information" field. By querying the database's metadata, the system can obtain the definitions of all foreign key constraints, including information such as foreign key fields, referenced tables, and referenced fields.

[0124] For each foreign key constraint identified, the system records the field that the foreign key points to as a dependent field. For example, if the "Department Number" field of "Employee Information" is found to be a foreign key pointing to "Department Information," then "Department Information" is added to the dependency list. At the same time, the dependency type is recorded as "Foreign Key Dependency" for subsequent analysis.

[0125] Step 2: Parsing Data Source References in Form and Page Configurations: In low-code platforms, form and page configurations for various sections often reference data from other sections as data sources or dropdown options. For example, in the "Employee Information" edit form, the "Department" field might be configured as a dropdown selection box, with its option data sourced from the "Department Information" section. The system needs to parse this configuration information to identify the data source reference relationships.

[0126] In practice, the system scans all form and page configurations associated with the target section and extracts fields that reference data sources. Data source references can take various forms, such as API calls, database queries, and static data. For references that explicitly point to other sections, such as those configured with "Data Source Section = Department Information", the system marks "Department Information" as a dependent section and records the dependency type as "Data Source Dependency".

[0127] Step 3: Parsing Field References in Business Rules and Linkage Logic: Low-code platforms typically support configuring business rules and linkage logic, such as triggering updates to other fields when a field value changes, or performing data validation upon form submission. These rules and logic expressions may involve fields from other categories. The system needs to parse these expressions and identify the external fields referenced within them.

[0128] For example, a business rule is configured in the "Employee Information" section: when an employee's department changes, automatically update the employee's department manager field. This rule may reference the "Department Manager" field in the "Department Information" section. The system parses the rule expression, extracts the fields from other related sections, and marks the corresponding sections as dependent sections, with the dependency type being "Business Rule Dependency".

[0129] Step 4: Generate a static dependency list: All dependency categories identified in the previous three steps are deduplicated and merged to form a static dependency list for the target category. The static dependency list reflects all dependencies that can be clearly identified at the system configuration level. These dependencies are deterministic and stable, and will not change with runtime behavior.

[0130] The static dependency list is stored in the form of a data structure, such as a JSON object or a relational table, containing information such as the identifier of each dependency category, dependency type, and dependency strength. This information will provide the foundational data for subsequent dynamic monitoring and dependency chain aggregation.

[0131] Technical Benefits: First, accurate dependency identification. Dependencies extracted directly from configuration information are deterministic, preventing misjudgments or omissions. Second, broad coverage. Through three-dimensional parsing, dependencies at multiple levels, including the database layer, application layer, and logic layer, can be identified. Third, high execution efficiency. Static parsing does not require monitoring runtime behavior and can be executed in batches when the system is idle, minimizing its impact on system performance. Fourth, reusable results. The generated static dependency list can be stored in a metadata database for reuse in subsequent migration tasks.

[0132] In a preferred embodiment, the scanning and identification of dependency chains further includes dynamic dependency monitoring and dependency chain aggregation steps: During runtime, all data query requests for the target category are recorded through the data access proxy layer; Parse the SQL statement or API parameters in the query request and extract the information of other columns involved in the related query. Within a preset time window, count the frequency with which the target category and other categories appear in the same query request. Other categories whose co-occurrence frequency exceeds a preset threshold are recorded as dynamically dependent categories. The static dependency list and the dynamic dependency category are merged and deduplicated to construct the direct dependency list of the target category. Using each item in the direct dependency list as a new target, the static dependency resolution and dynamic dependency monitoring steps are recursively executed until no new dependency item can be found, thereby forming a dependency chain containing all levels of dependency relationships.

[0133] Specifically, the system dynamically monitors data query requests at runtime: During runtime, the system records all data access requests for the target category through a data access proxy layer. As the unified entry point for all data requests, the data access proxy layer intercepts every query operation, whether initiated through an application programming interface (API) or executed by a background task. For each intercepted query request, the proxy layer records the business objects and related information involved in the query.

[0134] The specific information recorded includes: the time the query was initiated, the main category of the query (i.e., which category the query primarily targets), other categories involved in the query, and the frequency of query execution. For example, when a user views employee information on the employee management page, the page may display not only data from the employee table but also the name of the department to which the employee belongs. This requires accessing data from the department table simultaneously. The proxy layer will record that this query involves both the "Employee Information" and "Department Information" categories.

[0135] Dynamic Dependency Identification through Co-occurrence Frequency Statistics: The system periodically performs statistical analysis on recorded query logs. The statistical time window can be set according to business characteristics, such as the most recent week or the most recent month. Within this time window, the system calculates the number of times the target category appears in the same query along with every other category; this number is called the "co-occurrence frequency."

[0136] The frequency of co-occurrence reflects the degree of correlation between two sections in business operations. For example, if the "Employee Information" and "Department Information" sections appear together in queries thousands of times in the past week, it indicates that these two sections are frequently accessed together in business operations and have a strong business correlation. Conversely, if the co-occurrence frequency between a section and the target section is extremely low, it may only be an accidental association or there may be no substantial dependency.

[0137] The system sets a co-occurrence frequency threshold, such as 100 times. Other categories with a co-occurrence frequency exceeding this threshold are marked as potentially dynamically dependent categories. These dynamic dependencies can arise from various situations: direct correlation queries in the application code, third-party systems accessing the system via APIs and associating multiple categories, or dependencies generated by reporting tools performing cross-table queries. These dependencies may not be reflected in the system's static configuration, but they do exist during actual runtime.

[0138] Merge static and dynamic dependencies to build a direct dependency list: Merge the static dependency list obtained through static resolution with the dynamic dependency list identified through dynamic monitoring in this step, and remove duplicates to finally form the direct dependency list of the target category.

[0139] The direct dependency list includes all identifiable other categories that have a direct dependency relationship with the target category. Each dependency, in addition to recording the category's identifier, also indicates whether the dependency originated from static resolution or dynamic monitoring, and provides dependency strength information (such as co-occurrence frequency). This information helps in subsequently determining the importance and stability of the dependency.

[0140] Recursively constructing the complete dependency chain: Simply identifying direct dependencies is insufficient, as dependencies may be transitive. For example, category A depends on category B, and category B depends on category C; therefore, there is an indirect dependency between A and C. If only A and B are migrated without migrating C, problems may still occur when A needs to access data from C. Therefore, it is necessary to recursively find dependencies at all levels.

[0141] The specific approach is as follows: Using each item in the direct dependency list as a new target, repeatedly execute the static dependency resolution step and the dynamic dependency monitoring step of this embodiment to find the dependencies of these items. For example, starting with target item A, we obtain its direct dependency list, which includes B and C. Then, using B as the target, we search for B's dependencies and find that B depends on D and E; using C as the target, we search for C's dependencies and find that C depends on F. We then continue searching using D, E, and F as targets until no new dependent items are found.

[0142] This recursive process is similar to digging deeper and deeper along the dependency chain, identifying all directly and indirectly related sections. The final result is a set of all sections reachable from the starting section, which can be organized according to the hierarchical relationship of dependencies to form a complete dependency chain.

[0143] Forming a complete dependency chain: All categories discovered during the recursive process are deduplicated and organized according to their hierarchical dependencies, thus forming a complete dependency chain for the target category. This chain can be represented using a tree structure or a directed graph, clearly showing the dependency paths between each category.

[0144] The system groups all categories in this dependency chain into the same migration group and generates a unique migration group identifier for that group. This identifier will be used for subsequent migration task scheduling to ensure that these related categories are migrated as a whole.

[0145] The results of dependency chain identification are stored in a metadata repository for use in subsequent migration tasks. When structural changes and migrations are required for a target category, the system can directly query its dependency chain to identify other categories that need to be migrated simultaneously, thereby avoiding cross-category data reference errors caused by inconsistent migration progress.

[0146] Technical Benefits: First, comprehensive dependency identification. Combining static and dynamic methods, it covers both configuration and runtime dependencies, avoiding the omission of any relationships that might affect data consistency. Second, discovery of implicit dependencies. By analyzing actual query logs, it can discover hard-coded relationships in the code, access to third-party systems, and other dependencies not reflected in the configuration, filling the blind spots of static parsing. Third, construction of a complete dependency chain. Recursive lookup ensures complete capture of multi-level dependencies, avoiding only processing direct dependencies while ignoring indirect dependencies, thus truly guaranteeing the collaborative migration of related data. Fourth, accurate migration group division. Migration groups determined based on the complete dependency chain ensure that all interdependent sections are migrated in the same batch, avoiding data referencing errors. Fifth, reusable results. The dependency chain is stored in the metadata database, which can be directly used in subsequent migrations, avoiding repeated identification and improving system efficiency.

[0147] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for arranging columns on a low-code platform, characterized in that, Includes the following steps: In response to a user's structural change operation on the target column in the visual interface, a column metadata change instruction is generated; wherein, the structural change operation includes modification of column level, field definition or field attribute; Based on the column metadata change instruction, a second metadata version corresponding to the currently effective first metadata version is created in the metadata repository, and the first metadata version is kept in an effective state, while the second metadata version is marked as a state to be migrated; The data access proxy layer intercepts all data access requests for the target column, selects the target metadata version from the first metadata version and the second metadata version according to the preset routing strategy, and routes the data access request to the corresponding physical data storage area to perform data operations based on the target metadata version. An asynchronous data migration task is initiated. Based on the structural differences between the first metadata version and the second metadata version, the existing data stored in the physical data storage area corresponding to the first metadata version is gradually migrated to the physical data storage area that conforms to the definition of the second metadata version. During the migration process, the data access proxy layer ensures the consistency and integrity of the data. Once the asynchronous data migration task is completed, the second metadata version is marked as active, and the routing policy of the data access proxy layer is updated so that subsequent data access requests are routed based on the second metadata version by default.

2. The low-code platform column arrangement method according to claim 1, characterized in that, The preset routing strategy is a dynamic routing strategy based on data state, including: In response to the data access request intercepted by the data access proxy layer, the target data record identifier corresponding to the data access request is parsed. Query the preset migration status record table to obtain the migration status corresponding to the target data record identifier; wherein, the migration status includes not migrated, migrating in progress, and migrated; Perform routing selection based on the migration status: When the migration status is not migrated, the data access request is routed to the first physical data storage area corresponding to the first metadata version; When the migration status is "migrated", the data access request is routed to the second physical data storage area corresponding to the second metadata version; When the migration status is in progress, the data access request is temporarily stored or routed according to a preset locking mechanism.

3. The low-code platform column arrangement method according to claim 2, characterized in that, The migration status record table is maintained in conjunction with the asynchronous data migration task, specifically including: When performing data migration, the asynchronous data migration task reads the existing data records in the first physical data storage area one by one, performs data format conversion according to the structure definition of the second metadata version, and writes them into the second physical data storage area. After each data record is successfully written to the second physical data storage area, the asynchronous data migration task triggers an atomic update operation to update the migration status of the data record in the migration status record table from not migrated or in migration to migrated; The atomic update operation and the data write operation are located in the same database transaction, ensuring the consistency between changes in the physical storage location and state of data records, so as to avoid inconsistent states where data has been migrated but the state has not been updated.

4. The low-code platform column arrangement method according to claim 1, characterized in that, After marking the second metadata version as active, the process further includes the following steps: Start access monitoring of the first physical data storage area corresponding to the first metadata version; Record the number of data access requests received by the first physical data storage area within a preset statistical period; If the number of data access requests is zero for N consecutive preset statistical periods, it is determined that the first physical data storage area has no business access dependency, and a physical deletion operation is triggered on the first physical data storage area, and the first metadata version is removed from the metadata repository simultaneously.

5. The low-code platform column arrangement method according to claim 4, characterized in that, Before triggering the physical deletion operation on the first physical data storage area, the following steps are also included: At least one batch of data records is extracted from the first physical data storage area and the second physical data storage area respectively as verification samples; For each verification sample, compare whether the values ​​of its key fields are consistent in the two storage areas; The step of triggering a physical deletion operation on the first physical data storage area includes the following steps: If the key field values ​​of each verification sample are consistent in both storage areas, a physical deletion operation is triggered on the first physical data storage area.

6. The low-code platform column arrangement method according to claim 2, characterized in that, The step of performing route selection based on the migration status further includes the following steps: When the intercepted data access request is a modification request and the migration status of the target data record is in migration, perform the following operations: Pause the ongoing migration task for this data record and record the migration interruption point; The modification request is allowed to directly affect the first physical data storage area, update the content of the data record, and update the version identifier of the data record to the new version number after modification. After the modification request is completed, the migration status of the data record in the migration status record table is rolled back to non-migration, and its migration task identifier is cleared. The identifier of the data record is added back to the migration queue, and subsequent migration tasks will re-migrate based on the latest modified data.

7. The low-code platform column arrangement method according to claim 6, characterized in that, The step of performing route selection based on the migration status further includes the following steps: When the intercepted data access request is a read request and the migration status of the target data record is in migration, perform the following operations: Query the processing progress of this data record in the current migration task, including whether the write to the second physical data storage area has been completed; If the write operation has been completed, the latest migrated version of the data record is read from the second physical data storage area; If the write operation is not completed, the latest version of the data record is read from the first physical data storage area; The retrieved data records are returned to the requester.

8. The low-code platform column arrangement method according to claim 1, characterized in that, It also includes the following steps: When the target section has data dependencies with other sections, scan and identify the dependency chain before performing structural change operations; Based on the dependency chain, multiple related columns are grouped into migration groups, and a unified migration batch number is assigned to each migration group. The asynchronous data migration task is performed in units of migration groups to ensure that data from multiple columns with dependencies are migrated in the same batch, avoiding cross-column data reference errors caused by inconsistent migration progress. When the migration of any item in the migration group is interrupted, the progress of the entire migration group is rolled back to the previous checkpoint.

9. The low-code platform column arrangement method according to claim 8, characterized in that, The scanning and identification of dependency chains includes a static dependency resolution step: Parse the field definition of the target column, identify whether there is a foreign key constraint configuration in the field, and if so, record the column target pointed to by the foreign key as a static dependent column; Parse the form configuration and page configuration of the target column to identify whether it references data from other columns as data sources or drop-down options. If so, record the column corresponding to the data source as a static dependent column. Parse the business rules and linkage logic configured in the target column, identify other column fields involved in the rule expression, and record the involved column targets as static dependent columns; Use all the static dependency columns marked in the above steps as the static dependency list of the target column.

10. The low-code platform column arrangement method according to claim 9, characterized in that, The scanning and identification of dependency chains also includes dynamic dependency monitoring and dependency chain aggregation steps: During runtime, all data query requests for the target category are recorded through the data access proxy layer; Parse the SQL statement or API parameters in the query request and extract the information of other columns involved in the related query. Within a preset time window, count the frequency with which the target category and other categories appear in the same query request. Other categories whose co-occurrence frequency exceeds a preset threshold are recorded as dynamically dependent categories. The static dependency list and the dynamic dependency category are merged and deduplicated to construct the direct dependency list of the target category. Using each item in the direct dependency list as a new target, the static dependency resolution and dynamic dependency monitoring steps are recursively executed until no new dependency item can be found, thereby forming a dependency chain containing all levels of dependency relationships.

Images