Index switching method, electronic equipment, storage medium and program product

Abstract

The invention provides an index switching method, electronic equipment, a storage medium and a program product, and relates to the technical field of computers. According to the method, an independent incremental log file is introduced to record data change, so that the service data is completely free from invasion, and a dictionary such as a timestamp in the service data does not need to be relied on or modified; the efficiency and the reliability of data migration are improved in a mode of synchronizing stock data firstly and then sequentially reading incremental logs to level incremental data; finally, by monitoring the delay and switching the index alias, the dynamic adjustment of the index fragmentation number can be safely and smoothly completed under the condition that the service application is not perceived and the service is not interrupted, and the complexity and the implementation cost of the scheme are effectively reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an index switching method, electronic device, storage medium, and program product. Background Technology

[0002] With the rapid development of internet applications, the explosive growth of data volume has posed a severe challenge to the elasticity and performance of backend data storage and retrieval systems. Elasticsearch, as a widely used distributed search engine, has its core advantage in providing high-throughput read and write capabilities by horizontally partitioning index data into multiple shards and storing them in a distributed manner.

[0003] However, its architecture has an inherent limitation: once an index is created, its number of shards cannot be directly modified. This limitation means that the number of shards set in the early stages of business operations often fails to match the subsequent growth in data volume or the evolution of business models. When the amount of indexed data continues to increase or the write pressure surges, the original number of shards may become a performance bottleneck, leading to increased query latency and decreased write throughput, directly impacting the user experience.

[0004] At this point, a common solution is to create a new index with an appropriate number of shards, migrate the data from the old index to the new index, and ultimately switch business traffic to the new index.

[0005] However, this existing technical solution has several significant drawbacks in practice. Its incremental data synchronization heavily relies on the existence of a reliable and monotonically increasing timestamp field (such as update_time) in the business data in order to perform range queries based on time windows. For indexes that do not have such a field, the business logic or data model needs to be modified to add the timestamp, which not only causes "intrusive" pollution to the business data, but also increases the complexity and implementation cost of the solution. Summary of the Invention

[0006] The purpose of this application is to provide an index switching method, electronic device, storage medium, and program product to improve the problems of existing methods that rely on timestamp fields to achieve incremental data query and synchronization, which can cause business data pollution, and increase the complexity and implementation cost of incremental solutions.

[0007] In a first aspect, embodiments of this application provide an index switching method, the method comprising: After a preset time, data operations on the source index will be written to the incremental log file; Synchronize the existing data of the source index before the preset time to the target index, wherein the number of shards of the target index is different from that of the source index; After the existing data synchronization is completed, the incremental data recorded after the preset time in the incremental log file is read and synchronized to the target index; After the synchronization delay of the incremental data meets the preset conditions, the index alias is switched to migrate business access from the source index to the target index.

[0008] In the above implementation process, by introducing an independent incremental log file to record data changes, complete non-intrusion on business data is achieved without relying on or modifying dictionaries such as timestamps in the business data; by first synchronizing existing data and then sequentially reading incremental logs to catch up with incremental data, the efficiency and reliability of data migration are improved; finally, by monitoring latency and switching index aliases, the dynamic adjustment of the number of index shards can be completed safely and smoothly without the business application being aware of it or interrupting service, effectively reducing the complexity of the solution and the implementation cost.

[0009] Optionally, switching the index alias after the synchronization delay of the incremental data meets a preset condition includes: Once the synchronization delay of the incremental data meets the preset conditions, the bidirectional data synchronization mechanism between the target index and the source index is initiated. After confirming that the bidirectional data synchronization mechanism is operating stably, switch the index alias.

[0010] In the above implementation process, bidirectional synchronization is initiated after the incremental data is basically caught up, so that the source index and the target index keep the data consistent in real time, forming a hot backup redundancy, which creates a more stable data foundation for the switching operation and reduces the risk of data inconsistency during the switching.

[0011] Optionally, initiating the bidirectional data synchronization mechanism between the target index and the source index includes: In response to a first synchronization request from the source index to the target index, incremental data in the incremental log file of the source index is continuously synchronized to the target index; In response to a second synchronization request from the target index to the source index, incremental data in the incremental log file of the target index is continuously synchronized to the source index; The first synchronization request and the second synchronization request include a synchronization flag, which is used to indicate that data synchronization is performed but the operation is not recorded in its own incremental log file.

[0012] In the above implementation process, the marking mechanism prevents the write operations generated during synchronization from being recorded as new incremental logs, fundamentally cutting off the path of infinite loop synchronization of data between the source index and the target index.

[0013] Optionally, the synchronization marker is a preset key-value pair field added to the HTTP header of the first synchronization request and the second synchronization request. By using widely supported HTTP protocol headers to transmit the marker, no modification to the underlying communication protocol is required, resulting in strong compatibility.

[0014] Optionally, before switching the index alias, the method further includes: In response to a data write request from a business application, business data is written to the source index and operation records are written to the incremental log file.

[0015] In the above implementation process, all business writes are persisted to the incremental log at the same time as they are successfully written to the source index, providing a complete and orderly data source for subsequent synchronization; moreover, business applications do not need to be aware of the underlying log mechanism and can write normally in the original way, which simplifies business logic and reduces the risk of transformation.

[0016] Optionally, after detecting that the bidirectional data synchronization mechanism is operating stably, switching the index alias includes: Once the bidirectional data synchronization mechanism is confirmed to be operating stably, data writing is prohibited. After a preset period of time during which data writing is prohibited, switch the index alias.

[0017] In the above implementation process, the write-prohibition operation puts the system into a static state, ensuring that all writes in transit are fully processed by bidirectional synchronization. After that, the alias switching is performed, which can eliminate the problem of data inconsistency caused by data delay.

[0018] Optionally, the switching of index aliases includes: Suspend business applications' data write requests to the source index; The index alias pointing to the source index is redirected to the target index, and the business application's data write requests to the target index are restored.

[0019] In the above implementation process, by first pausing the writing to the source index, the risk of concurrent write conflicts that may exist at the moment of switching is eliminated, ensuring the atomicity of the alias repointing operation; then the alias switching is completed and the writing is immediately resumed, compressing the business interruption time to the shortest (usually on the order of seconds), so that the business application can complete the migration of the underlying data storage with almost no awareness, and realize the business switching without interruption.

[0020] Optionally, after switching the index alias, the following may also be included: If a query anomaly or data write anomaly is detected in the target index, the index alias will be redirected to the source index.

[0021] In the above implementation process, when the target index experiences performance or functional abnormalities after switching, business traffic can be quickly switched back to the source index whose status is known and whose data is kept synchronized, thereby avoiding the spread of the fault and ensuring business continuity.

[0022] Optionally, the incremental log file is stored in binary format. Each log record includes a header information field and a tail content field. The header information field includes an operation timestamp, log data length, operation type, data identifier length, data content length, and data routing information length. The tail content field includes the data identifier content, data routing information content, and data content. Binary format significantly saves storage space compared to text format and enables rapid record location and parsing through a fixed-length header information field.

[0023] Optionally, the incremental log file includes multiple sequentially numbered log segment files, each with a set size. Fixed-size files avoid the operating system-level management and backup challenges caused by the unlimited growth of a single file, while also facilitating disk space pre-allocation to reduce fragmentation and improve I / O efficiency.

[0024] Optionally, reading the incremental data recorded after the preset time in the incremental log file and synchronizing it to the target index includes: The timestamp range in the incremental log file is determined to include the target log segment file at the preset time. Log records in the target log segment file whose data operation timestamps are equal to or later than the preset time, as well as log records in log segment files numbered after the target log segment file, are synchronized to the target index.

[0025] In the above implementation process, by reading the timestamp at the header of each segment file, the target file containing the preset time can be found in a short time without scanning massive amounts of data, thus improving the efficiency of data reading and synchronization.

[0026] Optionally, the method further includes: Delete the log segment files that have been synchronized to the target index. This reduces storage costs and prevents the log files from accumulating and running out of disk space.

[0027] Secondly, embodiments of this application provide an electronic device, including a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method provided in the first aspect above are performed.

[0028] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method provided in the first aspect above.

[0029] Fourthly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the steps of the method provided in the first aspect above.

[0030] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating an index switching method provided in this application embodiment; Figure 2 A schematic diagram illustrating the recording of incremental logs provided in an embodiment of this application; Figure 3 This is a schematic diagram of synchronized existing data provided in an embodiment of this application; Figure 4 This is a schematic diagram of synchronous incremental data provided in an embodiment of this application; Figure 5 A schematic diagram illustrating index alias switching provided in an embodiment of this application; Figure 6 A schematic diagram of the structure of log records in an incremental log file provided in an embodiment of this application; Figure 7 This is a schematic diagram of a plurality of segmented log files provided in an embodiment of this application; Figure 8 A structural block diagram of an index switching device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of an electronic device for performing an index switching method, provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] It should be noted that the terms "system" and "network" in the embodiments of this invention can be used interchangeably. "Multiple" refers to two or more; therefore, in the embodiments of this invention, "multiple" can also be understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0035] It should also be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0036] This application provides an index switching method. By introducing an independent incremental log file to record data changes, this method achieves complete non-intrusion on business data without relying on or modifying fields such as timestamps in the business data. By first synchronizing existing data and then sequentially reading incremental logs to catch up with incremental data, the efficiency and reliability of data migration are improved. Finally, by monitoring latency and switching index aliases, the dynamic adjustment of the number of index shards can be completed safely and smoothly without the business application being aware of it or interrupting service, effectively reducing the complexity and implementation cost of the solution.

[0037] Please refer to Figure 1 , Figure 1 A flowchart of an index switching method provided in this application embodiment, the method including the following steps: Step S110: After a preset time, write the data operations on the source index to the incremental log file.

[0038] The source index refers to the original Elasticsearch index that needs to be migrated and its shards adjusted, while the target index refers to the newly created target Elasticsearch index with a new number of shards. In other words, the number of shards in the target index is different from the number of shards in the source index.

[0039] An incremental log file can be a file that records all data change operations sequentially, independent of the Elasticsearch system. It records operation logs, independent of any user data fields in the index, with a fixed format, and is used only for data synchronization during the migration process, not for the final state of the data. Similar to a database binlog or an accounting ledger, it achieves zero intrusion into business data.

[0040] The preset time refers to the starting point of the entire migration process. All data records that already exist in the source index before this time are called existing data, and all data change operations on the source index generated by the business system after this time are called incremental data.

[0041] For example, suppose an e-commerce platform's order index initially has 5 shards. As the number of orders surges, query performance declines, and it needs to be expanded to 10 shards online without stopping or querying the service.

[0042] like Figure 2 As shown, at a predetermined preset time, such as time T, the logger is started. After that, all write requests to the source index (such as user placing an order, user modifying the order address, canceling an order, etc.) will be captured by the logger and written to the incremental log file at the same time as the data is successfully written to the source index. When writing the data operation to the incremental log file, key information of each operation can be recorded, such as operation time, type, data ID, new data content, etc., and serialized into a record and written to the incremental log file.

[0043] For example, after time T, user A places an order, the system writes the order to the source index, the logger synchronously generates a log record and writes it to the incremental log file, and this log record is used to record this data operation.

[0044] Step S120: Synchronize the existing data of the source index before the preset time to the target index.

[0045] After a preset time, a data synchronization task can be started synchronously. This task will read all existing data of the source index before the preset time, and then use batch processing to write all of this data into the newly created target index in parallel.

[0046] For example, such as Figure 3 As shown, the existing inventory synchronization task scans the source index X, reads the 1 million orders that existed before time T, and can write them to the target index Y at a rate of 1,000 orders per batch.

[0047] During the process of synchronizing existing data to the target index, the business can still write to and read from the source index normally without being affected, and the target index is not visible to the business during this period.

[0048] Step S130: After the existing data synchronization is completed, read the incremental data recorded after the preset time in the incremental log file and synchronize it to the target index.

[0049] Once the existing data synchronization is complete, the incremental synchronization task can be started. After starting the incremental synchronization task, it can sequentially read each log record from the incremental log file, starting from a preset position, as follows: Figure 4 As shown. Then, based on the log entries, the same operations (such as adding, updating, or deleting corresponding data) can be reproduced on the target index.

[0050] For example, if the incremental synchronization task reads the first log record, which is a newly added order, it can insert that order into the target index. If it reads the second log record, which is an update record for an order, it can find that order in the target index and update it. The synchronization process continues, constantly catching up with new changes made to the source index by the business.

[0051] Step S140: After the synchronization delay of incremental data meets the preset conditions, switch the index alias to migrate business access from the source index to the target index.

[0052] Synchronization latency can refer to the difference between the operation time of the latest record in the incremental log file and the operation time that the incremental synchronization task has processed. If the synchronization latency meets the preset conditions (such as less than 3 seconds), it means that the data of the target index and the data of the source index are almost in real time consistent.

[0053] During incremental data synchronization, the system can monitor the synchronization delay in real time. That is, it continuously compares the latest write time of the incremental log file with the operation time already processed by the incremental synchronization task. When the difference between the two (i.e., the synchronization delay) is consistently lower than the preset threshold, it indicates that the synchronization delay meets the preset conditions and the data is considered to have been basically caught up.

[0054] After the data is largely matched, an atomic operation can be performed to change the pointer of the index alias from the source index to the target index, such as... Figure 5 As shown.

[0055] An index alias is a virtual name for one or more real indexes. Business applications can access data through this index alias without directly binding to the specific index name. Switching index aliases can achieve seamless traffic migration.

[0056] In other words, after switching the index alias to point to the target index, subsequent business accesses can be migrated from the source index to the target index.

[0057] In the above implementation process, by introducing an independent incremental log file to record data changes, complete non-intrusion on business data is achieved without relying on or modifying dictionaries such as timestamps in the business data. By first synchronizing existing data and then sequentially reading incremental logs to catch up with incremental data, the performance pressure and uncontrollable latency issues caused by repeated queries of the source index in traditional solutions are avoided, improving the efficiency and reliability of data migration. Finally, by monitoring latency and switching index aliases, the dynamic adjustment of the number of index shards can be completed safely and smoothly without the business application being aware of it or interrupting service, effectively reducing the complexity and implementation cost of the solution.

[0058] Based on the above embodiments, when switching index aliases, a bidirectional data synchronization mechanism between the target index and the source index can be started after the synchronization delay of incremental data meets the preset conditions. After the bidirectional data synchronization mechanism is detected to be running stably, the index alias can be switched.

[0059] The bidirectional data synchronization mechanism refers to establishing a data synchronization channel between the source and target indexes during specific stages of data migration. This ensures that any new data operation received by one index is synchronized to the other index in real time, thus maintaining dynamic consistency of the two indexes before the switch. Its purpose is to create a secure data overlap area, providing higher reliability and second-level rollback capabilities for the switch operation.

[0060] Once the synchronization delay of incremental data meets the preset conditions, the bidirectional data synchronization mechanism is automatically triggered, that is, a bidirectional data synchronization channel is established: Source index A -> Target index B: Continue running the original incremental synchronization task to synchronize the new data operations of the source index to the target index.

[0061] Target Index B -> Source Index A: Initiate a new reverse synchronization task. This task reads the incremental log of target index B and synchronizes its data changes back to source index A. Starting from a preset time, the target index will also write data operations to its own incremental log file.

[0062] After bidirectional synchronization is initiated, the system continuously tracks the data. When the delays of both synchronization channels are close to zero, i.e., the delays are less than the set threshold, such as 2 seconds, the bidirectional data synchronization mechanism can be considered to be operating stably.

[0063] In some implementations, the hash values ​​of key data in the two indexes can be sampled and compared periodically to ensure consistency. When the data meets the consistency requirement and the delay is less than a set threshold, the bidirectional data synchronization mechanism can be considered to be operating stably.

[0064] In some implementations, the resource consumption of the processes of the two synchronization tasks can also be monitored. If the resource consumption is within a reasonable range and does not affect business operations, the bidirectional data synchronization mechanism can be considered to be operating stably. That is, this method can be used as an auxiliary condition for judging the stable operation of the bidirectional data synchronization mechanism.

[0065] In some implementations, stable operation can be determined when the bidirectional delay is less than a set threshold and the data meets the consistency requirement, and these indicators remain stable within a set time period (e.g., 2 minutes).

[0066] Once stable operation is confirmed, the index alias can be switched, and then the business solution can be switched to the target index to complete the index migration.

[0067] In the above implementation process, bidirectional synchronization is initiated after the incremental data is basically caught up, so that the source index and the target index keep the data consistent in real time, forming a hot backup redundancy, which creates a more stable data foundation for the switching operation and reduces the risk of data inconsistency during the switching.

[0068] Based on the above embodiments, in the method of initiating a bidirectional data synchronization mechanism, in response to a first synchronization request from the source index to the target index, incremental data in the incremental log file of the source index is continuously synchronized to the target index. In response to a second synchronization request from the target index to the source index, incremental data in the incremental log file of the target index is continuously synchronized to the source index. The first and second synchronization requests include synchronization flags, which indicate that data synchronization is performed but the operation is not recorded in its own incremental log file.

[0069] The first synchronization request can be initiated by the synchronization engine, which aims to synchronize incremental data from the source index to the write request of the target index.

[0070] The second synchronization request is also initiated by the synchronization engine, aiming to synchronize the incremental data of the target index back to the write requests of the source index.

[0071] Both types of synchronization requests are controlled and scheduled by the synchronization engine, which is different from the raw data operation requests directly initiated by business applications.

[0072] The first and second synchronization requests include a synchronization flag, whose core function is to declare to the receiving index that this request originates from the synchronization engine's internal data replication, rather than a direct business operation. In other words, it identifies a write request as originating from a data synchronization task rather than a primary business request. When processing a request with this flag, the receiving index will perform the data write, but will prevent this write operation from being recorded again in its own incremental log file, thus resolving the data loop problem in bidirectional synchronization.

[0073] In some implementations, the synchronization flag is a predefined key-value pair field added to the HTTP headers of both the first and second synchronization requests. This method utilizes widely supported HTTP headers to transmit the flag, eliminating the need to modify the underlying communication protocol and offering strong compatibility.

[0074] For example, a set of key and value combinations can be pre-defined in the HTTP header, which can be used to represent the source attribute of the request, i.e., which index the synchronous request originated from.

[0075] For example, the key name can be a name that has a clear meaning and is unlikely to conflict with business fields. Values ​​can define two specific values, one for each direction of the synchronous flow.

[0076] When identifying synchronization markers, you can parse the HTTP header of the request and then check whether it contains a preset key-value pair field. If it does, it means that the request carries a synchronization marker; if it does not, it means that the request does not carry a synchronization marker.

[0077] After a preset time, the incremental logging function of the source index remains enabled, continuously recording all business writes. When the bidirectional data synchronization mechanism is enabled, the incremental logging function of the target index is also enabled simultaneously. Thereafter, any operation that directly writes to the target index will also be recorded.

[0078] Once the bidirectional data synchronization mechanism is activated, two independent synchronous data streams are started: The first synchronization stream (source index -> target index): This stream continuously reads the incremental log file of the source index, sequentially retrieving each newly added business operation record. For each read log record, the synchronization engine constructs a write or delete request, ready to send it to the target index. Before sending this request, the synchronization engine inserts a specific synchronization flag, and then sends the request carrying the synchronization flag (i.e., the first synchronization request) to the target index.

[0079] The second synchronization stream (target index -> source index): This stream continuously reads the incremental log file of the target index, retrieving records sequentially. For each log record read, the synchronization engine constructs a write or delete request, ready to send it to the source index. Before sending this request, the synchronization engine inserts a specific synchronization flag, and then sends the request carrying the synchronization flag (i.e., the second synchronization request) to the source index.

[0080] Whether it's the source index or the target index, its logging component checks whether a write request contains a preset synchronization flag before processing it. If the flag is present, the data write or delete operation defined in the request is executed normally, updating the data. However, the step of recording this operation in its own incremental file is skipped because the operation itself was synchronized from the peer. If it were recorded again, it would be read by the local synchronization stream and synchronized back, creating an infinite loop. If the write request does not contain a synchronization flag, the data operation is executed normally, and this operation must also be recorded in its own incremental log file. This is because the request is identified as an original business request, and its operation needs to be recorded as the source of data changes for this index, for consumption by the synchronization stream.

[0081] In other words, if it is a data write request from a business application, the data write request will not carry a synchronization flag. That is, in response to the data write request, the business data will be written to the source index and the operation record will be written to its incremental log file.

[0082] The following example illustrates the above implementation process.

[0083] Assuming the business continues to operate, a user changes their order address.

[0084] At this point, the application sends a data write request to update order x1 in the source index. This request has no synchronization flag. The source index's logging component recognizes that the data write request has no synchronization flag, and therefore records this operation in the source index's incremental log file.

[0085] The first synchronization stream works as follows: It reads the update record for order x1 from the source index, constructs a first synchronization request to update x1, adds a synchronization flag, and sends this request to the target index. The target index's logging component recognizes the synchronization flag in the request, executes the update operation for x1, but does not record this update in the target index's incremental log file.

[0086] At this point, there is no update record for x1 in the incremental log file of the target index. Therefore, when the second synchronization stream reads the incremental data of the target index, it will not read this record, and naturally will not generate a request to synchronize it back to the source index. The loop is successfully blocked.

[0087] The second synchronization stream works similarly. If data is written directly to the target index at some point, the operation will be recorded in the incremental log file of the target index, and then synchronized back to the source index by the second synchronization stream with a synchronization tag. The source index does not record the tagged request in the incremental log file, thus preventing a loop.

[0088] In the above implementation process, the marking mechanism prevents the write operations generated during synchronization from being recorded as new incremental logs, fundamentally cutting off the path of infinite loop synchronization of data between the source index and the target index.

[0089] Based on the above embodiments, after the bidirectional data synchronization mechanism is detected to be operating stably, data writing can be prohibited. After the data writing is prohibited for a preset time, the index alias is switched.

[0090] In practice, the system can issue a write-prohibition command by means of configuration center, service gateway or direct notification to application cluster. The command carries a write-prohibition flag.

[0091] At the application layer, business applications can listen for this write-prohibited flag. Once the flag is detected, all write requests to the source and target indexes are temporarily suspended (e.g., placed in a memory queue) or an immediate system maintenance notification is returned. Read requests remain unaffected.

[0092] For the traffic layer, routing rules can be dynamically modified at the entry point of the Elasticsearch cluster (such as a load balancer, reverse proxy, or dedicated write proxy) to temporarily redirect all write request traffic to the source and target indexes to a buffer queue or return to wait for a response.

[0093] At this point, the source and target indexes no longer receive new business writes, but the existing two-way data synchronization mechanism is still running.

[0094] Simultaneously with issuing the write-prohibition command, a timer of preset duration is started. This preset duration is used to wait for data to catch up without delay. That is, even after the bidirectional data synchronization mechanism is running stably, there may still be a slight data delay. If data writing is prohibited at this time, the bidirectional data synchronization mechanism will continue to run, and the slight data delay can be quickly caught up. Therefore, the index alias is switched after waiting for the preset duration.

[0095] Once the index alias is successfully confirmed, the write-block flag can be cleared, and the application layer or traffic layer will resume normal routing. All new business write requests will be directed to the new target index according to the index alias.

[0096] Incremental logging of the source index can be disabled subsequently, and the running status of the target index can be monitored.

[0097] In the above implementation process, the write-prohibition operation puts the system into a static state, ensuring that all writes in transit are fully processed by bidirectional synchronization. After that, the alias switching is performed, which can eliminate the problem of data inconsistency caused by data delay.

[0098] Based on the above embodiments, when switching index aliases, the business application's data write requests to the source index can be paused first, then the index alias pointing to the source index can be redirected to the target index, and the business application's data write requests to the target index can be resumed.

[0099] After detecting that the synchronization delay of incremental data meets the preset conditions, a write-prohibition command is issued to instruct all business applications to temporarily stop initiating new data write requests to the source index. After each business application listens to the write-prohibition command, it will suspend all business logic that would trigger updates, deletions, or other operations on the source index.

[0100] Then perform the index alias switching operation, such as by performing a single alias update operation through Elasticsearch's REST API.

[0101] After the index alias is successfully switched, a recovery write command is issued. When the business application hears the recovery write command, its new data write requests are routed to the target index through the index alias.

[0102] In the above implementation process, by first pausing the writing to the source index, the risk of concurrent write conflicts that may exist at the moment of switching is eliminated, ensuring the atomicity of the alias repointing operation; then the alias switching is completed and the writing is immediately resumed, compressing the business interruption time to the shortest (usually on the order of seconds), so that the business application can complete the migration of the underlying data storage with almost no awareness, and realize the business switching without interruption.

[0103] Based on the above embodiments, if a query anomaly or data write anomaly of the target index is detected after switching the index alias, the index alias will be redirected to the source index.

[0104] Among them, query anomalies can refer to the query performance indicators of the target index exceeding the acceptable range, such as query latency higher than the baseline, abnormally high query error rate, and a surge in the number of slow queries.

[0105] Data write anomalies can refer to a failure in the write capability of the target index, such as an increased write rejection rate, an abnormal increase in write latency, or a write success but inconsistent data (e.g., the data cannot be queried after writing).

[0106] To ensure a safe rollback, after switching index aliases, you can avoid immediately cleaning up the old environment. Instead, maintain a specific state, such as stopping the synchronization stream from the source index to the target index while keeping the synchronization stream from the target index to the source index running. This allows the source index to continuously synchronize all new writes from the target index, making it a real-time updated hot backup. Furthermore, the source index can remain open and allocated sufficient resources to receive traffic at any time. Additionally, a one-click rollback interface or application API can be prepared in the console or automation system.

[0107] The system can deploy comprehensive monitoring probes to continuously monitor for anomalies in target index queries or data writes. Once an anomaly is detected, a rollback process is automatically or manually triggered. First, a write-disable command can be issued to all business applications or traffic gateways. Similar to the write-disable mechanism during a switchover, all write requests to the target index are suspended; read requests can usually continue.

[0108] Since the reverse synchronization stream (from target index to source index) is constantly running, theoretically the data in the source index is already very close to the target index.

[0109] In write-prohibited mode, the system can check the latency of the reverse synchronization stream. If the latency is 0 or close to 0, the index alias is directly switched back to the source index. If the reverse synchronization stream is stuck due to an anomaly in the target index, the system can attempt to read the remaining unsynchronized operations from the incremental log file of the target index and apply them to the source index through a backup channel, or record the missing data range for later compensation. Then, a data consistency check can be performed (e.g., comparing the list of data IDs written in the last 1000 records).

[0110] Once it is confirmed that the data has been caught up or the difference is acceptable, an atomic operation opposite to the switch can be performed, which is to switch the index alias back to the source index.

[0111] Then, the write-ban command can be cleared, and business writes can be resumed. At this point, all traffic returns to the source index through the index alias. Reverse synchronization streams are stopped, the target index is marked as abnormal, taken offline for troubleshooting, and no longer receives traffic.

[0112] In the above implementation process, when the target index experiences performance or functional abnormalities after switching, business traffic can be quickly switched back to the source index whose status is known and whose data is kept synchronized, thereby avoiding the spread of the fault and ensuring business continuity.

[0113] Based on the above embodiments, the incremental log files of the source index and the target index can be stored in binary format. The structure of each log record includes a header information field and a tail content field. The header information field includes the operation timestamp, log data length, operation type, length of data identifier, length of data content, and length of data routing information. The tail content field includes the content of data identifier, content of data routing information, and data content.

[0114] The header information field refers to the fixed-length section at the beginning of each log record, which contains metadata fields describing the basic attributes and length of the record.

[0115] The tail content field is the variable-length portion following the header, storing the actual business data content.

[0116] The format of the incremental log file is as follows: Figure 6 As shown, the meanings of each field are as follows: position: Represents the position where data has been read in the current file. When reading a piece of data, position points to the end of tail; head and tail together make up a piece of data. hesd: Represents the header information field, which includes the following fields: time: Operation timestamp, the precise time when the data operation occurred, usually an 8-byte (64-bit) integer, which can represent millisecond precision; logLength: Log data length, refers to the total number of bytes in the entire record (header + tail), helping to quickly locate record boundaries; opType: Operation type, identified by a single byte encoding, such as 0x01 = Add / Update, 0x00 = Delete; idLength: The length of the data identifier, the byte length of the document ID (_id), which is generally 4 bytes; docLength: The length of the data content, the byte length of the document source data (_source), generally occupies 4 bytes, and can represent up to nearly 4G of data, far exceeding the actual data length, which is sufficient for use; routingLength: The length of the data routing information, the length of the routing key in bytes, is 0 when there is no route, and occupies 4 bytes.

[0117] tail: Represents the tail content field, which includes the following fields: idContent: The content of the data identifier; routingContent: The content of data routing information; docContent: Data content; end: The end of the file, i.e., the length of the file.

[0118] After a preset time, when the new data operation of the source index is written to the incremental log file, the operation record is converted into the structure of the above log record and stored in the incremental log file.

[0119] When a synchronization task needs to read logs, it reads header data of the corresponding length from the current position, parses it according to its set format, and obtains relevant information in the header information field. Then, it can read the information in the tail content field based on the header information. Based on the read data, the complete write operation can be reconstructed.

[0120] The data structure of log records can be divided into two categories: newly added data and deleted data. For deletion operations, no data content is required; only a data identifier and data routing information are needed. For example, if the operation type is set to 0x00, the data content length is set to 0, and the end only contains the data identifier and data routing information. When there is no routing information, the route length is set to 0, and the end does not contain the route content. During reading, the route length determines whether to skip reading the route content. For adding operations, all the above fields are included. Its structured language can be represented as follows: [ { / / Add data "_id":"xxx232", / / Data ID "routing":"xx", / / Data routing "opType": 1, / / Added, upsert "opTime":1718767608214, / / Data write time, in milliseconds "source":"xxxx" / / Actual data content },{ / / Delete data; no data content is needed, only the id and routing information are required. "_id":"12343", / / Data ID "routing":"xx", / / Data routing "opType": 0, / / Delete "opTime":1718767608214 / / Data write time, in milliseconds } ] In some implementations, the log data length field for each record can be used to verify record integrity. If the number of remaining bytes is less than the length declared in the header during reading, it can be determined that the file is corrupted or the record is incomplete.

[0121] In the above implementation process, the binary format significantly saves storage space compared to the text format, and achieves fast record location and parsing through a fixed-length header information field.

[0122] Based on the above embodiments, such as Figure 7As shown, to facilitate fast reading of synchronized data, the incremental log file consists of multiple sequentially numbered log segment files, each with a set size. In practice, the incrementally written log files are switched to multiple files of fixed length, each approximately 2GB in size, to balance file management efficiency with disk I / O performance. After each data write, the system checks if the current file size exceeds 2GB; if so, a new file is automatically created and written.

[0123] The entire incremental log stream is divided into multiple physical files of a fixed size, known as log segment files. Each log segment file can contain log records within a consecutive time period, and these log segment files can be numbered chronologically. The numbering is a sequence of numbers used to identify the creation order of the log segment files, typically starting from 1 and incrementing, and formatted with a fixed number of digits for easy sorting by name.

[0124] When writing logs, the current log record is written to the log segment file with the latest number. When the size of the log segment file with the latest number reaches the set size, the system closes the file and creates a new log segment file, and assigns a new number to the new log segment file.

[0125] In the above implementation process, fixed-size files avoid the operating system-level management and backup problems caused by the infinite growth of a single file, while facilitating disk space pre-allocation to reduce fragmentation and improve I / O efficiency.

[0126] Based on the above embodiments, when reading incremental data from the source index and synchronizing it to the target index, the timestamp range in the incremental log file can be determined to include the target log segment file at a preset time. Then, log records in the target log segment file whose data operation timestamps are later than or equal to the preset time, as well as log records in the log segment files numbered after the target log segment file, can be synchronized to the target index.

[0127] In practice, each log segment file begins with a timestamp of the data operation performed on the first log record in that segment, typically occupying 8 bytes. The timestamps of data operations performed on the headers of two adjacent log segment files can form a timestamp range.

[0128] After the synchronization task is started, the first 8 bytes of data operation timestamps of two adjacent log segments in the incremental log file can be read sequentially to form a timestamp range. Then, the timestamp range is compared with the preset time. If it does not include the preset time, the next adjacent timestamp range is obtained and the comparison continues until the timestamp range includes the preset time.

[0129] For example, when a synchronization task is started, the incremental log file stores log segment file 1, log segment file 2, log segment file 3, and log segment file 4 in chronological order. When reading incremental data, the timestamp range of file 1 and file 2 is first obtained and then compared with a preset time. If the preset time is not included, the timestamp range of file 2 and file 3 is obtained and compared with the preset time. If the timestamp range includes the preset time, it indicates that file 2 and file 3 are the target log segment files.

[0130] This indicates that the target log segment file is the location of the file from which data needs to be pulled, allowing for quick location of specific files, such as adjacent files 2 and 3. During writing, each log record can be retrieved by parsing files 2 and 3. Since each log record includes a data operation timestamp, the timestamp can be compared with a preset time to locate log records whose timestamps are equal to or later than the preset time, and these log records can be synchronized to the target index.

[0131] Since the log segment files are numbered in chronological order, the data operation time of each log record in the log segment files numbered after this log segment file (such as file 4 in the example above) is after the preset time. Therefore, the log records in these log segment files can be directly synchronized to the target index without the need for comparison of timestamp ranges.

[0132] In the above implementation process, by reading the timestamp at the header of each segment file, the target file containing the preset time can be found in a short time without scanning massive amounts of data, thus improving the efficiency of data reading and synchronization.

[0133] Based on the above embodiments, in order to save storage resources, log segment files that have been synchronized to the target index can also be deleted.

[0134] As in the example above, after locating the target log segment file, synchronizing the relevant log records in the target log segment file to the target index, the target log segment file can be deleted.

[0135] Of course, in practical applications, a certain amount of time can be reserved. For example, the log segment files can be retained for a certain period of time after being synchronized to the target index, and then deleted after that period. Alternatively, after synchronizing several log segment files, the first synchronized log segment file can be deleted, and the synchronized log segment files can be deleted in the same way.

[0136] Specifically, the synchronization task can record the synchronization status of each log segment file, including synchronization in progress, synchronization complete, etc. After a log segment file completes synchronization, the synchronization task can mark it as synchronized and record the completion time. The segment can be deleted after a set time has elapsed since the completion time.

[0137] Of course, the system can also delete the log segment files that are synchronized first at fixed intervals, or delete the log segment files that have reached the set duration at fixed intervals.

[0138] For each deleted log segment file, it may not be deleted directly, but moved to a dedicated recycle directory. The recycle files can be finally cleaned up according to a longer retention policy (such as 7 days).

[0139] Please refer to the above method embodiments. Figure 8 , Figure 8 This is a structural block diagram of an index switching device 200 provided in an embodiment of this application. The device 200 may be a module, program segment, or code on an electronic device. It should be understood that the device 200 corresponds to the above method embodiment and is capable of performing the various steps involved in the method embodiment. The specific functions of the device 200 can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here.

[0140] Optionally, the device 200 includes: The incremental logging module 210 is used to write data operations on the source index to the incremental log file after a preset time. The existing data synchronization module 220 is used to synchronize the existing data of the source index before the preset time to the target index, wherein the number of shards of the target index is different from that of the source index; The incremental data synchronization module 230 is used to read the incremental data recorded after the preset time in the incremental log file after the existing data synchronization is completed, and synchronize it to the target index. The alias switching module 240 is used to switch the index alias after the synchronization delay of the incremental data meets a preset condition, so as to migrate the business access from the source index to the target index.

[0141] Optionally, the alias switching module 240 is used to initiate a bidirectional data synchronization mechanism between the target index and the source index after the synchronization delay of the incremental data meets a preset condition; and to switch the index alias after the bidirectional data synchronization mechanism is detected to be running stably.

[0142] Optionally, the alias switching module 240 is configured to, in response to a first synchronization request from the source index to the target index, continuously synchronize incremental data in the incremental log file of the source index to the target index; and in response to a second synchronization request from the target index to the source index, continuously synchronize incremental data in the incremental log file of the target index to the source index. The first synchronization request and the second synchronization request include a synchronization flag, which is used to indicate that data synchronization is performed but the operation is not recorded in its own incremental log file.

[0143] Optionally, the synchronization marker is a preset key-value pair field added to the HTTP header of the first synchronization request and the second synchronization request.

[0144] Optionally, the alias switching module 240 is used to write business data to the source index and write operation records to the incremental log file in response to a data write request from a business application.

[0145] Optionally, the alias switching module 240 is used to prohibit data writing after detecting that the bidirectional data synchronization mechanism is operating stably; and to switch the index alias after prohibiting data writing for a preset time.

[0146] Optionally, the alias switching module 240 is used to pause the data write request of the business application to the source index; redirect the index alias pointing to the source index to the target index; and resume the data write request of the business application to the target index.

[0147] Optionally, the alias switching module 240 is further configured to redirect the index alias to the source index if a query anomaly or data write anomaly of the target index is detected.

[0148] Optionally, the incremental log file is stored in binary format, and the structure of each log record includes a header information field and a tail content field. The header information field includes an operation timestamp, log data length, operation type, length of data identifier, length of data content, and length of data routing information. The tail content field includes the content of data identifier, content of data routing information, and data content.

[0149] Optionally, the incremental log file includes multiple sequentially numbered log segment files, each log segment file having a set size.

[0150] Optionally, the incremental data synchronization module 230 is used to determine that the timestamp range in the incremental log file includes the target log segment file at the preset time; and to synchronize log records in the target log segment file whose data operation timestamps are equal to or later than the preset time, as well as log records in log segment files numbered after the target log segment file, to the target index.

[0151] Optionally, the device 200 further includes: The data deletion module is used to delete log segment files that have been synchronized to the target index.

[0152] It should be noted that those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0153] Please refer to Figure 9 , Figure 9 This is a schematic diagram of an electronic device for executing an index switching method, provided in an embodiment of this application. The electronic device may include: at least one processor 310, such as a CPU; at least one communication interface 320; at least one memory 330; and at least one communication bus 340. The communication bus 340 is used to establish communication between these components. In this embodiment, the communication interface 320 is used for signaling or data communication with other node devices. The memory 330 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. Optionally, the memory 330 may also be at least one storage device located remotely from the aforementioned processor. The memory 330 stores computer-readable instructions; when these computer-readable instructions are executed by the processor 310, the electronic device performs the aforementioned method process.

[0154] Understandable. Figure 9 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown. Figure 9 The components shown can be implemented using hardware, software, or a combination thereof.

[0155] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it performs the method process executed by the electronic device in the above method embodiments.

[0156] This embodiment discloses a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can perform the methods provided in the above-described method embodiments, such as including: After a preset time, data operations on the source index will be written to the incremental log file; Synchronize the existing data of the source index before the preset time to the target index, wherein the number of shards of the target index is different from that of the source index; After the existing data synchronization is completed, the incremental data recorded after the preset time in the incremental log file is read and synchronized to the target index; After the synchronization delay of the incremental data meets the preset conditions, the index alias is switched to migrate business access from the source index to the target index.

[0157] In summary, this application provides an index switching method, electronic device, storage medium, and program product. This method achieves complete non-intrusion on business data by introducing an independent incremental log file to record data changes, without relying on or modifying dictionaries such as timestamps in the business data. By first synchronizing existing data and then sequentially reading incremental logs to catch up with incremental data, the efficiency and reliability of data migration are improved. Finally, by monitoring latency and switching index aliases, the dynamic adjustment of the number of index shards can be safely and smoothly completed without the business application being aware of it or interrupting service, effectively reducing the complexity and implementation cost of the solution.

[0158] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0159] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0160] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0161] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0162] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An index switching method, characterized by, The method comprises: writing data operation for source index into an incremental log file after a preset time point; synchronizing inventory data of the source index before the preset time point to a target index, wherein the target index is different from the source index in the number of shards; after the inventory data synchronization is completed, reading incremental data recorded in the incremental log file after the preset time point and synchronizing to the target index; after the synchronization delay of the incremental data meets a preset condition, switching index alias to migrate business access from the source index to the target index.

2. The method of claim 1, wherein, The switching of the index alias after the synchronization delay of the incremental data meets the preset condition comprises: after the synchronization delay of the incremental data meets the preset condition, starting a bidirectional data synchronization mechanism between the target index and the source index; after monitoring that the bidirectional data synchronization mechanism is stably running, switching the index alias.

3. The method of claim 2, wherein, The starting of the bidirectional data synchronization mechanism between the target index and the source index comprises: in response to a first synchronization request from the source index to the target index, continuously synchronizing incremental data in an incremental log file of the source index to the target index; in response to a second synchronization request from the target index to the source index, continuously synchronizing incremental data in an incremental log file of the target index to the source index; wherein the first synchronization request and the second synchronization request comprise a synchronization marker, and the synchronization marker is used to indicate that data synchronization is performed but the operation is not recorded in the incremental log file of itself.

4. The method of claim 3, wherein, The synchronization marker is a preset key-value pair field added in the HTTP header of the first synchronization request and the second synchronization request.

5. The method of claim 3, wherein, Before switching the index alias, the method further comprises: in response to a data write request of a business application, writing business data to the source index and writing operation records to an incremental log file.

6. The method of claim 2, wherein, The switching of the index alias after monitoring that the bidirectional data synchronization mechanism is stably running comprises: after monitoring that the bidirectional data synchronization mechanism is stably running, prohibiting data write; after prohibiting data write for a preset time length, switching the index alias.

7. The method of claim 1, wherein, The switching of the index alias comprises: suspending a data write request of the business application to the source index; repointing an index alias pointing to the source index to the target index, and resuming a data write request of the business application to the target index.

8. The method of claim 1, wherein, After switching the index alias, further comprising: if detecting query exception or data write exception of the target index, repointing the index alias to the source index.

9. The method of claim 1, wherein, The incremental log file is stored in a binary format, and the structure of each log record comprises a header information field and a tail content field, the header information field comprises operation timestamp, log data length, operation type, length of data identifier, length of data content, and length of data routing information, and the tail content field comprises content of data identifier, content of data routing information, and data content.

10. The method of claim 9, wherein, The incremental log file comprises a plurality of sequentially numbered log segment files, and the size of each log segment file is a set size.

11. The method of claim 10, wherein, The reading the incremental data recorded after the preset time point in the incremental log file and synchronizing to the target index comprises: determining a target log segment file in which a timestamp range in the incremental log file includes the preset time point; synchronizing, to the target index, log records in the target log segment file whose data operation timestamps are equal to or later than the preset time point and log records in a log segment file numbered after the target log segment file.

12. The method of claim 11, wherein, The method further comprises: deleting the log segment file that has been synchronized to the target index.

13. An electronic device, comprising: A device comprises a processor and a memory, the memory storing computer readable instructions that, when executed by the processor, perform the method of any one of claims 1-12.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, performs the method of any one of claims 1-12.

15. A computer program product, characterised in that, The computer program instructions, when read and executed by a processor, perform the method of any one of claims 1-12.

Images