A cloud platform distributed storage active segment sealing optimization method, device and medium

Abstract

The application discloses a cloud platform distributed storage active segment sealing optimization method, equipment and medium, relates to the technical field of storage optimization, and comprises the following steps: cutting an industrial data object into fixed-length pages, writing the fixed-length pages into a main write page node and a mirror page node, writing a witness page containing an additional number into a witness page node, forming an active segment page chain, triggering a segment sealing operation when the active segment page chain reaches the end of a preset time window, performing consistency check on the active segment page chain through the main write page node, the mirror page node and the witness page node, and cutting off an unfinished tail page, thereby solidifying the active segment page chain into a sealed segment and generating a corresponding sealed segment list; the application solidifies a continuously written data page chain into a data sealed segment with a clear boundary, and uniformly identifies the data sealed segment in combination with the sealed segment list, improves data structure stability and data organization order, and further improves data retrieval accuracy, data traceability reliability, data archiving efficiency in a distributed storage environment and overall management capability.

Description

Technical Field

[0001] This invention relates to the field of storage optimization technology, and in particular to a method, device and medium for optimizing the sealing of active segments in distributed storage of a cloud platform. Background Technology

[0002] With the development of Industrial Internet, intelligent manufacturing, and Industrial Internet of Things (IIoT) technologies, a large number of industrial devices continuously generate structured or semi-structured industrial data objects. Industrial cloud platforms are gradually becoming a crucial infrastructure for storing and processing industrial data. Against this backdrop, distributed storage technology based on cloud computing architecture is widely used in industrial data management, enabling distributed writing, redundant backup, and high-availability access to massive amounts of industrial data objects through multi-node collaboration. Existing distributed storage technologies typically employ object storage or block storage structures, achieving reliable data storage through data sharding, data replication, and consistency protocols. For example, industrial data objects are split into multiple data blocks and distributed across different storage nodes, with replication mechanisms ensuring data security. Simultaneously, to meet the demands of continuous data acquisition and high-frequency writing in industrial scenarios, some distributed storage technologies adopt sequential write mechanisms or log-based write methods to improve data writing efficiency and stability.

[0003] However, there are still areas for improvement in existing storage optimizations. First, if there is a node write delay or the tail data is not written during the multi-node write process, it can easily cause inconsistent data order or incomplete structure, thus affecting the accuracy of subsequent data retrieval and data traceability. Second, traditional storage structures usually use data blocks or objects as the basic management center. During the continuous writing process, data can easily form a large number of discrete data blocks, lacking stable data segment boundaries, which is not conducive to subsequent data archiving and structured management. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a cloud platform distributed storage active segment sealing optimization method to solve the problems of data inconsistency or incomplete structure, and the formation of a large number of discrete data blocks lacking stable data segment boundaries.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a method for optimizing the sealing of active segments in distributed storage on a cloud platform, comprising, The industrial cloud platform receives industrial data objects sent by industrial equipment, parses them to generate industrial domain codes, determines the target storage domain based on the industrial domain codes, and simultaneously requests an append number from the sequence controller of the target storage domain to form a write ticket. By writing the ticket, the main write page node, mirror page node, and witness page node are selected in the target storage domain. The industrial data object is then divided into fixed-length pages according to the append number order and written to the main write page node and mirror page node. At the same time, the witness page containing the append number is written to the witness page node, forming an active segment page chain. When the active segment page chain reaches the end of the preset time window, the segment sealing operation is triggered. The active segment page chain is subjected to consistency verification through the main write page node, mirror page node and witness page node, and the incomplete tail page is truncated, thereby solidifying the active segment page chain into a sealed segment and generating a corresponding sealed segment list. An industrial traceability index is established based on the sealed segments and the sealed segment list to form a stable set of sealed segments; Perform segment-level synchronization operations on the stable archived segment set, send the archived segments and the list of archived segments to the central cloud storage node for archived segment reconstruction and return a synchronization confirmation mark, thus completing the stable archiving of the industrial cloud platform's distributed storage structure.

[0007] As a preferred embodiment of the cloud platform distributed storage active segment sealing optimization method of the present invention, the industrial domain code is generated by extracting factory identifiers, workshop identifiers, production line identifiers and process identifiers from industrial data objects and combining them according to preset mapping rules.

[0008] As a preferred embodiment of the cloud platform distributed storage active segment sealing optimization method of the present invention, wherein: the formation of the write ticket specifically includes: Extract the industrial domain code and timestamp from the industrial data object, and submit a write request to the sequence controller of the target storage domain. The sequence controller generates a unique append number based on the current write order. The appended number is combined and encapsulated with the industrial domain code and timestamp to form a write ticket.

[0009] As a preferred embodiment of the cloud platform distributed storage active segment sealing optimization method described in this invention, the selection of the main write page node, mirror page node, and witness page node refers to performing node mapping calculation in the node list of the target storage domain according to the append number written in the ticket, and determining the main write page node, mirror page node, and witness page node respectively based on the mapping result.

[0010] As a preferred embodiment of the cloud platform distributed storage active segment sealing optimization method of the present invention, wherein: the formation of the active segment page chain specifically includes: Obtain the append number from the written ticket and read the content of the industrial data object. Perform sequential segmentation of the industrial data object according to the preset page length to generate multiple fixed-length pages. The fixed-length pages are sequentially numbered according to the append number and written to the main write page node and the mirror page node in turn. A witness page is generated based on the append number and the fixed-length page number, and the witness page is written into the witness page node. At the same time, a sequential connection relationship is established between the fixed-length pages according to the append number order to form an active segment page chain.

[0011] As a preferred embodiment of the cloud platform distributed storage active segment sealing optimization method of the present invention, wherein: the step of solidifying the active segment page chain into a sealed segment and generating a corresponding sealed segment list specifically includes: Read fixed-length pages from the active segment page chain from the main write page node, read corresponding fixed-length pages from the mirror page node, read witness pages from the witness page node, and perform sequential sorting on the fixed-length page sequence according to the append number order to form a segment-level page sequence; Perform sequential consistency checks on the segment-level page sequence based on the append number records in the witness page sequence, and determine the position of the last valid append number in the active segment page chain during the check process; The segment-level page sequence is truncated by the last valid append number position, and the boundaries of the sealed segment are determined according to the continuous interval of the append number to generate the sealed segment. A segment structure fingerprint is generated based on the append number range, the number of fixed-length pages, and the node identifier information of the main write page node and the mirror page node of the sealed segment. The segment structure fingerprint, append number range, and number of fixed-length pages are then written into the sealed segment list.

[0012] As a preferred embodiment of the cloud platform distributed storage active segment sealing optimization method of the present invention, wherein: the formation of a stable sealed segment set specifically includes: The sealed segments are grouped and organized according to the industrial domain code to form a sealed segment group sequence; The sequential continuity of the sealed segments is checked based on the append number interval in the sealed segment grouping sequence, and the structural consistency of the sealed segments is checked based on the segment structure fingerprint and fixed-length page number information in the sealed segment list. Valid sealed segments are selected based on the sequential continuity check results and the structural consistency check results, and an industrial traceability index is established based on the industrial domain code, append number range, and segment structure fingerprint of the valid sealed segments. The valid sealed segments are arranged sequentially according to the interval of the appendix number to form a stable set of sealed segments.

[0013] As a preferred embodiment of the cloud platform distributed storage active segment sealing optimization method described in this invention, the return of a synchronization confirmation flag specifically includes: Organize the sealed segments and their corresponding lists in the stable sealed segment set into a sealed segment synchronization sequence according to the order of the appended number intervals; The archived segments and the list of archived segments in the synchronization sequence are sent to the central cloud storage node. The central cloud storage node performs archived segment structure reconstruction processing based on the append number range, fixed-length page number and node identification information in the archived segment list. Based on the segment structure fingerprint in the archived segment list, perform a structure consistency check on the reconstructed archived segments, and generate a synchronization confirmation flag upon successful verification, then return to the target storage domain.

[0014] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the cloud platform distributed storage active segment sealing optimization method as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the cloud platform distributed storage active segment sealing optimization method as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By sealing the active segment page chain at the end of the preset time window and performing consistency checks and tail page truncation on the data written by multiple nodes, the problems of inconsistent data order and incomplete data structure caused by node writing delays or incomplete tail data writing during the multi-node writing process are effectively avoided, thereby ensuring that the data structure remains stable and complete during the writing phase. At the same time, by solidifying the continuously written data page chain into data sealing segments with clear boundaries and uniformly identifying them with a sealing segment list, the storage structure that was originally prone to forming discrete data blocks is transformed into a structured data organization method with clear segment boundaries, thereby improving the stability of the data structure and the orderliness of data organization. Furthermore, by performing consistency checks on the sealing segment structure information and fixed-length page data content, the accuracy of data retrieval, the reliability of data traceability, and the data archiving efficiency and overall management capabilities in a distributed storage environment are further improved. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart for optimizing the active segment sealing method of distributed storage in cloud platforms.

[0019] Figure 2 A flowchart for generating active segment page chains.

[0020] Figure 3 A flowchart for generating the archived segments and the list of archived segments.

[0021] Figure 4 A flowchart for generating synchronization confirmation markers. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 As an embodiment of the present invention, this embodiment provides a cloud platform distributed storage active segment sealing optimization method, including the following steps: S1: The industrial cloud platform receives industrial data objects sent by industrial equipment, parses them to generate industrial domain codes, determines the target storage domain based on the industrial domain codes, and simultaneously requests an append number from the sequence controller of the target storage domain to form a write ticket.

[0026] The industrial cloud platform receives industrial data sent by industrial equipment, performs field parsing processing on the industrial data, reads factory identifiers, workshop identifiers, production line identifiers, and process identifiers from the industrial data, and performs combined encoding processing on the factory identifiers, workshop identifiers, production line identifiers, and process identifiers according to preset mapping rules to generate industrial domain codes. Specifically, it concatenates or performs numerical encoding operations on each identifier according to a preset field order to generate a unique identifier string. The preset mapping rules are used to specify the arrangement order and encoding method of the factory identifier, workshop identifier, production line identifier, and process identifier in the industrial domain code. For example, the preset mapping rules can specify that the industrial domain code is formed by concatenating the factory identifier field, workshop identifier field, production line identifier field, and process identifier field in a fixed order.

[0027] The industrial cloud platform performs a retrieval process in the target storage domain mapping table based on the industrial domain code. The target storage domain mapping table records the correspondence between industrial domain codes and target storage domains. The correspondence is a mapping relationship established in advance according to the encoding rules or business partitioning rules of the industrial domain codes. It is used to divide different industrial domain codes into different storage domains. The industrial cloud platform determines the target storage domain corresponding to the industrial domain code based on the retrieval results. For example, when the industrial domain code contains a specific factory identifier and production line identifier, the target storage domain is determined by looking up the table to match the corresponding storage domain number.

[0028] The industrial cloud platform extracts the industrial domain code and timestamp from the industrial data object and sends a write request to the sequential controller of the target storage domain. The write request contains the industrial domain code and timestamp information.

[0029] The sequence controller of the target storage domain generates a unique append number based on the arrival order of write requests and returns this unique append number to the industrial cloud platform. The sequence controller, deployed as a sequence allocation management center within the target storage domain, is used to assign monotonically increasing append numbers to data write requests entering the current target storage domain. The sequence controller can be implemented as a single-node service or as a coordination component based on a distributed consensus protocol to ensure the uniqueness and order of append numbers in multi-device concurrent write scenarios. The append number is a sequence number used to identify the write order of industrial data objects; it remains unique and monotonically increasing within the same target storage domain and is used to determine the write position of the industrial data object in the active segment page chain.

[0030] The industrial cloud platform combines and encapsulates the unique append number, industrial domain code, and timestamp to form a write ticket. The write ticket records the unique append number, industrial domain code, and timestamp information. The write ticket is used for subsequent write scheduling of the primary write page node, mirror page node, and witness page node in the target storage domain.

[0031] It should also be noted that the write ticket is a write authorization record generated after the sequence controller returns an append number, used to identify that the current industrial data object has obtained a write sequence number. The write ticket includes at least the append number, industrial domain code, and timestamp information, and is used for subsequent write scheduling of the primary write page node, mirror page node, and witness page node in the target storage domain.

[0032] S2: By writing the ticket, select the main write page node, mirror page node, and witness page node in the target storage domain, and divide the industrial data object into fixed-length pages according to the append number order and write them to the main write page node and mirror page node. At the same time, write the witness page containing the append number to the witness page node to form an active segment page chain.

[0033] S2.1: The industrial cloud platform reads the append number from the written ticket and obtains all storage node identification information from the node list of the target storage domain. The node list is a collection of storage node registration information obtained from the target storage domain. The node list records the identification and node sequence number of all storage nodes in the target storage domain that can participate in data writing.

[0034] The industrial cloud platform performs node mapping processing on the node list based on the append number. The node mapping processing determines the mapping position according to the correspondence between the append number and the length of the node list. For example, the node mapping sequence number is obtained by performing a modulo operation on the append number and the length of the node list. The node mapping sequence number corresponds to a storage node identifier in the node list.

[0035] The process of locating the first storage node identifier in the node list based on the node mapping sequence number involves using the node mapping sequence number as the index position in the node list, sequentially reading the corresponding storage node identifiers, and identifying the first storage node identifier as the primary write page node. Next, following the node mapping sequence number, the process continues to select the second storage node identifier (i.e., sequentially reading the storage node identifiers from the next index position) and identifies the second storage node identifier as the mirror page node. Finally, following the node mapping sequence number, the process continues to select the third storage node identifier (i.e., sequentially reading the subsequent storage node identifiers) and identifies the third storage node identifier as the witness page node, thus completing the process of determining the primary write page node, mirror page node, and witness page node.

[0036] It should also be noted that the primary write page node is used to store the primary data copy of fixed-length pages; the mirror page node is used to store the mirror copy corresponding to the primary write page node, so as to provide data recovery capability in the event of failure of the primary write node; and the witness page node is used to save the witness record during the data page writing process, so as to assist in the execution of consistency verification in the subsequent segment sealing stage.

[0037] To further explain, in order to ensure the reliability of the primary write page node, mirror page node, and witness page node in a distributed environment, this invention further adds node state constraints and fault domain isolation rules to the node mapping process.

[0038] Specifically, after determining the node mapping sequence number by taking the remainder of the append sign and the node list length, the industrial cloud platform filters valid nodes in the node list based on node status information. The node status information includes at least the node's online status, available capacity, and load status. When a node (a storage node in the node list) is offline or its available capacity is lower than a preset capacity threshold, the current node does not participate in the current node mapping calculation (i.e., the calculation process of determining the node mapping sequence number by taking the remainder of the append sign and the node list length). The preset capacity threshold is set based on the total storage capacity of the storage nodes, preferably 10% to 20% of the total node capacity. This is to reserve necessary capacity buffers while ensuring that the node still has available write space, thereby avoiding write failures or frequent data migrations when the node is close to full load.

[0039] After determining the primary write page node, the industrial cloud platform performs fault domain isolation verification on candidate nodes during the selection of mirror page nodes and witness page nodes. The fault domain must include at least a rack domain or availability zone, and each storage node records the corresponding fault domain identifier information in the node list.

[0040] The fault domain isolation rule is as follows: When a candidate mirror page node and the primary write page node are in the same fault domain, the industrial cloud platform continues to sequentially select the next node as the mirror page node from the node list until the mirror page node and the primary write page node are in different fault domains. When determining the witness page node, the industrial cloud platform simultaneously verifies the fault domain relationship between the witness page node and the primary write page node, as well as the mirror page node. When the witness page node and the primary write page node are in the same fault domain, or the same fault domain as the mirror page node, the industrial cloud platform continues to select new candidate nodes until the witness page node and the primary write page node, as well as the mirror page node, are all in different fault domains.

[0041] By using the above fault domain isolation rules, the primary write page node, mirror page node, and witness page node are deployed in different fault domains, thereby avoiding the simultaneous failure of multiple replicas due to a single rack or single availability zone failure.

[0042] In cases of uneven node capacity (meaning there are differences in available capacity or load levels among storage nodes), the industrial cloud platform can also dynamically sort the node list according to the node load status, so that nodes with low loads participate in node mapping first, thereby reducing the node load skew caused by concentrated writing of hot industrial domain codes.

[0043] In scenarios involving node expansion (adding new storage nodes to the node list) or node decommissioning (removing existing storage nodes from the node list or marking them as unavailable), the industrial cloud platform updates the node list and re-executes the node mapping rules, enabling the new nodes to participate in subsequent data writing while ensuring the stability of the existing archive segment structure. This achieves a smooth transition of the distributed storage system in node expansion and contraction scenarios.

[0044] S2.2: The industrial cloud platform reads the append number from the write ticket and reads the complete data content from the industrial data (referring to the original data content in the industrial data sent by the industrial equipment in S1). The industrial cloud platform performs sequential segmentation processing on the industrial data object according to the preset page length. The preset page length is used to specify the number of data bytes contained in each fixed-length page. For example, the preset page length can be set to 4KB or 8KB. The industrial cloud platform continuously extracts data segments from the starting position according to the preset page length, following the data order of the industrial data object. The data order is the storage arrangement order of the data in the industrial data object. Each data segment forms a fixed-length page until all the data of the industrial data object is segmented, thereby generating multiple fixed-length pages.

[0045] The industrial cloud platform performs page-level sequential numbering on industrial data objects based on append numbers. The page-level order refers to the sequential arrangement of fixed-length pages within the original data after the industrial data has been segmented. The append number identifies the write order of the industrial data object in the target storage domain and is an object-level number. The fixed-length page number identifies the sequential position of each fixed-length page within the industrial data object and is a page-level number; the fixed-length page number is the same as the page number. After an industrial data object is segmented into multiple fixed-length pages, the industrial cloud platform generates a page number for each fixed-length page according to the order of the fixed-length pages within the industrial data object, with page numbers starting from 1 and incrementing sequentially. Therefore, each fixed-length page is uniquely identified by a combination of "append number + page number," where the append number indicates the write order of the current industrial data in the target storage domain, and the page number indicates the page position within the current industrial data object. Fixed-length pages are written to the primary write page node according to their page numbers, and simultaneously, fixed-length pages with the same page number are written to the mirror page node. The primary write page node and the mirror page node each store completely identical fixed-length page data content.

[0046] The industrial cloud platform generates witness page records based on the append number and page number. Specifically, it combines and encapsulates the append number, page number, and storage node identifier information of the corresponding fixed-length page to form a witness page record. The witness page record stores the append number, page number, and storage node identifier information of the corresponding fixed-length page. The industrial cloud platform writes the witness page record to the witness page node. The industrial cloud platform organizes the fixed-length pages in the main write page node sequentially according to the page number order, and records the preceding page number and following page number for each fixed-length page to establish the connection relationship between adjacent fixed-length pages, thereby forming an active segment page chain. The active segment page chain is the data page sequence structure currently in a continuous write state, formed by connecting multiple fixed-length pages corresponding to the same append number in the order of page number.

[0047] To further explain, a definite mapping relationship is established between append numbers and page numbers: the same append number corresponds to one industrial data object, which is divided into multiple fixed-length pages. Each fixed-length page carries the same append number and its sequential position within the industrial data object is indicated by its page number, thus forming a mapping relationship where one append number corresponds to multiple page numbers. During consistency verification and segment sealing, fixed-length pages are used as the smallest verification unit. When a page number is missing, a write failure occurs, or the three-node records are inconsistent in a fixed-length page corresponding to the same append number, starting from the page number where the anomaly occurs, the fixed-length pages corresponding to the page number where the anomaly occurs and subsequent page numbers are determined to be invalid pages; fixed-length pages before the page number where the anomaly occurs remain valid and participate in the construction of the active segment page chain and the generation of the sealed segment.

[0048] When determining the position of the last valid append number, a joint determination is made based on the append number order and the page number order within the append number: if there are non-contiguous page numbers or incomplete writing in a certain append number, the position corresponding to the current append number is not considered a valid append number position, and the append number before the current append number that meets the conditions of continuous page numbers and complete writing is taken as the last valid append number position, thereby ensuring the continuity of the append number order in the sealed segment and the integrity of the page structure.

[0049] It should also be noted that this invention repeatedly emphasizes the collaborative recording of data page write status by the primary write page node, mirror page node, and witness page node, and the determination of the last valid append position through consistency verification of the three-party records. To ensure the reliable operation of the mechanism in a distributed environment, this invention explicitly defines the write order and status confirmation rules among the primary write page node, mirror page node, and witness page node.

[0050] Specifically, after the industrial data object is divided into multiple fixed-length pages, each fixed-length page is written sequentially according to its page number. The industrial cloud platform first writes the fixed-length page data to the main write page node. The fixed-length page data consists of the data fragments obtained after the industrial data is divided. After the main write page node returns a confirmation of successful writing, the fixed-length page data with the same page number is written to the mirror page node. After the mirror page node completes the data writing and returns a confirmation, the industrial cloud platform generates a corresponding witness page record based on the append number and page number of the current fixed-length page, and writes the witness page record to the witness page node.

[0051] The witness page record is used to record the status information of the fixed-length page write process. The witness page record includes at least the append number, page number, primary write page node identifier, and mirror page node identifier, and further includes the data verification value of the corresponding fixed-length page. The data verification value is a digest value obtained by performing a hash calculation on the fixed-length page data content, which is used for consistency verification in the subsequent segment sealing stage. The witness page record does not store the complete data page content, but rather stores metadata information describing the data page write status.

[0052] Regarding the determination of successful write operation, this invention stipulates that: when both the main write page node and the mirror page node complete the writing of fixed-length page data, and the witness page node successfully writes the corresponding witness page record, the fixed-length page is determined to be completed and becomes a valid page; if the main write page node writes successfully but the mirror page node writes unsuccessfully, the industrial cloud platform performs a retry write process on the mirror page node until the mirror page node completes the write or reaches the preset number of retries; if both the main write page node and the mirror page node write successfully but the witness page node writes unsuccessfully, the industrial cloud platform performs a supplementary write process on the witness page record to ensure that the witness page node records complete write status information.

[0053] To further explain, the preset number of retries is a parameter pre-set by the industrial cloud platform based on stability requirements and node communication reliability, preferably 3 to 5 times. When the number of retries is 3, the write latency can be controlled while ensuring the basic write success rate. When the number of retries is 5, the mirror write success rate can be further improved, thereby achieving a balance between write latency and data reliability.

[0054] In abnormal situations, if the main write page node fails to write, the current fixed-length page write operation is judged as a failure, the current page number will not participate in the subsequent active segment page chain construction, and the industrial cloud platform will re-execute the data page write scheduling; if the main write page node writes successfully but the mirror page node is unavailable for a long time, the fixed-length page is marked as a transient page, and the transient page will not participate in the valid page sequence determination in the subsequent segment sealing stage.

[0055] Through the aforementioned write order and confirmation rules (i.e., the write success determination and abnormal situations), a three-layer write structure of "primary write page data - mirror page copy - witness page record" is formed among the primary write page node, mirror page node, and witness page node. During the segment sealing stage, the industrial cloud platform compares the page number sequence in the primary write page node, the page number sequence in the mirror page node, and the witness page record in the witness page node to determine whether the data page corresponding to each page number has been completely written. Based on this, it determines the position of the last valid append number. The page number sequence is a sequence formed by the industrial cloud platform after reading fixed-length pages from each node and organizing them according to the page number order.

[0056] When a record with a certain page number is missing or inconsistent in the main write page node, mirror page node, or witness page node, the fixed-length page after the current page number is regarded as an incomplete write page, and the end page truncation is performed during the segment sealing process to ensure the structural integrity and sequential consistency of the sealed segment. The end page truncation process is to delete or ignore the fixed-length page and witness page record corresponding to the last valid append number position.

[0057] By clearly defining the writing order, success determination rules, and exception handling strategies among the primary write page node, mirror page node, and witness page node, the three-node collaborative mechanism of this invention can operate stably in a distributed writing environment and reliably determine the effective boundary of the active segment page chain during the segment sealing stage.

[0058] S3: When the active segment page chain reaches the end of the preset time window, the segment sealing operation is triggered. The active segment page chain is checked for consistency through the main write page node, mirror page node, and witness page node, and the incomplete tail page is truncated, thereby solidifying the active segment page chain into a sealed segment and generating the corresponding sealed segment list.

[0059] S3.1: When the active segment page chain reaches the end of the preset time window, the industrial cloud platform stops writing new fixed-length pages to the active segment page chain and simultaneously transfers the active segment page chain in the main write page node, the active segment page chain in the mirror page node, and the witness page sequence in the witness page node into the segment sealing processing state. The end time of the preset time window is determined by the industrial cloud platform according to a fixed time boundary, such as once every 5 minutes or once every 15 minutes. After the preset time window ends, no new write requests belonging to the current active segment page chain will be accepted, thereby changing the active segment page chain from the continuous write state to the segment sealing and consolidation state. The active segment page chain has a clear start and end range in the segment sealing and consolidation state, and the subsequent segment sealing boundary determination process has stable input.

[0060] The industrial cloud platform reads all fixed-length page sequences from the active segment page chain from the main write page node, all fixed-length page sequences corresponding to the main write page node from the mirror page node, and all witness page sequences corresponding to all fixed-length page sequences from the witness page node. The fixed-length page sequences in the main write page node record the page number order saved by the main write page node, the fixed-length page sequences in the mirror page node record the page number order saved by the mirror page node, and the witness page sequences in the witness page node record the append number, page number, and the storage relationship corresponding to the fixed-length pages. The industrial cloud platform performs sequential sorting processing on the fixed-length page sequences in the main write page node and the mirror page node according to the append number order and, within the same append number, according to the page number order, so that the fixed-length pages corresponding to the same append number are arranged in a unified order. After the sequential sorting is completed, a segment-level page sequence is formed. The segment-level page sequence no longer maintains the original scattered page state at the time of writing, but forms a continuous page sequence that can be directly used for sealing segment boundary determination and segment structure fingerprint generation. The active segment page chain changes from a page chain form to a segment-level page sequence form, and the distributed storage center changes from a page-level writing center to a segment-level sealing center.

[0061] S3.2: Perform sequential consistency verification on the segment-level page sequence based on the appended number records in the witness page sequence. Sequential consistency verification includes checking whether the page number order in the main write page node, the page number order in the mirror page node, and the page number records in the witness page node completely correspond. The industrial cloud platform compares the page number records in the main write page node, the mirror page node, and the witness page node page by page. If the page number, appended number, and page order are consistent, the corresponding fixed-length page is deemed valid. If the page number is missing, the appended number is broken, or the page order is inconsistent, the corresponding fixed-length page is deemed invalid. The industrial cloud platform continues to perform comparisons according to the appended number order until the first location that does not meet the consistency condition is found. Before the first location that does not meet the consistency condition, the last appended number that meets the complete write condition corresponds to the last valid appended number position. The complete write condition is that all page numbers corresponding to the same appended number exist consecutively. Furthermore, the records in the main write page node, mirror page node, and witness page node that correspond to all page numbers are consistent. The position of the last valid append number corresponds to the last industrial data object that has been fully written, and does not correspond to a single fixed-length page position. When any page number in multiple fixed-length pages corresponding to the same append number is missing, the write fails, or the records of the three nodes are inconsistent, the industrial cloud platform will determine the industrial data object corresponding to the current append number as an incompletely written object. The current append number will not be counted in the last valid append number position, and the content corresponding to append numbers after the current append number will not participate in the segment sealing. The determination of the last valid append number position ensures that the active segment page chain obtains a unique valid endpoint after the time window ends. The segment sealing process no longer depends on the last page state in the main write page node, nor on the last page state in the mirror page node. Instead, it uses the append number order of the witness page sequence records as the valid boundary basis of the active segment page chain, making the segment sealing result deterministic.

[0062] S3.3: The industrial cloud platform performs end-page truncation on the segment-level page sequence based on the position of the last valid append number. End-page truncation includes: deleting all fixed-length pages in the main write page node following the position of the last valid append number; deleting all fixed-length pages in the mirror page node following the position of the last valid append number; and excluding all witness page records in the witness page node following the position of the last valid append number. When there is an industrial data object with incomplete page numbers following the last valid append number, all fixed-length pages and all witness page records corresponding to the incomplete page number are deleted. Invalid end pages are located after the position of the last valid append number and do not participate in the generation of the sealed segment. After end-page truncation, the industrial cloud platform determines the sealed segment boundary according to the continuous append number interval. The starting position of the continuous append number interval is the first valid append number position within the current time window, and the ending position is the last valid append number position. Missing page numbers and broken append numbers are not allowed in the middle of the continuous append number interval. The industrial cloud platform performs segment-level truncation based on the continuous append number interval. Page sequences undergo boundary closure processing, which involves retaining only fixed-length pages within consecutive append-number intervals. These fixed-length pages are then reorganized into a continuous page sequence according to their append-number and page number order. Only fixed-length pages corresponding to complete industrial data objects are retained within these consecutive append-number intervals; industrial data objects with incomplete page numbers are excluded. The resulting segment-level page sequence forms a sealed segment, a read-only data segment converted from the active segment page chain. A sealed segment is formed when the active segment page chain meets the sealing conditions and completes consistency verification. After generation, the sealed segment maintains a fixed structure and no longer receives new data writes. Structurally, the sealed segment is no longer a continuously growing active segment page chain, but rather an independent storage center with a fixed start append-number, a fixed end append-number, and a fixed number of fixed-length pages. Subsequent industrial traceability index creation and segment-level synchronization operations on the industrial cloud platform are directly executed on the sealed segment. The sealed segment serves as the foundation for stable archiving and synchronization, preventing index drift and synchronization duplication caused by continuous changes in the active segment page chain.

[0063] S3.4: Generate a segment structure fingerprint based on the append number range, the number of fixed-length pages, and the node identifiers of the main write page node and the mirror page node of the sealed segment. The segment structure fingerprint is used to represent the structural state of the sealed segment. The generation process of the segment structure fingerprint includes reading the start append number of the sealed segment, reading the end append number of the sealed segment, reading the number of fixed-length pages corresponding to the sealed segment, reading the node identifier information of the main write page, and reading the node identifier information of the mirror page. The start append number of the sealed segment, the end append number of the sealed segment, the number of fixed-length pages corresponding to the sealed segment, the node identifier information of the main write page, and the node identifier information of the mirror page are then concatenated in a fixed order to form a structure encoding string. At the same time, the data verification values ​​of all fixed-length pages in the sealed segment are concatenated in the order of append number and page number to form a content verification string. The structure encoding string and the content verification string are then used together in a hash calculation to generate the segment structure fingerprint. The segment structure fingerprint is a unique structure identifier obtained by performing a digest calculation on the structural information of the sealed segment. It is used to verify the consistency of the sealed segment structure and the consistency of the data content of the fixed-length pages in the sealed segment during data synchronization or recovery. The industrial cloud platform writes the segment structure fingerprint, append number range, and fixed-length page count into the sealed segment list. The sealed segment list is also stored in a one-to-one correspondence with the sealed segments. The sealed segment list is a collection of metadata that records the sealed segment structure information. The sealed segment list includes at least the start append number, end append number, fixed-length page count, and corresponding storage node identifier information of the sealed segment.

[0064] After the preset time window ends, the active segment page chain is jointly organized, jointly verified, jointly truncated, and written to the sealed segment list through the joint organization, joint verification, joint truncation, and segment structure fingerprinting of the main write page node, mirror page node, and witness page node, forming sealed segments and a sealed segment list. The sealed segments have clear boundaries, and the sealed segment list has clear structural identifiers. The sealed segments and the sealed segment list together form the basis for organizing the subsequent stable sealed segment set. The active segment page chain completes the sealing of segments through continuous intervals of append numbers, and the sealed segments complete the structural solidification through segment structure fingerprinting. The distributed storage process changes from a continuous writing process to a segment-level archiving process. The focus of distributed storage optimization of the industrial cloud platform is upgraded from page-level write control to segment-level structure control. The sealed segments can directly support the continuous expansion of industrial traceability indexes and segment-level synchronization operations.

[0065] Preferably, compared to existing distributed storage methods that use page-level data as the basic management center, this invention enables the industrial cloud platform to form a stable data segment structure during continuous data writing by sequentially writing and structuring industrial data objects. After a preset time window, the active segment page chain is solidified into a sealed segment with clear boundaries, thereby avoiding structural instability caused by unclear data boundaries during continuous writing. At the same time, by generating segment structure fingerprints and recording them in the sealed segment list, the sealed segments have a unified structural identification capability at the formation stage, enabling rapid structural consistency determination during subsequent stable sealed segment collection, industrial traceability index establishment, and segment-level synchronization. Through the collaborative organization of sealed segments and the sealed segment list, the distributed storage structure of the industrial cloud platform is transformed from a page-level data management method to a segment-level structure management method, thereby improving data archiving stability, structural traceability, and cross-node synchronization consistency.

[0066] S4: Establish an industrial traceability index based on the sealed segments and the sealed segment list to form a stable sealed segment set.

[0067] S4.1: The industrial cloud platform reads the record information in the sealed segments and the sealed segment list, and extracts the industrial domain code, append number range, and fixed-length page number from the sealed segments. At the same time, it reads the segment structure fingerprint information from the sealed segment list. The sealed segments are grouped and sorted according to the industrial domain code, and sealed segments with the same industrial domain code are grouped into the same group. Within each group, the sealed segments are arranged in the order of the append number range, thus forming a sealed segment group sequence. Each sealed segment in the sealed segment group sequence retains the segment structure fingerprint record in the corresponding sealed segment list.

[0068] S4.2: The industrial cloud platform performs sequential continuity verification based on the append number interval in the grouping sequence of the sealed segments. Sequential continuity verification is completed by reading the start append number and end append number of each sealed segment. The industrial cloud platform compares the start append number of the current sealed segment with the end append number of the previous sealed segment. When the start append number and the end append number of the previous sealed segment satisfy the sequential increasing relationship, the sealed segment is determined to be sequentially continuous. When there is a break or repetition between the start append number and the end append number of the previous sealed segment, the sealed segment is determined to be sequentially discontinuous.

[0069] The sealed segment is subjected to structural consistency verification by using the segment structure fingerprint and fixed-length page number information in the sealed segment list. The structural consistency verification is performed by rereading the append number range, fixed-length page number, and node identifier information of the main write page node and mirror page node of the sealed segment, and reconstructing the structure encoding string according to the fixed arrangement used when the segment structure fingerprint was generated. The structure encoding string is then hash-encoded to generate the verification structure fingerprint. The data verification values ​​of the fixed-length pages in the sealed segment are recalculated and concatenated to generate the verification content digest. When the verification structure fingerprint is consistent with the segment structure fingerprint in the sealed segment list, and the verification content digest is consistent with the content verification information recorded in the sealed segment list, the sealed segment structure and data content are determined to be consistent.

[0070] S4.3: The industrial cloud platform filters valid sealed segments based on the results of sequential continuity verification and structural consistency verification. Sealed segments that pass both sequential continuity and structural consistency verification are identified as valid sealed segments. The industrial cloud platform establishes an industrial traceability index based on the industrial domain code, append number range, and segment structure fingerprint of the valid sealed segments. Specifically, the industrial cloud platform uses the industrial domain code as the index classification key, indexes and registers valid sealed segments with the same industrial domain code, and establishes an index mapping relationship between the append number range of the sealed segment and the corresponding segment structure fingerprint. The industrial traceability index records at least the industrial domain code, the starting append number of the sealed segment, the ending append number of the sealed segment, and the segment structure fingerprint information. When an industrial data traceability operation needs to be performed, the industrial cloud platform performs a search process in the industrial traceability index based on the target industrial domain code and locates the corresponding sealed segment based on the append number range, thereby determining the target sealed segment. The industrial cloud platform performs set processing on all valid archived segments according to the order of the append number intervals. Valid archived segments with consecutive append number intervals are arranged in order to form a stable archived segment set. The order is sorted in ascending order according to the starting append number of the archived segments. Each archived segment in the stable archived segment set maintains a one-to-one correspondence with the segment structure fingerprint record in the archived segment list, thereby providing a stable data set for subsequent archived segment synchronization operations.

[0071] To further explain, the stable archive segment set is the set of archive segments that have completed consistency verification and whose structure has been confirmed to be stable. Each archive segment in the stable archive segment set is consistent with the segment structure fingerprint record in the corresponding archive segment list.

[0072] S5: Perform segment-level synchronization operation on the stable archived segment set, send the archived segments and the archived segment list to the central cloud storage node for archived segment reconstruction and return a synchronization confirmation mark, thus completing the stable archiving of the industrial cloud platform's distributed storage structure.

[0073] S5.1: The industrial cloud platform reads all archived segment records from the stable archived segment set and simultaneously reads the archived segment list information corresponding to each archived segment; it extracts the append number range, fixed-length page number, and segment structure fingerprint of each archived segment, and performs sequential sorting processing on the archived segments according to the starting append number order of the append number range, so that the archived segments with smaller append number ranges are arranged first, and the archived segments with larger append number ranges are arranged last; the industrial cloud platform organizes the sequentially sorted archived segments and the corresponding archived segment list in sequence to form an archived segment synchronization sequence, and each archived segment in the archived segment synchronization sequence maintains a one-to-one correspondence with the archived segment list.

[0074] S5.2: The sealed segments and the sealed segment list are sent to the central cloud storage node according to the sequence of the sealed segments synchronization. The content sent includes all fixed-length page data in the sealed segments, append number interval information, fixed-length page quantity information, and segment structure fingerprint records. The central cloud storage node receives the sealed segments and the sealed segment list from the sealed segment synchronization sequence, and performs structural reconstruction processing on the sealed segments according to the append number interval, fixed-length page quantity, and node identification information in the sealed segment list. The structural reconstruction processing reorganizes the fixed-length page sequence in the sealed segments according to the append number interval order, so that the fixed-length page sequence is arranged continuously according to the append number order, and confirms that the fixed-length page quantity in the sealed segments is consistent with the record in the sealed segment list according to the fixed-length page quantity information, thereby completing the structural reconstruction of the sealed segments.

[0075] S5.3: The central cloud storage node reads the segment structure fingerprint from the archived segment list and extracts the append number range, fixed-length page count, and node identifier information from the reconstructed archived segment. It then reassembles the structure encoding string according to the fixed arrangement used when generating the segment structure fingerprint and performs hash encoding on the string to generate a verification structure fingerprint. The central cloud storage node compares the verification structure fingerprint with each segment structure fingerprint in the archived segment list. When the verification structure fingerprint completely matches the segment structure fingerprint recorded in the archived segment list, the archived segment structure is considered consistent. After the structure consistency verification is completed, a synchronization confirmation flag is generated and sent to the target storage domain, thereby completing the segment-level synchronization processing of the archived segment and achieving stable archiving of the distributed storage structure of the industrial cloud platform. The synchronization confirmation flag is the confirmation information generated by the central cloud storage node after completing the data reception and structure consistency verification of the archived segment. The synchronization confirmation flag includes at least the archived segment identifier, the segment structure fingerprint, and the synchronization completion timestamp.

[0076] This embodiment also provides a computer device applicable to the cloud platform distributed storage active segment sealing optimization method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the cloud platform distributed storage active segment sealing optimization method proposed in the above embodiment.

[0077] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0078] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the cloud platform distributed storage active segment sealing optimization method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0079] In summary, this invention effectively avoids data inconsistencies and incomplete data structures caused by node write delays or incomplete tail data writing during multi-node writing by performing segment sealing processing on the active segment page chain at the end of a preset time window and performing consistency checks and tail page truncation on the data written by multiple nodes. This ensures that the data structure remains stable and complete during the writing phase. Furthermore, by solidifying the continuously written data page chain into data sealing segments with clear boundaries and uniformly identifying them using a sealing segment list, the storage structure, which was originally prone to forming discrete data blocks, is transformed into a structured data organization method with clear segment boundaries. This improves data structure stability and data organization order. Finally, by performing consistency checks on the sealing segment structure information and fixed-length page data content, it further improves data retrieval accuracy, data traceability reliability, and data archiving efficiency and overall management capabilities in a distributed storage environment.

[0080] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for optimizing the sealing of active segments in distributed storage on a cloud platform, characterized in that: include, The industrial cloud platform receives industrial data objects sent by industrial equipment, parses them to generate industrial domain codes, determines the target storage domain based on the industrial domain codes, and simultaneously requests an append number from the sequence controller of the target storage domain to form a write ticket. By writing the ticket, the main write page node, mirror page node, and witness page node are selected in the target storage domain. The industrial data object is then divided into fixed-length pages according to the append number order and written to the main write page node and mirror page node. At the same time, the witness page containing the append number is written to the witness page node, forming an active segment page chain. When the active segment page chain reaches the end of the preset time window, the segment sealing operation is triggered. The active segment page chain is subjected to consistency verification through the main write page node, mirror page node, and witness page node, and the incomplete tail page is truncated, thereby solidifying the active segment page chain into a sealed segment and generating the corresponding sealed segment list. An industrial traceability index is established based on the sealed segments and the sealed segment list to form a stable set of sealed segments; Perform segment-level synchronization operations on the stable archived segment set, send the archived segments and the list of archived segments to the central cloud storage node for archived segment reconstruction and return a synchronization confirmation mark, thus completing the stable archiving of the industrial cloud platform's distributed storage structure.

2. The cloud platform distributed storage active segment sealing optimization method as described in claim 1, characterized in that: The industrial domain code is generated by extracting factory identifiers, workshop identifiers, production line identifiers, and process identifiers from industrial data objects and combining them according to preset mapping rules.

3. The cloud platform distributed storage active segment sealing optimization method as described in claim 1, characterized in that: The process of forming and writing the receipt specifically involves: Extract the industrial domain code and timestamp from the industrial data object, and submit a write request to the sequence controller of the target storage domain. The sequence controller generates a unique append number based on the current write order. The appended number is combined and encapsulated with the industrial domain code and timestamp to form a write ticket.

4. The cloud platform distributed storage active segment sealing optimization method as described in claim 1, characterized in that: The selection of the primary write page node, mirror page node, and witness page node refers to performing node mapping calculations in the node list of the target storage domain based on the append number written in the ticket, and determining the primary write page node, mirror page node, and witness page node respectively based on the mapping results.

5. The cloud platform distributed storage active segment sealing optimization method as described in claim 1, characterized in that: The formation of the active segment page chain specifically includes: Obtain the append number from the written ticket and read the content of the industrial data object. Perform sequential segmentation of the industrial data object according to the preset page length to generate multiple fixed-length pages. The fixed-length pages are sequentially numbered according to the append number and written to the main write page node and the mirror page node in turn. A witness page is generated based on the append number and the fixed-length page number, and the witness page is written into the witness page node. At the same time, a sequential connection relationship is established between the fixed-length pages according to the append number order to form an active segment page chain.

6. The cloud platform distributed storage active segment sealing optimization method as described in claim 1, characterized in that: The step of solidifying the active segment page chain into a sealed segment and generating a corresponding sealed segment list specifically involves: Read fixed-length pages from the active segment page chain from the main write page node, read corresponding fixed-length pages from the mirror page node, read witness pages from the witness page node, and sort the fixed-length page sequence according to the append number order to form a segment-level page sequence; Perform sequential consistency checks on the segment-level page sequence based on the append number records in the witness page sequence, and determine the position of the last valid append number in the active segment page chain during the check process; The segment-level page sequence is truncated by the last valid append number position, and the boundaries of the sealed segment are determined according to the continuous interval of the append number to generate the sealed segment. A segment structure fingerprint is generated based on the append number range, the number of fixed-length pages, and the node identifier information of the main write page node and the mirror page node of the sealed segment. The segment structure fingerprint, append number range, and number of fixed-length pages are then written into the sealed segment list.

7. The cloud platform distributed storage active segment sealing optimization method as described in claim 1, characterized in that: The formation of a stable set of archived segments specifically involves: The sealed segments are grouped and organized according to the industrial domain code to form a sealed segment group sequence; The sequential continuity of the sealed segments is checked based on the append number interval in the sealed segment grouping sequence, and the structural consistency of the sealed segments is checked based on the segment structure fingerprint and fixed-length page number information in the sealed segment list. Valid sealed segments are selected based on the sequential continuity check results and the structural consistency check results, and an industrial traceability index is established based on the industrial domain code, append number range, and segment structure fingerprint of the valid sealed segments. The valid sealed segments are arranged sequentially according to the interval of the appendix number to form a stable set of sealed segments.

8. The cloud platform distributed storage active segment sealing optimization method as described in claim 1, characterized in that: The return of the synchronization confirmation flag is specifically as follows: Organize the sealed segments and their corresponding lists in the stable sealed segment set into a sealed segment synchronization sequence according to the order of the appended number intervals; The archived segments and the list of archived segments in the synchronization sequence are sent to the central cloud storage node. The central cloud storage node performs archived segment structure reconstruction processing based on the append number range, fixed-length page number and node identification information in the archived segment list. Based on the segment structure fingerprint in the archived segment list, perform a structure consistency check on the reconstructed archived segments, and generate a synchronization confirmation flag upon successful verification, then return to the target storage domain.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the cloud platform distributed storage active segment sealing optimization method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the cloud platform distributed storage active segment sealing optimization method according to any one of claims 1 to 8.

Images