Unstructured data storage method and system based on distributed architecture

Abstract

The embodiment of the application provides a kind of unstructured data storage method and system based on distributed architecture, it is related to data storage technical field, with data acquisition terminal application customization appeal as core, constructs terminal perception information set, accordingly generates distributed node capability dynamic information, node capability adaptation matrix and terminal and node appeal response link, executes terminal identification fragmentation to unstructured data, constructs cross node fragmentation identification association link, obtains fragmentation identification storage set and association link log, monitors appeal change, generates appeal-driven storage optimization scheme and optimization track, constructs storage architecture operation and maintenance closed loop according to optimization scheme, exports terminal adaptive unstructured data storage architecture and operation and maintenance log, this architecture includes fragmentation identification storage location, terminal corresponding rule and node coordination rule, operation and maintenance log records whole process details, improve the performance, reliability and adaptability of unstructured data storage.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and more specifically, to a method and system for storing unstructured data based on a distributed architecture. Background Technology

[0002] In today's digital age, unstructured data is experiencing explosive growth, encompassing various forms such as images, videos, audio, and text, and is widely used across various industries. Traditional unstructured data storage methods are primarily based on centralized architectures, storing all data centrally in a single server or data center. While this storage approach was effective in the early stages when data volumes were relatively small and business scenarios were relatively simple, it could meet certain storage needs.

[0003] However, with the rapid increase in data volume and the increasing complexity of business scenarios, the drawbacks of centralized storage architecture have gradually become apparent. On the one hand, centralized storage faces storage capacity bottlenecks, making it difficult to meet the storage needs of massive amounts of unstructured data. On the other hand, its data processing capabilities are limited, and performance bottlenecks are prone to occur in high-concurrency access scenarios, leading to increased data access latency and affecting the normal operation of business systems. In addition, centralized storage architecture has low reliability; once the storage server fails, it may result in the loss of a large amount of data, causing huge losses to enterprises. At the same time, existing storage methods lack an effective response mechanism to the customized demands of terminal applications, and cannot dynamically adjust storage strategies according to the hardware parameters of the terminal, the scenario collection attributes, and the application functional requirements, making it difficult to achieve efficient storage and management of unstructured data. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a method and system for storing unstructured data based on a distributed architecture.

[0005] According to a first aspect of this application, a method for storing unstructured data based on a distributed architecture is provided, the method comprising: Based on the customized application requirements of data acquisition terminals, unstructured data terminal perception information is constructed. According to the original characteristics of data, terminal hardware parameters, scene acquisition attributes and application function requirements, a set of terminal perception information is generated. The set of terminal perception information includes the three-dimensional correspondence identifiers of data, terminals and applications and feature adaptation logic. Based on the terminal perception information set, a distributed node capability dynamic information is constructed. The storage requirements corresponding to the information characteristics are matched with the node hardware resource information to generate a node capability adaptation matrix and a terminal and node demand response link. Based on the node capability adaptation matrix, terminal identification fragmentation is performed on unstructured data. By combining the fragmented data with the terminal perception information identification, a cross-node fragment identification association link is constructed to obtain the fragment identification storage set and association link log. Monitor changes in terminal application customization requirements, optimize the shard identifier storage set according to the changes, adjust the node capability adaptation matrix and shard identifier association link structure accordingly, and generate a demand-driven storage optimization scheme and optimization trajectory. Based on the demand-driven storage optimization solution, a closed loop for storage architecture operation and maintenance is constructed, node capability dynamic information and shard identification association rules are updated, and terminal-adaptive unstructured data storage architecture and operation and maintenance logs are output. The terminal-adaptive unstructured data storage architecture includes shard identification storage location, terminal corresponding rules and node collaboration rules, and operation and maintenance logs record the details of the entire process of information construction, shard storage, optimization and iteration.

[0006] According to a second aspect of this application, a distributed architecture-based unstructured data storage system is provided. The distributed architecture-based unstructured data storage system includes a processor and a readable storage medium. The readable storage medium stores a program that, when executed by the processor, implements the aforementioned distributed architecture-based unstructured data storage method.

[0007] Based on any of the above aspects, a terminal-aware information set is constructed with the customized requirements of data acquisition terminal applications as the core. This set encompasses the three-dimensional correspondence and feature adaptation logic between data, terminals, and applications. By constructing distributed node capability dynamic information and generating a node capability adaptation matrix and a terminal-node requirement response link, efficient matching of storage requirements and node hardware resources is achieved, improving storage resource utilization and reducing storage costs. Unstructured data is segmented by terminal identification according to the node capability adaptation matrix, and a cross-node segmentation identification association link is constructed, making data storage more flexible and efficient, enhancing data reliability and availability. By monitoring changes in terminal application customized requirements and generating requirement-driven storage optimization schemes and optimization trajectories, it can respond to changes in business requirements in real time, dynamically adjust the storage architecture, and ensure that the storage system is always in an optimal state. Finally, the constructed storage architecture operation and maintenance closed-loop output terminal-adaptive unstructured data storage architecture and operation and maintenance logs improve the performance, reliability, and adaptability of unstructured data storage. Attached Figure Description

[0008] Figure 1 A flowchart illustrating the unstructured data storage method based on a distributed architecture provided in an embodiment of this application is shown. Figure 2 This paper illustrates a schematic diagram of the component structure of an unstructured data storage system based on a distributed architecture, as provided in an embodiment of this application. Detailed Implementation

[0009] Figure 1 A flowchart illustrating the unstructured data storage method based on a distributed architecture provided in this application embodiment is shown, and the detailed steps are described below.

[0010] Step S110: Based on the customized application requirements of the data acquisition terminal, construct unstructured data terminal perception information. According to the original characteristics of the data, terminal hardware parameters, scene acquisition attributes and application function requirements, generate a set of terminal perception information. The set of terminal perception information includes the three-dimensional correspondence identifiers of data, terminal and application and feature adaptation logic.

[0011] In the field of data processing, a large amount of unstructured data needs to be stored. This data comes from different data acquisition terminals and corresponds to different application requirements. When constructing a terminal to perceive information from unstructured data, it is first necessary to collect various types of information related to the data. The inherent characteristics of the data are its inherent attributes, such as the data format (e.g., the encoding method of text data, the pixel arrangement rules of image data), and the data source type (e.g., real-time data collected by sensors or historical archived data). Terminal hardware parameters encompass various hardware configurations of the data acquisition terminal, including processor model, memory size, storage space, and network interface type. These parameters directly affect the terminal's data acquisition and preliminary processing capabilities. Scene acquisition attributes involve the environmental conditions during data acquisition, such as network stability, electromagnetic interference in the surrounding environment, and the frequency and duration of data acquisition. Application functional requirements refer to the specific applications that the data will be used for. Different applications have different requirements for data storage, access, and processing. For example, some applications require fast data access response, while others have higher requirements for long-term data storage stability.

[0012] By integrating the original data features, terminal hardware parameters, scene acquisition attributes, and application functional requirements, a three-dimensional corresponding identifier is generated for each piece of unstructured data, containing a data identifier, a terminal identifier, and an application identifier. The feature adaptation logic determines the adaptation rules between the original data features, terminal hardware parameters, and scene acquisition attributes based on application functional requirements. For example, when the application functional requirement is real-time data processing, the data update frequency in the original data features should match the processing speed in the terminal hardware parameters, and the network stability in the scene acquisition attributes should also meet the requirements of real-time transmission. Based on the above three-dimensional corresponding identifier and feature adaptation logic, a terminal perception information set is finally generated, which completely records the relationship between data, terminals, and applications, as well as the adaptation rules between them.

[0013] Step S120: Construct distributed node capability dynamic information based on the terminal perception information set, match the storage requirements corresponding to the information features with the node hardware resource information, and generate a node capability adaptation matrix and a terminal and node demand response link.

[0014] Step S121: Extract the storage requirement dimension corresponding to each piece of data in the terminal sensing information set, covering storage capacity requirements, access response requirements, security protection requirements, fault tolerance and redundancy requirements, and life cycle requirements, to form a data storage requirement list, and mark the correspondence between each requirement dimension and the terminal sensing information identifier.

[0015] When extracting the storage requirement dimension corresponding to each piece of data from the terminal-perceived information set, a detailed analysis of each piece of data is required. Storage capacity requirement refers to the amount of space needed for data storage, which depends on the size and quantity of the data. For example, a long video clip typically requires a large storage capacity. Access response requirement reflects the required response speed when data is accessed. Data in some real-time application scenarios requires fast access response, while data in some non-real-time applications has lower response speed requirements. Security protection requirement involves ensuring the security of data during storage, including data encryption and access control. For sensitive data, the security protection requirement is higher. Fault tolerance and redundancy requirement refers to the ability of data to cope with node failures, data corruption, etc., during storage. This is usually achieved through data backup and redundant storage. Important data requires higher fault tolerance and redundancy requirements. Lifecycle requirement specifies the length of time data needs to be retained in the storage system. Different types of data have different lifecycles; after the lifecycle expires, the data will be deleted or archived.

[0016] The extracted storage requirement dimensions are matched with the data identifier, terminal identifier, and application identifier in the terminal perception information identifier. For example, if a piece of data is identified as D-001, its corresponding storage capacity requirement is C1, access response requirement is R1, security protection requirement is S1, fault tolerance and redundancy requirement is F1, and lifecycle requirement is L1, thus forming a data storage requirement list. This list clearly shows the various storage requirements of each piece of data and their correspondence with the terminal perception information identifier.

[0017] Step S122: Collect hardware resource information of all distributed nodes, including node physical storage capacity, read and write processing rate, security protection configuration, geographical location distribution, network bandwidth resources and hardware operating status, classify and record them according to the node's unique identifier, and form a node hardware resource list.

[0018] Hardware resource information is collected for all nodes in the distributed storage system. Node physical storage capacity is the actual storage space a node can provide, a fundamental indicator of its storage capability. Read / write processing speed reflects the speed at which a node performs read and write operations, directly impacting data storage and access efficiency. Security configuration includes the security hardware and software measures employed by the node, such as firewalls, intrusion detection systems, and data encryption modules. Geographical location records the node's physical location; nodes in different geographical locations may differ in data transmission latency and network stability. Network bandwidth resources represent a node's ability to transmit data with other nodes or terminals; higher bandwidth results in faster data transmission. Hardware operating status encompasses parameters such as CPU utilization, memory usage, disk I / O activity, and temperature, reflecting the node's current operating status and resource load.

[0019] Each distributed node is assigned a unique node identifier, such as N-001, N-002, etc. The collected hardware resource information is classified and recorded according to this unique identifier. For example, the physical storage capacity of node N-001 is within a specific range, the read and write processing rate is within a specific range, the security protection configuration includes a firewall and a data encryption module, the geographical location is in region A, the network bandwidth resources are within a specific range, and the CPU utilization rate in the hardware operation status is within a specific percentage range, thus forming a node hardware resource list.

[0020] Step S123: Decompose the node hardware resource list into dimensions and construct a node capability evaluation framework, covering storage capability, processing capability, transmission capability, security capability and stability capability. Each dimension, combined with specific hardware resource parameters, forms the basic content of node capability.

[0021] The node hardware resource inventory is broken down into dimensions to evaluate the node's capabilities from multiple perspectives. Storage capacity is primarily based on the node's physical storage capacity parameters, assessing the total amount of data the node can store, while also considering the impact of storage media type (e.g., SSD, HDD) on storage performance. Processing capacity refers to hardware resource parameters such as read / write processing speed, CPU utilization, and memory usage to determine the node's efficiency and concurrency handling capabilities for data read / write operations. Transmission capacity is based on network bandwidth resource parameters to determine the speed and stability of data transmission between the node and other nodes and terminals, while also considering the impact of network protocols on transmission capacity. Security capacity combines security protection configuration parameters, such as firewall protection level, data encryption algorithm strength, and access control policies, to evaluate the node's ability to ensure data security. Stability capacity comprehensively considers various parameters in hardware operation status, such as CPU temperature, disk failure rate, and system runtime, to analyze the node's stability and reliability during long-term operation.

[0022] By combining each capability dimension with specific hardware resource parameters, such as determining the node's storage level based on the node's physical storage capacity and storage medium type in the storage capability dimension, and classifying the processing capability level based on read / write processing speed and CPU and memory parameters in the processing capability dimension, the basic content of node capabilities is formed, which describes in detail the basic situation of each node in each capability dimension.

[0023] Step S124: Construct distributed node capability dynamic information based on the basic content of node capabilities, combine node hardware resource parameters and capability evaluation dimensions, mark the capability advantages and disadvantages of each node, form a node capability dynamic information set, and combine it with the node's unique identifier.

[0024] Building upon the fundamental capabilities of nodes, and combining real-time changes in node hardware resource parameters with capability assessment dimensions, dynamic information on distributed node capabilities is constructed. For each node, changes in its hardware resource parameters are continuously monitored, such as changes in remaining physical storage capacity, fluctuations in read / write processing rates, and real-time network bandwidth usage. Based on these changes, the node's performance in storage capacity, processing capacity, transmission capacity, security capacity, and stability capacity is dynamically evaluated, identifying each node's current strengths and weaknesses. For example, node N-003 initially has strong storage and processing capabilities (strengths) but relatively weak transmission capabilities (weaknesses). When the node's network bandwidth is improved, its transmission capability may shift from a weakness to a strength.

[0025] The aforementioned dynamic node capability information is associated with the node's unique identifier to form a dynamic node capability information set. This information set is updated in real time as the node's hardware resource parameters change, ensuring that it accurately reflects the current capability status of each node.

[0026] Step S125: Perform dimensional adaptation between the data storage requirement list and the node capability dynamic information set, match the requirement dimension of each data with the capability advantage dimension of the node, select nodes with adaptation capabilities as candidate storage nodes for the corresponding data, and mark the correspondence between candidate storage nodes and data identifiers.

[0027] The storage capacity requirements, access response requirements, security protection requirements, fault tolerance and redundancy requirements, and lifecycle requirements of each data item in the data storage requirements list are matched one by one with the capability advantages of each node in the dynamic node capability information set. For example, for data with high storage capacity requirements, nodes with advantages in storage capability are selected; for data with high access response requirements, nodes with advantages in processing and transmission capabilities are selected. Through this multi-dimensional matching, multiple nodes with suitable capabilities are selected for each data item, and these nodes are the candidate storage nodes for the corresponding data.

[0028] During the screening process, it is necessary to comprehensively consider various dimensions of data requirements to ensure that candidate storage nodes can meet the diverse storage needs of the data. Simultaneously, the correspondence between candidate storage nodes and data identifiers is marked; for example, candidate storage nodes for data identifier D-002 are N-004, N-005, and N-006, to facilitate further sorting and selection of candidate storage nodes.

[0029] Step S126: Combining the terminal hardware parameters and geographical location information in the terminal perception information, sort the candidate storage nodes, prioritize matching nodes that are geographically close to the terminal and have compatible hardware interfaces as the main storage nodes, and generate a node priority sorting table.

[0030] Candidate storage nodes are ranked by combining terminal hardware parameters and geographic location information from the terminal's perception information. The interface type and data transmission protocol in the terminal hardware parameters need to be compatible with the interfaces of the candidate storage nodes to ensure smooth data transmission between the terminal and the nodes. Regarding geographic location information, candidate storage nodes geographically close to the data acquisition terminal typically have lower data transmission latency, which can improve data transmission efficiency and access response speed.

[0031] During the sorting process, candidate storage nodes that are geographically close to the terminal and have compatible hardware interfaces are prioritized as the primary storage node. For example, if data acquisition terminal T-001 is located in region B and has interface type A, and candidate storage node N-004 is located in region B and has interface type A, while N-005 is located in region C and has interface type A, then N-004 will be prioritized over N-005. Based on these sorting rules, a node priority ranking table is generated for each data entry's candidate storage nodes, clearly defining the priority order of each candidate storage node.

[0032] Step S127: Construct a basic link for terminal and node demand response, connect the main storage node with the corresponding data acquisition terminal, configure the link transmission parameters according to the terminal data transmission characteristics and node network bandwidth capabilities, and mark the correspondence between the link and the terminal and node.

[0033] Based on the node priority ranking table, the candidate storage node with the highest priority is selected as the primary storage node. Then, a basic link connecting the data acquisition terminal and the primary storage node for request response is constructed. When constructing the link, the data transmission characteristics of the terminal need to be considered, such as the data transmission format, packet size, and transmission frequency, as well as the network bandwidth capacity of the primary storage node. Based on these factors, the link's transmission parameters are configured, including the transmission protocol, data transmission rate limits, packet fragment size, and retransmission mechanism, to ensure stable and efficient data transmission over the link.

[0034] At the same time, a unique link identifier is assigned to the basic link for responding to the request, and the correspondence between the link identifier and the data acquisition terminal identifier and the main storage node identifier is marked. For example, the link identifier L-001 corresponds to the terminal T-001 and the main storage node N-004, so as to clearly record the association between the link and the terminal and node.

[0035] Step S128: Extend the request response link to the candidate storage node, construct a full-link response structure between the terminal, the master node, and the candidate node, and set the link data flow logic according to the access response requirements in the data storage requirement list. The link data flow logic includes data synchronization direction, response timing, and load distribution method to match the access response requirements in the data storage requirement list.

[0036] Step S1281: Collect the communication interface parameters, protocol type and transmission capabilities of the primary storage node and the candidate storage node, determine the range of transmission rates, data encapsulation formats and connection stability that can be supported between nodes, form a list of communication capabilities between nodes, and combine it with the unique identifier of the node.

[0037] The communication capabilities between the primary storage node and all candidate storage nodes are collected. Communication interface parameters include the interface type, number, and data transmission pin definitions; protocol type refers to the network protocol used for data communication between nodes, such as TCP / IP or UDP; transmission capability reflects performance indicators such as the maximum data transmission rate, transmission latency, and packet loss rate between nodes. By collecting this information, the range of supported transmission rates between nodes is determined, i.e., the highest and lowest data transmission speeds achievable between nodes; data encapsulation format refers to the packaging method and format specifications of data during transmission; and connection stability refers to the duration and interruption frequency of the communication connection between nodes. This collected inter-node communication capability information is combined with the node's unique identifier. For example, the communication interface parameters between primary storage node N-004 and candidate storage node N-005 are interface parameter A, protocol type is protocol type B, transmission rate range is rate range C, data encapsulation format is format D, and connection stability is stability level E, forming a list of inter-node communication capabilities.

[0038] Step S1282: Based on the configuration parameters of the basic link for terminal and node demand response, extend the link to each candidate storage node, construct a hierarchical full-link response structure, assign a unique identifier to each extended link, and combine the extended link identifier with the identifiers of the main storage node, candidate storage node, and terminal.

[0039] Based on the configuration parameters of the basic link for responding to requests from the terminal and the main storage node, such as transmission protocols and transmission rate limits, the link is extended to each candidate storage node. During the extension process, a hierarchical full-link response structure is constructed according to factors such as the location relationship between the main storage node and the candidate storage nodes and the network topology. For example, the main storage node serves as the core layer, and the candidate storage nodes serve as the edge layer, forming a tree-like or star-like link structure.

[0040] Each extended link is assigned a unique extended link identifier, such as L-001-1, L-001-2, etc., where L-001 is the basic link identifier, and the following numbers are the extended link sequence number. Simultaneously, the extended link identifier is combined with the primary storage node identifier, candidate storage node identifier, and terminal identifier. For example, extended link identifier L-001-1 corresponds to primary storage node N-004, candidate storage node N-005, and terminal T-001, thereby clarifying the position and association of each extended link in the overall link response structure.

[0041] Step S1283: Based on the access response requirements in the data storage requirement list, set the data synchronization direction in the link data flow logic, adopt the flow mode of one-way synchronization of core shard data from the primary storage node to the candidate storage node, and two-way feedback of storage status and load information from the candidate storage node to the primary storage node, and mark the flow rules.

[0042] Based on the data access response requirements in the data storage requirement list, the data synchronization direction in the link data flow logic is set. For core shard data, to ensure data consistency and the dominant position of the primary storage node, a one-way synchronization mode from the primary storage node to the candidate storage node is adopted. That is, the primary storage node actively sends core shard data to the candidate storage node, and the candidate storage node receives and stores this data without sending core shard data back to the primary storage node. For storage status and load information, to enable the primary storage node to understand the operation status of the candidate storage nodes in a timely manner, a two-way feedback mode from the candidate storage node to the primary storage node is adopted. That is, the candidate storage node periodically sends its own storage status (such as used storage space, remaining storage space, etc.) and load information (such as CPU utilization, memory usage, etc.) to the primary storage node, and the primary storage node can also actively request this information from the candidate storage nodes.

[0043] The flow rules that indicate the direction of data synchronization are defined. For example, the core shard data synchronization rule is "primary storage node - candidate storage node, real-time synchronization", and the storage status and load information feedback rule is "candidate storage node - primary storage node, periodic feedback (such as once every 5 minutes) and on-demand feedback", to ensure that the direction and method of data flow are clear and unambiguous.

[0044] Step S1284: Set the link response timing, set differentiated synchronization timing according to the access frequency of different data shards, configure the corresponding synchronization interval for the corresponding access frequency range, and match the response time requirements in the data storage requirement list.

[0045] Based on the access frequency of different data shards in the data storage requirement list, differentiated link response timings are set. Data shards with high access frequency require more frequent synchronization to ensure that the data in the candidate storage nodes remains highly consistent with the primary storage node, meeting the requirements for fast access; for data shards with low access frequency, the synchronization interval can be appropriately extended to reduce network transmission load and node processing pressure.

[0046] Configure corresponding synchronization intervals for different access frequency ranges. For example, data shards with an access frequency range of "extremely high" (e.g., multiple accesses per second) should be configured with a synchronization interval of "real-time synchronization"; data shards with an access frequency range of "high" (e.g., multiple accesses per minute) should be configured with a synchronization interval of "once every 1 minute"; data shards with an access frequency range of "medium" (e.g., multiple accesses per hour) should be configured with a synchronization interval of "once every 10 minutes"; and data shards with an access frequency range of "low" (e.g., several accesses per day) should be configured with a synchronization interval of "once every 1 hour". These differentiated synchronization timing settings match the response time requirements in the data storage needs list, ensuring that data is available promptly when needed.

[0047] Step S1285: Formulate a link load allocation method. Based on the processing capacity and network bandwidth resources in the node capability dynamic information, allocate the data transmission load of each link and formulate a load allocation strategy. When the link load reaches a preset threshold, some data transmission tasks will be reallocated to preset low-load links.

[0048] Based on the dynamic information of node capabilities, including processing power (such as CPU processing speed and memory processing capacity) and network bandwidth resources, a link load allocation method is determined. Nodes with strong processing power and sufficient network bandwidth resources can handle more data transmission load, while nodes with weaker processing power and limited network bandwidth resources are allocated less load. During the allocation process, the link transmission capacity between the primary storage node and each candidate storage node needs to be comprehensively considered to ensure load balance across links and prevent data transmission delays or failures due to excessive load on certain links.

[0049] Develop a load balancing strategy, clearly defining the initial load allocation ratio for each link. Simultaneously, set preset thresholds for link load. When the load on a link (e.g., data transmission rate, number of data packets) reaches this threshold, a load redistribution mechanism is activated, transferring some data transmission tasks from the high-load link to a preset low-load link. For example, when link L-001-1 reaches 80% of its maximum capacity, some low-priority data fragments originally transmitted through this link are redistributed to link L-001-2, whose load rate is below 50%, to achieve dynamic load balancing.

[0050] Step S1286: Configure link fault-tolerant flow rules, set flow adjustment rules when link failure, node abnormality and overload occur, determine the conditions for enabling backup links, data switching sequence and fault recovery process, with the goal of achieving data flow continuity and response stability, and configure link fault-tolerant flow rules.

[0051] To ensure the continuity and stability of data flow across the link, fault-tolerant flow rules are configured. For link failure scenarios, such as link interruption or data transmission errors, flow adjustment rules are set. When a link failure is detected, a backup link is immediately activated for data transmission. The conditions for activating the backup link can be that the link failure lasts for more than a preset time (e.g., 3 seconds) or that a preset number of transmission errors occur consecutively (e.g., 5 times). The data switching sequence specifies the time interval and order for switching from the failed link to the backup link to minimize the impact of data transmission interruptions.

[0052] For node anomalies, such as candidate storage nodes crashing or becoming unresponsive, rules are set to transfer data transmission tasks on that node to other normal candidate storage nodes. The criteria for determining node anomalies are established; for example, if no response information is received from a node within a preset time (e.g., 10 seconds), it is considered an abnormal node. The load adjustment rules for overload conditions are aligned with the load distribution strategy in step S1285. When the link load exceeds the limit and cannot be resolved through load redistribution, corresponding fault-tolerance measures are activated, such as temporarily restricting the transmission of low-priority data and prioritizing the transmission of high-priority data.

[0053] The fault recovery process specifies how data transmission should switch back from the backup link or other node to the original link or node after the failed link or node recovers, as well as the methods for data synchronization and consistency verification. By configuring these link fault-tolerant flow rules, it is ensured that data flow can continue under various abnormal conditions, maintaining response stability.

[0054] Step S1287: Combine the link data flow logic with the node capability adaptation matrix to ensure that the synchronization direction, response timing and load distribution method are consistent with the data and node capability adaptation dimensions. When setting the link data flow logic, make it consistent with the requirements in the data storage requirement list.

[0055] The established link data flow logic (including data synchronization direction, response timing, load distribution method, fault-tolerant flow rules, etc.) is validated by combining it with the node capability adaptation matrix. The node capability adaptation matrix records the capability adaptation relationship between data and nodes, such as the storage capacity and processing capacity advantages of a node for a certain data. The validation process checks whether the synchronization direction in the link data flow logic matches the data storage requirements and node capabilities. For example, whether data requiring rapid synchronization has been configured with an appropriate synchronization direction and link; whether the response timing meets the data access frequency and response time requirements, and whether it is adapted to the node's processing capacity; and whether the load distribution method considers the node's network bandwidth resources and processing capacity to avoid the node's capabilities being hampered by excessive load.

[0056] Simultaneously, ensure that the data flow logic of the data link is consistent with all requirements in the data storage needs list (such as storage capacity requirements, security protection requirements, etc.). For example, for data with high security protection requirements, encrypted transmission should be used in terms of data synchronization direction and data encapsulation format, and the security and reliability of its transmission should be prioritized in terms of response timing. Through the above combined verification, ensure that the data flow logic of the data link can meet all data storage requirements and is compatible with the node capabilities.

[0057] Step S1288: Construct a link status monitoring module to collect real-time data on the transmission rate, load, connection status, and fault information of each link, combined with the unique link identifier and dynamic node capability information.

[0058] Construct a dedicated link status monitoring module to collect real-time status information of each link in the full-link response structure. The status information includes transmission rate, i.e., the data transmission speed of the current link; load status, such as the bandwidth utilization of the link and the transmission queue length; connection status, such as whether the link is connected and whether the connection is stable; and fault information, such as the type of error that occurred, the time of the error, and the number of errors.

[0059] The collected link status information is associated with unique link identifiers to accurately identify the status of each link. Simultaneously, by combining dynamic node capability information, the relationship between link status and node capability is analyzed. For example, when the processing capacity of a node decreases, the transmission rate and load of its related links may be affected. The link status monitoring module provides the real-time collected data to the load distribution adjustment mechanism and fault tolerance mechanism as a basis for their decision-making. For instance, when a link is detected to be overloaded, the load distribution adjustment mechanism can reallocate the load based on this data; when a link failure is detected, the fault tolerance mechanism can activate corresponding fault tolerance measures based on the fault information.

[0060] Step S1289: Simulate the link operation status under different access scenarios, verify the feasibility of data flow logic, response timing and load distribution method, adjust link parameters based on monitoring data, and optimize the adaptability of the entire link response structure.

[0061] The feasibility of the end-to-end response structure was verified by simulating different access scenarios, including high-concurrency access scenarios (a large number of users accessing data simultaneously), burst access scenarios (a large number of data access requests occurring in a short period of time), and low-access scenarios. During the simulation, the operational status of the link was observed according to the set data flow logic, response timing, and load distribution method, such as data transmission latency, throughput, packet loss rate, node load changes, and link fault tolerance.

[0062] By combining monitoring data collected by the link status monitoring module, we analyze whether the link operation status meets expectations under different access scenarios, and whether the data flow logic, response timing, and load distribution methods can meet the requirements of the data storage needs list. If problems are found, such as excessive link transmission latency or unbalanced load distribution during high-concurrency access, we adjust the link parameters based on the monitoring data, such as optimizing the synchronization interval, adjusting the load distribution ratio, and modifying the conditions for enabling backup links. Through multiple simulations and parameter adjustments, we continuously optimize the adaptability of the entire link response structure to ensure its stable and reliable operation in practical applications.

[0063] Step S12810: Output the configured full-link response structure of the terminal, master node and candidate nodes, and simultaneously organize the link identifier, correspondence, data flow logic and monitoring module configuration information to supplement the node capability adaptation matrix.

[0064] After configuring, verifying, and optimizing the end-to-end response structure, the configured end-to-end response structure between the terminal, main storage node, and candidate storage nodes is output. This structure includes the identifiers of all links, the hierarchical relationships between links, and the correspondence between links and terminals and nodes. Simultaneously, a link identifier list is compiled, clarifying the unique identifier and meaning of each link; a link correspondence table records the associations between links and terminals, main storage nodes, and candidate storage nodes; a data flow logic document details the data synchronization direction, response sequence, load distribution method, fault-tolerant flow rules, etc.; and monitoring module configuration information includes monitoring indicators, collection frequency, and data storage location. This compiled information is then added to the node capability adaptation matrix, ensuring that the matrix includes not only the capability adaptation relationship between data and nodes but also the link information of data flow between nodes.

[0065] Step S129: Based on the node capability dynamic information set, node priority ranking table and demand response link configuration, construct the node capability adaptation matrix, determine the data and node adaptation relationship, capability matching dimension and link response rules, label the matrix adaptation logic, output the node capability adaptation matrix and terminal and node demand response link, and simultaneously organize the node capability dynamic information set, candidate storage node list and link configuration information.

[0066] A node capability adaptation matrix is ​​constructed by integrating the dynamic information set of integrated node capabilities, the node priority ranking table, and the configuration of request response links (including the configuration of basic links and extended links). In the matrix, rows represent data identifiers, columns represent node identifiers, and matrix elements represent the adaptation relationship between data and nodes, such as "adapted," "partially adapted," and "not adapted." The matrix also indicates the capability matching dimensions between data and nodes, i.e., which data requirement dimensions match which node advantage capability dimensions. For example, the adaptation of data D-003 with node N-007 is based on the matching of storage capacity and security capability dimensions. Link response rules are also included in the matrix, explaining the request response link and related response rules used when data is stored on that node. For example, when data D-003 is stored on node N-007, link L-002 is used, and the response timing is "medium access frequency, synchronization once every 10 minutes."

[0067] The annotation matrix adaptation logic is the basis and method for determining the data and node adaptation relationship, capability matching dimension, and link response rules. For example, "the adaptation relationship is determined by multi-dimensional matching of the data storage requirement list and the node capability dynamic information set, combined with the node priority ranking table; the capability matching dimension is the intersection of the data requirement dimension and the node advantage dimension; the link response rules are determined based on the data access frequency and the communication capability between nodes."

[0068] Output the completed node capability adaptation matrix and the terminal and node request response link. At the same time, organize the node capability dynamic information set, candidate storage node list (including candidate storage node identifier, correspondence with data identifier, etc.) and link configuration information (including link identifier, correspondence, data flow logic, etc.) to form a complete node capability adaptation and link configuration document.

[0069] Step S130: Perform terminal identification sharding on unstructured data according to the node capability adaptation matrix, combine the sharded data with the terminal perception information identification, construct cross-node shard identification association links, and obtain the shard identification storage set and association link logs.

[0070] Step S131: Based on the original data features and volume parameters in the terminal perception information set, perform terminal identification fragmentation processing on the unstructured data, splitting it into multiple data fragments. Each data fragment carries the terminal perception identification fragment of the corresponding data, and marks the fragment's unique identifier, the data to which it belongs, and the identification fragment information.

[0071] Based on the inherent characteristics (such as data format, data type, and data content characteristics) and volume parameters (such as data size and data length) of the data in the terminal-perceived information set, unstructured data is segmented using terminal identification. The inherent characteristics of the data determine the segmentation method and basis; for example, video data can be segmented according to time segments, while text data can be segmented according to chapters or paragraphs. Volume parameters determine the size of each data segment; typically, data is split into multiple relatively uniformly sized segments to facilitate storage and transmission.

[0072] During the data sharding process, each data shard carries a terminal-aware identifier segment corresponding to the data. This identifier segment is a piece of information related to the shard extracted from the three-dimensional corresponding identifier of the data. For example, the terminal-aware identifier of data identifier D-004 is "D-004-T-002-A-001" (where T-002 is the terminal identifier and A-001 is the application identifier), and the identifier segment of a certain shard can be "D-004-T-002-A-001-P01" (where P01 is the shard sequence number). At the same time, each data shard is marked with a unique shard identifier (such as S-001, S-002, etc.), a home data identifier (i.e., the original data identifier to which the shard belongs, such as D-004), and identifier segment information to ensure that each data shard can be uniquely identified and associated with the original data.

[0073] Step S132: Deeply integrate each data shard with the corresponding terminal perception information identifier, and label the terminal identifier information, the data identifier of the shard, the unique identifier of the shard, and the corresponding storage requirement dimension to form a shard and information binding set, so that each data shard contains its corresponding terminal perception attribute information.

[0074] Each data shard generated in step S131 is deeply combined with its corresponding terminal perception information identifier. Terminal identification information (such as T-002) and home data identifier (such as D-004) are extracted from the terminal perception information identifier, and combined with the shard's unique identifier (such as S-001) and the storage requirement dimension corresponding to the data shard obtained from the data storage requirement list (such as storage capacity requirement C2, access response requirement R2, etc.), and jointly labeled on the data shard.

[0075] Through this deep integration, each data shard contains rich terminal-aware attribute information, forming a shard and information binding set. For example, data shard S-001 is labeled with the following information: "Terminal Identification Information: T-002; Home Data Identifier: D-004; Unique Shard Identifier: S-001; Storage Capacity Requirement: C2; Access Response Requirement: R2; Security Protection Requirement: S2; Fault Tolerance and Redundancy Requirement: F2; Lifecycle Requirement: L2". This tightly links the data shard with the terminal-aware information, facilitating subsequent storage and management.

[0076] Step S133: Based on the node capability adaptation matrix, allocate the data shards in the shard and information binding set to the corresponding main storage nodes one by one, record the storage location of the shard data, the unique identifier of the node, the content of the binding information and the capability adaptation dimension, and generate a shard allocation record table.

[0077] Based on the data-node matching relationship in the node capability adaptation matrix, each data shard in the shard and information binding set is assigned to its corresponding primary storage node. During the allocation process, the node capability adaptation matrix is ​​searched according to the data identifier to determine the primary storage node for that data, and then the data shard is assigned to that primary storage node.

[0078] The system records the storage location of each data shard, i.e., the specific storage path or address of the data shard within the primary storage node; the unique node identifier, i.e., the identifier of the primary storage node (e.g., N-008); binding information, i.e., the terminal identifier, the data owner identifier, the shard's unique identifier, and the storage requirement dimension; and the capability adaptation dimension, i.e., the capability dimension (such as storage capacity, processing capacity, etc.) that matches the storage requirements of the primary storage node with the data shard. Based on the above records, a shard allocation record table is generated, which clearly displays the allocation status and related information of each data shard.

[0079] Step S134: After receiving the data shards, the primary storage node configures the corresponding storage strategy based on the advantageous dimensions in the node's dynamic capability information. Combining the sharded data, terminal perception information identifiers, and node capability information, it generates primary storage records and sharded storage indexes, and marks the storage strategy adaptation logic.

[0080] After receiving the allocated data shards, the primary storage node configures a corresponding storage strategy based on its strengths in the dynamic information of its node capabilities. For example, if the primary storage node's strengths are storage capacity and security capabilities, the storage strategy might include using high-performance storage media (such as SSDs) to store the data shards to improve read and write speeds; simultaneously, encrypting the data shards and setting strict access control to ensure data security.

[0081] By combining the specific content of the data shards, the terminal awareness information identifiers, and the node capability information (such as storage media type, security protection configuration, etc.), a master storage record is generated. The master storage record details the storage status of the data shards in the master storage node, including storage path, encryption method, access permission list, etc. The shard storage index is an index table that records information such as the owner data identifier, shard unique identifier, and storage location (master storage node identifier and storage path) of all data shards, facilitating quick retrieval and access to data shards.

[0082] The annotation storage policy adaptation logic explains how the storage policy configuration is based on the node capability advantages and data shard storage requirements. For example, "Based on the storage capability advantages of node N-008 (large capacity SSD storage) and the high storage capacity requirements of data shard S-001, the storage path is configured as an SSD storage pool; based on the node's security capability advantages (AES-256 encryption support) and the high security protection requirements of data shards, encrypted data storage is configured."

[0083] Step S135: Through the terminal and node request response link, the data fragments and corresponding binding information are synchronized to the candidate storage nodes. The candidate storage nodes perform backup storage according to the storage policy of the primary storage node, generate candidate storage records, and combine the primary storage node identifier and synchronization timestamp.

[0084] By leveraging the established terminal and node request response link, data shards and their corresponding binding information in the primary storage node are synchronized to the candidate storage nodes. During the synchronization process, the data synchronization direction, response timing, and other rules set in the link data flow logic are followed to ensure that data shards and binding information can be accurately and timely transmitted to the candidate storage nodes.

[0085] After receiving the data fragments and binding information, the candidate storage node performs backup storage operations according to the primary storage node's storage policy. This means that the candidate storage node adopts the same or compatible storage methods, encryption methods, and access permission settings as the primary storage node to ensure the consistency and availability of the backup data. A candidate storage record is generated, recording the storage location, storage method, and binding information of the data fragments within the candidate storage node. This information is combined with the primary storage node identifier (indicating which primary storage node the backup data originated from) and a synchronization timestamp (recording the specific time of data synchronization) to trace the data's origin and synchronization history.

[0086] Step S136: Construct cross-node fragment identification association links. Based on terminal perception information identification and fragment identification fragments, establish the identification correspondence relationship and data connection logic of different fragments of the same data, and set fragment collaborative access rules.

[0087] Step S1361: Extract all data fragments and terminal perception identifier fragments corresponding to the same data identifier in the fragment and information binding set, classify them into single data fragment clusters, and label the unique data identifier, total number of fragments and core features of terminal perception information.

[0088] From the fragment and information binding set, all data fragments and terminal perception identifier segments corresponding to the same data are extracted based on the attribution data identifier. For example, for data with attribution data identifier D-006, all corresponding data fragments are S-008, S-009, and S-010, and the terminal perception identifier segments are "D-006-T-003-A-002-P01", "D-006-T-003-A-002-P02", and "D-006-T-003-A-002-P03". These data fragments and terminal perception identifier segments are grouped together to form a single data fragment cluster.

[0089] In a single data shard cluster, a unique data identifier is marked, namely the data attribution identifier (D-006); the total number of shards, namely the number of data shards contained in the data (e.g., 3); and the core features of terminal perception information, namely the key information extracted from the terminal perception identifier fragment, such as terminal identifier (T-003), application identifier (A-002), data format characteristics (e.g., video format), etc., in order to quickly identify and manage the data shard cluster.

[0090] Step S1362: Based on the terminal perception information identifier and fragment identifier fragment, analyze the identifier correspondence of each fragment of the same data, extract common identifier features and unique identifier fragments, determine the identifier correspondence relationship between fragments, establish an identifier correspondence index, and record the corresponding dimension and corresponding fragment identifier.

[0091] Based on terminal perception information identifiers and fragment identifiers for each data fragment, the identifier correspondence between fragments of the same data is analyzed. Common identifier features refer to the identifier portions shared by all fragments of the same data, such as the data owner identifier, terminal identifier, and application identifier. These common identifier features associate each fragment as a component of the same data. Specific identifier fragments refer to the identifier portions unique to each fragment, such as the fragment sequence number and the fragment's location information within the data, used to distinguish different fragments.

[0092] By analyzing common identifier features and unique identifier fragments, the identifier correspondence between fragments is determined. For example, the identifier correspondence between fragments S-008 and S-009 is "both belong to D-006, and the fragment serial numbers are adjacent (P01 and P02)". An identifier correspondence index is established, which uses the unique data identifier as the key to record the unique identifier fragment of each fragment, the unique fragment identifier, and the correspondence dimensions between them (such as serial number correspondence, position correspondence, etc.). For example, "D-006: S-008 (P01) — S-009 (P02) — S-010 (P03), corresponding dimension: consecutive serial numbers".

[0093] Step S1363: Combining the original characteristics and content logic of the data, establish the data connection logic between the shards, determine the content connection nodes, data integration order and cross-node connection rules of each shard, and achieve the data integrity after shard integration is consistent with the original data.

[0094] By combining the inherent characteristics of the data (such as data type, format, content structure, etc.) and content logic (such as the temporal order, spatial order, logical relationships, etc.), a data connection logic is established between different segments of the same data. A content connection node refers to the connection point between the content of adjacent segments. For example, in video data segments, the end frame of the previous segment and the beginning frame of the next segment are content connection nodes; in text data segments, the last sentence of the previous segment and the first sentence of the next segment constitute a content connection node.

[0095] The data integration order specifies the sequence in which the fragments are combined into complete data. This order is usually determined based on the logical content of the data, such as chronological order or chapter order. Cross-node connection rules govern how fragments are retrieved and connected when different fragments of the same data are stored on different nodes. For example, when data needs to be integrated, fragment data is retrieved sequentially from the nodes containing each fragment according to the data integration order, and then the content is connected and integrated locally or on a designated node.

[0096] By establishing these data connection logics, we can ensure that the data content after integrating each fragment can be restored to be completely consistent with the original data, thus guaranteeing data integrity.

[0097] Step S1364: Set the cross-node call logic in the shard collaborative access rules, determine the call order, data transmission path and node collaboration method of associated shards when the terminal accesses a certain shard, and adapt to the access requirements in the terminal application customization requirements.

[0098] The shard collaborative access rules define cross-node invocation logic. When a terminal needs to access a specific data shard, it may also need to access other related shards of that data to obtain the complete required information. The cross-node invocation logic determines the invocation order of related shards, i.e., invoking related shards sequentially according to the data integration order; the data transmission path, i.e., the request-response link traversed when obtaining shard data from the node where the related shard resides; and the node collaboration method, i.e., how the primary storage node and candidate storage nodes cooperate to complete the invocation and data transmission of related shards. For example, the primary storage node acts as a coordination center, responsible for requesting related shard data from other nodes and integrating all shard data before returning it to the terminal.

[0099] Cross-node call logic needs to be adapted to the access requirements of customized terminal applications, such as access response speed and data integrity requirements. For access requirements that require fast response, cross-node call logic can prioritize calling related shards on local nodes or geographically proximate nodes, and use parallel transmission to obtain data from multiple shards; for access requirements with high data integrity requirements, the call logic can add a data verification step to ensure that the obtained related shard data is accurate.

[0100] Step S1365: Formulate read and write permission allocation in the sharding collaborative access rules. Based on the security capabilities in the node capability dynamic information and the security level in the terminal requirements, allocate read and write permissions for each node to the sharded data. Configure full read and write permissions for the primary storage node and limited read, write and backup permissions for the candidate storage nodes.

[0101] Based on the security capabilities of each node in the dynamic node capability information (such as security protection configuration, data encryption capabilities, etc.) and the security level requirements in the customized needs of terminal applications, a read / write permission allocation scheme is formulated in the sharded collaborative access rules. The primary storage node, as the main storage location for data, is typically configured with full read / write permissions, meaning it can perform read and write operations on data shards, including data modification, deletion, and updates.

[0102] Candidate storage nodes are primarily used for data backup and disaster recovery, and are configured with limited read / write and backup permissions. Limited read permissions mean that a candidate storage node can read data shards stored on its own node, but may not be able to read associated shards on other nodes. Limited write permissions mean that a candidate storage node can typically only receive data shards synchronized from the primary storage node, and cannot actively modify or write data shards, unless temporary write permissions are granted according to fault tolerance rules in special circumstances such as primary storage node failure. Backup permissions ensure that candidate storage nodes can back up and store received data shards, maintaining consistency with the data on the primary storage node.

[0103] By allocating read and write permissions as described above, we can ensure data security and consistency, as well as achieve data backup and disaster recovery functions.

[0104] Step S1366: Combining the identifier corresponding index, data connection logic and shard collaborative access rules, construct the core framework of cross-node shard identifier association links, label link nodes, corresponding links and access rules, and combine data unique identifiers and terminal perception information identifiers.

[0105] This core framework combines identifier-based indexes, data connection logic, and shard collaborative access rules to construct a cross-node shard identifier association link. Centered on a unique data identifier, this framework links all shards of the same data through their corresponding identifier indexes. Link nodes include primary storage nodes and candidate storage nodes storing each data shard; corresponding links refer to the request-response links connecting these nodes, used for shard data transmission and collaborative access; and access rules, or shard collaborative access rules, define how to access and invoke data shards on each node.

[0106] The core framework is marked with the unique identifier of the link node, the identifier of the corresponding link, and the specific content of the access rules. The above marking information is associated with the unique data identifier and the terminal perception information identifier. For example, the cross-node sharding identifier associated with the data unique identifier D-006 is marked as "Link node: N-009 (primary), N-010 (candidate); Corresponding link: L-003 (primary link), L-003-1 (extended link); Access rules: cross-node call logic (sequential call, primary node coordination), read and write permission allocation (primary node full read and write, candidate node limited read and write and backup); Terminal perception information identifier: D-006-T-003-A-002".

[0107] Step S1367: Synchronize the core framework of the cross-node shard identifier association link to all corresponding storage nodes to ensure that each node understands the identifier correspondence between shards, the data connection logic and the shard collaborative access rules, and supports cross-node collaborative operation.

[0108] The established cross-node shard identifier association core framework is synchronized to all primary and candidate storage nodes that store the data shard. The synchronization process can be performed via inter-node communication links, ensuring that each node accurately receives and stores the core framework information.

[0109] Once each node understands the identifier correspondence between shards, the data connection logic, and the shard collaborative access rules, it can clearly identify how to find related shards, how to integrate shard data, and how to perform access control and data transmission when performing cross-node collaborative operations. This enables efficient and accurate cross-node collaborative operations, improving data access efficiency and availability.

[0110] Step S1368: Configure dynamic link adjustment rules. Based on changes in terminal application customization requirements and node capability updates, set link structure adjustment trigger conditions, including changes in requirements, node anomalies, and shard migration, to adapt to storage architecture optimization needs.

[0111] To enable cross-node sharding identification association links to adapt to changes, dynamic link adjustment rules are configured. These rules combine changes in terminal application customization requirements (such as changes in access requirements, adjustments to security levels, etc.) and node capability updates (such as node hardware upgrades, changes in capability advantage dimensions, etc.) to set trigger conditions for link structure adjustments.

[0112] The "Change of Request" trigger condition refers to the link structure adjustment triggered when the customized requirements of the terminal application change, causing the existing link structure to be unable to meet the new requirements. For example, when the data access frequency changes from "medium" to "high," the response timing and load distribution of the link need to be adjusted. The "Node Anomaly" trigger condition refers to the adjustment triggered when a storage node experiences an anomaly (such as failure, performance degradation, etc.), affecting the normal operation of the link. For example, if node N-010 fails and cannot provide service, the sharded data it carries needs to be transferred to other nodes, and the link structure needs to be adjusted. The "Shard Migration" trigger condition refers to the link structure adjustment triggered when a data shard is migrated from one node to another, updating the storage location of the shard and the associated link information. By configuring the above trigger conditions, the link structure can be dynamically adjusted according to the actual situation to adapt to the optimization needs of the storage architecture, ensuring that the link always maintains a high-efficiency and stable operating state.

[0113] Step S1369: Output the completed cross-node shard identifier association link, and simultaneously organize the corresponding index of the identifier, data connection logic, shard collaborative access rules and dynamic adjustment rules.

[0114] The output is a completed cross-node shard identifier association link, which includes the core framework, link nodes, corresponding links, access rules, etc. At the same time, the identifier correspondence index (recording the identifier correspondence relationship between shards), data connection logic (describing the connection method and order of shard content), shard collaborative access rules (specifying the cross-node call logic and read / write permission allocation), and dynamic adjustment rules (setting the trigger conditions and methods for link structure adjustment) are compiled to form a complete cross-node shard identifier association link document.

[0115] Step S137: Configure node collaboration rules in the cross-node sharding identification association link to determine the division of labor between the primary storage node and the candidate storage node, including data read and write permissions, update synchronization responsibilities and fault tolerance processing procedures, and match the advantageous dimensions in the dynamic information of node capabilities.

[0116] In the cross-node sharding identification association link, node collaboration rules are further configured to clarify the specific division of labor between the primary storage node and the candidate storage nodes. Regarding data read and write permissions, as described in step S1365, the primary storage node has full read and write permissions, while the candidate storage nodes have limited read, write, and backup permissions. This matches the advantages of the primary storage node in processing and security capabilities in the node capability dynamic information, ensuring that data read and write operations are dominated by the more capable node.

[0117] The update synchronization responsibility stipulates that the primary storage node is responsible for synchronizing the updated content of the data shards to the candidate storage nodes, while the candidate storage nodes are responsible for receiving and storing the updated data and feeding back the synchronization results to the primary storage node. This division of labor matches the advantages of the primary storage node in terms of transmission and processing capabilities, and the advantages of the candidate storage nodes in terms of storage capabilities.

[0118] In the fault-tolerant processing flow, when the primary storage node fails, a designated candidate storage node temporarily takes over the responsibilities of the primary storage node, performing data read, write, and synchronization operations. When the candidate storage node fails, the primary storage node is responsible for migrating its stored sharded data to other candidate storage nodes. This process matches the stability and fault-tolerant capabilities of each node in the dynamic node capability information to ensure rapid switching and recovery in the event of a failure.

[0119] Step S138: Extract the operation trajectory of the entire process of data sharding, storage records, binding and combination and link construction, record the complete path of data from splitting to storage and combination, mark the node participation, link usage information and corresponding details of each link, and form a related link log.

[0120] This process extracts the entire operational trajectory of data sharding, from generation to storage and the construction of cross-node association links. The data splitting stage includes operations such as analysis of raw data characteristics and volume parameters, shard processing, and shard identification. The storage stage includes operations such as shard allocation to primary storage nodes, primary storage record generation, and backup storage on candidate storage nodes. The integration stage includes operations such as constructing cross-node shard identification association links and configuring node collaboration rules. The complete path of data in these stages is recorded, including which processing steps the data underwent, which nodes it was allocated to, and which links it was transmitted through. The node participation in each stage is marked, indicating which nodes participated in the operation; link usage information, indicating which request-response links were used during the operation; and corresponding details, indicating the correspondence between the data shard identifier and other relevant identifiers (such as terminal identifiers and node identifiers). Based on the above records, an association link log is formed, which records link-related information throughout the entire lifecycle of the data sharding.

[0121] Step S139: Integrate the correspondence between the main storage record, candidate storage record, shard storage index and shard identifier to form a preliminary set of shard identifier storage, and mark the storage status, identifier integrity and node collaboration information of each shard.

[0122] The primary storage record generated by the primary storage node, the candidate storage record generated by the candidate storage node, the shard storage index generated in step S134, and the shard identifier correspondence established in step S1362 are integrated to form a preliminary shard identifier storage set. This preliminary set marks the storage status of each data shard, such as "normal storage," "backup storage," "migration in progress," and "deleted"; identifier integrity, i.e., whether the terminal-perceived identifier fragment of the data shard is complete, and whether there are any missing or incorrect identifiers; and node collaboration information, i.e., the primary storage node and candidate storage node involved in the storage and access process of this data shard, as well as the collaboration methods and rules between them.

[0123] By integrating this information, the initial set of fragment identifier storage comprehensively reflects the storage status and relationships of data fragments.

[0124] Step S1310: Supplement key information in the initial set of fragment identifier storage, including fragment storage time, terminal identifier matching degree and node capability adaptation, supplement and improve the information in the set, and output the fragment identifier storage set and associated link logs.

[0125] The initial set of shard identifiers is supplemented with key information. Shard storage time refers to the specific time when data shards are stored on the primary storage node and candidate storage nodes; terminal identifier matching degree refers to the degree of matching between the terminal-aware identifier fragment of the data shard and the corresponding terminal identifier in the terminal-aware information set, used to verify the accuracy of the identifier; node capability adaptation status refers to whether the capabilities of the primary storage node and candidate storage nodes continuously meet the storage requirements of the data shards, and the changes in capability adaptation dimensions.

[0126] The aforementioned key information is added to the initial shard identifier storage set to complete the set's information content. The final output is the completed shard identifier storage set and associated link logs. The shard identifier storage set contains all storage-related information for the data shards, and the associated link logs record the link operation trajectory of the data shards. Together, they constitute the core data for unstructured data shard storage and management.

[0127] Step S140: Monitor changes in terminal application customization requirements, optimize the shard identifier storage set based on the changes, adjust the node capability adaptation matrix and shard identifier association link structure accordingly, and generate a requirement-driven storage optimization scheme and optimization trajectory.

[0128] Step S141: Establish rules for monitoring changes in customized terminal application requirements, collect data in real time to collect information on adjustments to the application function requirements of the terminal, including changes in access frequency, security level, storage cycle, and sharing scope, and form a list of changes in requirements, combined with the terminal's unique identifier.

[0129] Establish rules for monitoring changes in customized terminal application requirements, specifying the content, methods, and frequency of monitoring. Collect data in real time from terminals regarding adjustments to application function requirements. This includes changes in access frequency (the number and frequency of data access by the terminal); security level adjustments (the increase or decrease in data security protection requirements by the terminal); storage cycle changes (the extension or shortening of the time data needs to be retained in the storage system); and changes in sharing scope (the changes in which other terminals or applications can access the data).

[0130] The collected information on changes in requests is combined with the unique identifier of the terminal. For example, if the request change for terminal T-004 is "access frequency changed from 'medium' to 'high', and security level adjusted from 'medium' to 'high'", a list of request changes is formed. This list records the content and time of the request changes for each terminal.

[0131] Step S142: Extract the storage information of each data shard from the shard identifier storage set, including storage location, node identifier, terminal perception information binding relationship and shard identifier information, and combine it with the operation trajectory in the associated link log to form shard storage traceability data.

[0132] Extract the storage information of each data fragment from the fragment identifier storage set. The storage location is the specific storage path of the data fragment in the node; the node identifier is the identifier of the main storage node and the candidate storage node that stores the data fragment; the terminal perception information binding relationship is the association between the data fragment and the terminal identifier, the home data identifier, etc.; the fragment identifier information is the unique identifier of the fragment, the identifier fragment, etc.

[0133] By combining the data sharding operation trajectory recorded in the associated link logs, such as shard generation time, allocation process, storage node change history, and link usage, shard storage traceability data is formed. This data completely records the storage history and related information of the data shards from their generation to their current state, facilitating the tracing and analysis of data storage status.

[0134] Step S143: Perform reverse comparison and adaptation between the list of changes in demands and the fragmented storage traceability data, match the changes in demands corresponding to the terminal identifier with the fragmented storage information, analyze whether the existing storage architecture meets the changed demands, identify the unmet items, record the unmet items as adaptation gaps, and determine the optimization direction.

[0135] The process involves matching the terminal identifiers in the list of changed demands with the terminal awareness information in the sharded storage traceability data to find the storage information of all data shards corresponding to each terminal identifier. Then, the changes in the terminal's demands are compared with the storage information of these data shards to analyze whether the existing storage architecture can meet the changed demands.

[0136] For example, if terminal T-004's requirements change to increased access frequency, it's necessary to analyze whether the access response speed of its corresponding data shards stored on the current node can meet the new access frequency demands. Similarly, if the security level is upgraded, it's necessary to analyze whether the current node's security protection configuration meets the new security level requirements. If the existing storage architecture cannot meet the changed requirements, identify the specific non-compliance items, i.e., adaptation gaps, such as "the processing capacity of node N-011 cannot meet the high access frequency requirements of data shard S-012" or "the security protection configuration of node N-011 does not meet the advanced security level requirements." Based on these adaptation gaps, determine the optimization direction, such as "migrating data shard S-012 to a node with stronger processing capabilities" or "upgrading the security protection configuration of node N-011."

[0137] Step S144: Analyze the causes of the adaptation gap, including the mismatch between node capabilities and new requirements, the sharding logic not conforming to the changes in requirements, the link response rate not meeting the adjusted access frequency, and insufficient adaptability of storage strategies, and generate a gap cause analysis report.

[0138] A causal analysis was conducted on the identified adaptation gaps. A mismatch between node capabilities and new requirements means that the existing node's hardware resources and capabilities cannot meet the changed demands of the terminal, such as lower processing power, storage capacity, and security capabilities compared to the new requirements. Inconsistent sharding logic means that the logical rules governing the correspondence between data shards and nodes, and the correspondence between shard identifiers, cannot adapt to the new requirements; for example, the storage location distribution of shards leads to increased access latency. Insufficient link response rate means that the existing response links cannot match the increased access frequency in terms of data transmission speed and synchronization frequency, resulting in data access latency. Inadequate storage strategy adaptability means that the storage strategies (such as storage media selection, encryption methods, backup strategies, etc.) of the primary and candidate storage nodes are incompatible with the new requirements; for example, the original encryption methods are no longer applicable after the security level is upgraded. Based on the above analysis, a gap causal analysis report was generated, detailing the specific causes of each adaptation gap.

[0139] Step S145: Based on the gap cause analysis report and optimization direction, formulate a sharded storage optimization strategy, determine the data shards to be adjusted, target storage nodes, node capability adaptation adjustment direction, and shard identifier associated link structure optimization scheme, and combine the strategy with the changes in requirements and adaptation gaps.

[0140] Based on the gap analysis report and optimization direction, a sharded storage optimization strategy is formulated. This includes identifying which data shards require adjustment due to the adaptation gap; target storage nodes, which are the new storage nodes selected for the data shards requiring adjustment, and these nodes should have the capability to meet the new requirements; node capability adaptation adjustment direction, which specifies the concrete direction and measures for capability enhancement for nodes requiring capability adjustments (such as security protection configuration upgrades); and an optimization scheme for the shard identifier association link structure, which outlines how to adjust the structure of cross-node shard identifier association links, such as link extensions and modifications to node collaboration rules, to adapt to data shard migration and node capability adjustments.

[0141] The fragmented storage optimization strategy is combined with the changes in requirements and the adaptation gaps. For example, "for the requirement to increase the access frequency of terminal T-004, the adaptation gap is that the processing capacity of node N-011 is insufficient. The optimization strategy is to migrate data fragment S-012 to the target storage node N-013 (the node with superior processing capacity), adjust the adaptation relationship between S-012 and the node in the node capability adaptation matrix, optimize the fragment identifier association link structure, and use link L-004 to connect terminal T-004 and node N-013."

[0142] Step S146: Generate a demand-driven storage optimization plan according to the sharded storage optimization strategy, mark the optimization object, target storage node, adjustment method, timing arrangement and adaptation standard, and combine the plan with terminal identification, demand change list and gap cause analysis report.

[0143] Based on the sharded storage optimization strategy, specific demand-driven storage optimization solutions are generated. The solutions clearly define the optimization targets (data shards and related nodes to be adjusted), the target storage nodes (the destination nodes for data shard migration), the adjustment methods (including specific operational methods such as data shard migration, node capability adjustment, and link structure optimization), the timing (the execution order and time plan for each optimization operation, such as "first migrate data shard S-012, then adjust the node capability adaptation matrix, and finally optimize the link structure"), and the adaptation standards (the goals to be achieved after optimization, such as "the access response time of data shard S-012 should be less than a preset threshold (e.g., 200 milliseconds), and the security level of node N-013 should reach high level"). The demand-driven storage optimization solutions are then combined with terminal identification, a list of demand changes, and a gap cause analysis report to clarify which demand change on which terminal the solution addresses, which adaptation gaps it resolves, and the basis for the solution, ensuring its clear relevance and operability.

[0144] Step S147: Adjust the node capability adaptation matrix according to the demand-driven storage optimization scheme, update the data and node adaptation relationship, capability matching dimensions and priority ranking, and synchronously correct the link response rules in the matrix to adapt to the changed terminal demands.

[0145] Based on the demand-driven storage optimization scheme, the node capability adaptation matrix is ​​adjusted. The data and node adaptation relationship is updated, and the primary storage node corresponding to the data identifier of the data shards to be adjusted is updated from the original node to the target storage node; the capability matching dimension is also updated accordingly to reflect the advantageous capability dimensions of the target storage node that match the new requirements of the data shards; the priority ranking is re-determined according to the priority of the target storage node to ensure that the new adaptation relationship conforms to the priority order.

[0146] The link response rules in the synchronization correction matrix refer to the request response links and related response rules adopted when data shards are stored on the new primary storage node, such as updating link identifiers, response timing, and load distribution methods, to adapt to the changed terminal requests. The adjusted node capability adaptation matrix can accurately reflect the new adaptation relationship between data and nodes and the link response rules.

[0147] Step S148: Adjust the cross-node shard identifier association link structure. Based on the new storage location and changes in requirements, update the identifier correspondence between shards, data connection logic and node collaboration rules, and optimize the shard collaborative access rules.

[0148] Based on the new storage location (target storage node) of the data shards and the changes in terminal requirements, the cross-node shard identifier association link structure is adjusted. The identifier mapping relationship between shards is updated, and the storage node identifier of the data shard is updated to the target storage node identifier; the storage location-related content in the data connection logic is also updated accordingly to ensure that the shards in the new storage location can be correctly found when integrating sharded data; the node collaboration rules are adjusted according to the new primary storage node and candidate storage nodes to clarify the division of labor and collaboration methods between the new nodes.

[0149] Optimize the sharded collaborative access rules, and adjust the cross-node call logic and read / write permission allocation according to changes in requirements (such as increased access frequency or enhanced security level). For example, for cases of increased access frequency, optimize the cross-node call logic by adopting faster links and parallel transmission methods; for cases of enhanced security level, strengthen the control of read / write permissions and add encryption measures during data transmission.

[0150] Step S149: Migrate the data shards that need to be adjusted to the target storage node, update the shard storage location, storage record and shard storage index, and record the migration details and adaptation effect in conjunction with the migration process and the demand-driven storage optimization solution.

[0151] Step S1491: Analyze the demand-driven storage optimization scheme, extract the data shard identifier, original storage node identifier, target storage node identifier, migration sequence and adaptation standards to be migrated, form a shard migration execution list, and combine the list with the optimization objects and adaptation gaps in the scheme.

[0152] This analysis of demand-driven storage optimization solutions extracts key information required for data shard migration. The data shard identifier to be migrated is the unique identifier of the data shard to be moved from the original storage node (e.g., S-012); the original storage node identifier is the identifier of the primary storage node where the data shard is currently located (e.g., N-011); the target storage node identifier is the identifier of the new primary storage node to which the data shard will be migrated (e.g., N-013); the migration timing refers to the time arrangement and order of data shard migration, such as "migrate during off-peak business hours (e.g., 2 AM to 4 AM)"; and the adaptation standards refer to the performance indicators and adaptation requirements that the data shard should achieve after migration, such as access response time and security level.

[0153] The extracted information is combined with the optimization object (i.e., the data shard) and adaptation gap (such as insufficient processing capacity of the original node) in the plan to form a shard migration execution list, for example, "Optimization object: S-012; Adaptation gap: N-011 insufficient processing capacity; Data shard identifier to be migrated: S-012; Original storage node identifier: N-011; Target storage node identifier: N-013; Migration time: Starts at 2:00 AM; Adaptation standard: Access response time < 200 milliseconds, Security level: High".

[0154] Step S1492: Based on the fragment migration execution list, construct a dedicated migration channel through the terminal and node request response link, configure the channel transmission parameters, match the size of the data fragments and the terminal perception identification fragments and the node network bandwidth capabilities, set the migration priority, and sort the data fragments by access frequency for migration.

[0155] Based on the shard migration execution checklist, a dedicated migration channel is built using existing terminal and node request response links. This channel is specifically designed for data shard migration to avoid impacting normal business data transmission. Channel transmission parameters are configured, including transmission rate limits, packet size, and transmission protocols. These parameters need to be matched to the size of the data shards (e.g., shard size), the complexity of the terminal-aware identification fragments, and the network bandwidth capabilities of the original and target storage nodes to ensure the stability and efficiency of data transmission during the migration process.

[0156] Set migration priorities. Assign higher migration priorities to data shards with high access frequency and significant impact on business operations, and migrate them first. Data shards with low access frequency can be migrated later. Sort the data shards to be migrated in descending order of access frequency, and perform the migration operation sequentially to reduce the impact of migrating high-frequency data shards on business operations.

[0157] Step S1493: The original storage node reads the data fragments to be migrated and their corresponding binding information, performs data encapsulation processing according to the adaptation standard, and the encapsulation content includes data fragments, terminal perception identification fragments, binding information content and storage requirement dimensions, generates a migration data packet, and marks the data packet ownership identifier and identifier integrity information.

[0158] The original storage node reads the data shards to be migrated and their corresponding binding information (such as terminal identification information, home data identifier, storage requirement dimension, etc.) according to the shard migration execution list. It then encapsulates this data according to the adaptation standards. The encapsulated content includes the actual content of the data shards, terminal-aware identifier fragments (ensuring the integrity of the data shard's identifier information), binding information content (ensuring the association between the data shards and the terminal-aware information is not lost after migration), and storage requirement dimensions (to guide the target storage node in making the correct storage configuration).

[0159] A migration data packet is generated, and each packet is assigned a unique packet ownership identifier (such as MP-001) to identify the migration task and data fragment corresponding to the packet. At the same time, the packet's identification integrity information is marked, that is, the terminal awareness identification fragment contained in the packet is checked and confirmed to be complete and accurate, ensuring that the migrated data fragment can be correctly associated with the terminal awareness information.

[0160] Step S1494: Transmit migration data packets to the target storage node through a dedicated migration channel. After receiving the data packets, the target storage node performs unpacking and storage operations according to the advantageous dimensions and adaptation standards in the node capability dynamic information. Combined with data fragmentation, terminal perception information identification and target storage node capability information, a target storage record is generated.

[0161] A dedicated migration channel is established to transfer migration data packets from the original storage node to the target storage node. Upon receiving the data packets, the target storage node first verifies them, checking whether the integrity and identification integrity information of the data packets meet the requirements. After successful verification, the node performs decapsulation operations according to the advantageous dimensions (such as storage capacity, security capabilities, etc.) and adaptation standards in the node's dynamic capability information, extracting data fragments, terminal-aware identification fragments, and bound information content from the data packets.

[0162] Based on the specific content of the data shards, the terminal's awareness information identifiers, and the target storage node's capability information (such as storage media type and security protection configuration), storage operations are performed. The data shards are stored in appropriate locations on the target storage node, and corresponding storage policies (such as encryption methods and access permissions) are configured according to storage requirements. A target storage record is generated, recording the storage location, storage policy, and binding information of the data shards on the target storage node.

[0163] Step S1495: The original storage node deletes the data shards that have been migrated, updates its own storage records and remaining hardware resource status, marks the data migration completion identifier, target storage node information and migration timestamp, generates a migration completion record, and synchronously updates the storage load parameters in the node capability dynamic information.

[0164] After confirming that the data shard has been successfully migrated to the target storage node (e.g., upon receiving a successful migration confirmation message from the target storage node), the original storage node deletes the data shard it was storing to free up storage space. It then updates the storage record on the original storage node, marking the data shard's status as "migrated," and records the target storage node information and the migration completion timestamp.

[0165] A migration completion record is generated, which includes information such as data shard identifier, original storage node identifier, target storage node identifier, migration completion time, and data migration completion identifier. The storage load parameters in the node capability dynamic information are updated synchronously, reflecting the change in the node's current storage resource status: the used storage space of the original storage node decreases, and the remaining storage space increases.

[0166] Step S1496: Synchronously update the shard storage index, correct the data storage location, node identifier, binding information content and identifier correspondence of the migrated shards, and update the shard storage index to reflect the actual storage status after data migration.

[0167] The shard storage index is a crucial index that records information such as the storage location of all data shards. After data shard migration, it needs to be updated synchronously. This involves correcting the data storage location of the migrated shards, updating the original storage path or address to the storage path or address in the target storage node; updating the node identifier to the unique identifier of the target storage node; updating the node-related information in the binding information content accordingly; and correcting the storage node-related portion of the identifier mapping to the target storage node identifier. These corrections ensure that the shard storage index accurately reflects the actual storage state after data migration, enabling subsequent data access and management to correctly locate the data shards.

[0168] Step S1497: Combining the migration process with the demand-driven storage optimization scheme, record the optimization objects, adaptation gaps and optimization goals corresponding to the migrated shards, mark the parameter adjustments, link usage and node collaboration during the migration process, and form a migration detail record table.

[0169] The data shard migration process is documented in conjunction with demand-driven storage optimization solutions. The optimization object corresponding to the migrated shard refers to the data shard itself (e.g., S-012); the adaptation gap refers to the reason driving this migration (e.g., insufficient processing capacity of the original node); the optimization goal refers to the effect that the migration needs to achieve (e.g., access response time <200 milliseconds).

[0170] The migration process includes recording parameter adjustments such as changes to the transmission rate limit and packet size of the dedicated migration channel; link usage, including the identification and transmission status of the dedicated migration channel used; and node collaboration, detailing the cooperation methods and information exchange between the original and target storage nodes during the migration process (e.g., the original storage node sending data packets and the target storage node returning confirmation information). Based on these records, a migration detail log table is created, providing a comprehensive record of the entire data shard migration process and related information.

[0171] Step S1498: Analyze the data sharding storage adaptation effect after migration, compare the access response rate, node capability adaptation degree and terminal demand satisfaction before and after migration, form an adaptation effect analysis table, and combine it with the migration detail record table.

[0172] The storage adaptation effect after migrating data shards to the target storage node is analyzed. This includes comparing the data shard access response rate before and after migration (the time interval from when the terminal initiates an access request to when the data shard is received); the node capability adaptation level (the degree of matching between the target storage node's capabilities and the data shard storage requirements); and the satisfaction of terminal requirements (whether the migrated data shards meet the changed customized requirements of terminal applications, such as increased access frequency or enhanced security levels).

[0173] By comparing and analyzing data, an adaptation effect analysis table is generated. For example, "Access response rate before migration: 300 milliseconds; Access response rate after migration: 150 milliseconds; Node capability adaptation degree: Before migration, the original node matched 2 dimensions; After migration, the target node matches 4 dimensions; Terminal requirement satisfaction: Access frequency and security level requirements are both met." The adaptation effect analysis table is combined with the migration detail record table to comprehensively reflect the effect of data sharding migration.

[0174] Step S1499: Synchronize the migration results and adaptation effects of the terminal and node request response links to all corresponding storage nodes, update the shard storage locations and corresponding relationships in the cross-node shard identifier association links, and update the relevant information in the cross-node shard identifier association links to maintain the consistency of collaborative operations among nodes.

[0175] The migration results of the data shards (such as whether the migration was successful, target storage node information, etc.) and the adaptation effect analysis table are synchronized to all storage nodes related to the data shards through the terminal and node request response link, including the original storage node, the target storage node and other candidate storage nodes. After receiving the synchronization information, the above nodes update the shard storage location and corresponding relationship in the cross-node shard identifier association link they maintain, ensuring that all nodes know the new storage location and association information of the data shards.

[0176] At the same time, update other relevant information in the cross-node sharding identifier associated link, such as link usage information and the part of the node collaboration rules involving the data shard, to maintain the consistency of collaborative operations between nodes and avoid collaborative operation failures or errors due to inconsistent information.

[0177] Step S14910: Output the data sharding migration execution results, and simultaneously organize the migration details record table, adaptation effect analysis table, sharding storage index update information and node capability adjustment data, and supplement them to the optimization trajectory.

[0178] Output the execution results of the data sharding migration, indicating whether the migration task was successfully completed and whether the adaptation criteria were met. Simultaneously compile a migration detail log table (recording the migration process), an adaptation effect analysis table (recording the migration effect), sharded storage index update information (recording index changes), and node capability adjustment data (such as updates to the original storage node's storage load parameters), supplementing the optimization trajectory with this information. The optimization trajectory is a document recording the entire demand-driven storage optimization process; by supplementing this information, it fully reflects the execution status and results of the data sharding migration optimization step.

[0179] Step S1410: Record information on the entire process of demand change monitoring, reverse adaptation, optimization plan formulation and execution, form an optimization trajectory, mark the operation content and basis of each optimization step, and output the demand-driven storage optimization plan and optimization trajectory.

[0180] This system records the entire process, from monitoring changes in end-application customization requests to reverse-engineering the request change list and sharded storage traceability data, and then to the formulation and execution of a request-driven storage optimization solution (including adjustments to the node capability adaptation matrix, shard identifier association structure, and data shard migration). Each optimization step is annotated with its operational content, specifying the specific operations performed and their rationale (e.g., based on the request change list or gap cause analysis report). Based on these records, an optimization trajectory is created, comprehensively documenting the entire storage optimization process and key information. Finally, a request-driven storage optimization solution and trajectory are output. The solution guides the specific implementation of storage optimization, while the trajectory records the optimization process and results.

[0181] Step S150: Based on the demand-driven storage optimization solution, construct a closed loop for storage architecture operation and maintenance, update the dynamic information of node capabilities and the sharding identifier association rules, and output the terminal-adaptive unstructured data storage architecture and operation and maintenance logs. The terminal-adaptive unstructured data storage architecture includes the sharding identifier storage location, terminal corresponding rules and node collaboration rules. The operation and maintenance logs record the details of the entire process of information construction, sharding storage and optimization iteration.

[0182] Step S151: Based on the demand-driven storage optimization scheme and optimization trajectory, sort out the storage architecture operation and maintenance requirements, covering dynamic updates of node capabilities, iteration of shard identifier association rules, optimization of link response performance and prediction of demand changes, and form an operation and maintenance requirement list.

[0183] Based on the optimization measures proposed in the demand-driven storage optimization scheme and the optimization process recorded in the optimization trajectory, the operational requirements of the storage architecture are identified. The requirement for dynamic node capability updates refers to the need to establish a mechanism that allows node capability information to be updated in real time based on changes in node hardware resources and the effects of storage optimization. The requirement for iterative sharding identifier association rules refers to the need for continuous iteration and improvement of sharding identifier association rules as data shards migrate and link structures are adjusted. The requirement for link response performance optimization refers to the continuous monitoring and optimization of performance indicators such as transmission rate and stability of the demand response link. The requirement for predicting demand changes refers to predicting possible future demand changes by analyzing historical demand change data, so as to adjust and optimize the storage architecture in advance. These operational requirements are compiled into an operational requirement list, clarifying the goals and directions of operation and maintenance.

[0184] Step S152: Construct dynamic update rules for node capabilities, combine changes in node hardware resources and the execution effect of demand-driven storage optimization solutions, update the advantages and disadvantages in the dynamic information of node capabilities in real time, correct the node capability evaluation parameters, and synchronize the actual status of the node.

[0185] A dynamic update rule for node capabilities is established to achieve real-time updates of node capability information. This rule combines real-time changes in node hardware resources (such as increases or decreases in physical storage capacity, fluctuations in read / write processing rates, and changes in network bandwidth) with the execution effects of demand-driven storage optimization schemes (such as changes in node load and adaptation effects after data sharding migration). When node hardware resources change or optimization schemes are implemented, the strengths and weaknesses in the node capability dynamic information are updated in real time based on the changes and execution effects. For example, if a node upgrades its network bandwidth, its transmission capability may change from a weakness to a strength; after data sharding migration, the node's storage load decreases, and its storage capability advantages may become more prominent. Node capability evaluation parameters are revised, that is, the indicators and standards used to evaluate each capability dimension of the node are adjusted to accurately reflect the actual capability status of the node. Through the above updates and revisions, it is ensured that the node capability dynamic information remains synchronized with the actual status of the node.

[0186] Step S1521: Establish real-time data collection rules for node hardware resources, set the collection frequency and collection dimensions. The collection dimensions cover the node's remaining storage capacity, read and write processing rate, network bandwidth usage, security protection status and hardware operation status, forming real-time data of node hardware resources, combined with the node's unique identifier.

[0187] Establish real-time data collection rules for node hardware resources, specifying the collection frequency and dimensions. The collection frequency can be set based on node stability and resource changes. For example, for nodes with rapidly changing resources, the collection frequency can be set to once per minute; for nodes with relatively stable resources, the collection frequency can be set to once per hour. Collection dimensions include: remaining storage capacity (the amount of space currently available for data storage); read / write processing rate (the node's current average read and write speeds); network bandwidth usage (the percentage of network bandwidth currently used and the actual transmission rate); security protection status (security-related parameters such as firewall status and encryption module operation); and hardware operating status (hardware parameters such as CPU utilization, memory usage, disk I / O, and temperature). Following the set collection frequency and dimensions, real-time data collection is performed on the hardware resources of all nodes, forming real-time node hardware resource data. This data is then combined with the node's unique identifier to ensure accurate identification of the hardware resource status of each node.

[0188] Step S1522: Combining real-time data of node hardware resources with the execution effect of demand-driven storage optimization solutions, analyze the impact of the optimization solution execution on node hardware resources, including changes in storage load, processing capacity usage, and bandwidth resource consumption, and generate a resource impact analysis report.

[0189] The analysis combines real-time data on node hardware resources with the execution effects of demand-driven storage optimization solutions. It analyzes the impact of the optimization solution on node storage load, specifically the changes in used and remaining storage space after data sharding migration; the impact on processing capacity utilization, specifically the changes in CPU and memory usage during and after the optimization process; and the impact on bandwidth consumption, specifically the changes in node network bandwidth usage caused by data sharding migration and synchronization operations. Based on this analysis, a resource impact analysis report is generated, detailing the specific impact of the optimization solution on the hardware resources of each node.

[0190] Step S1523: Based on the resource impact analysis report, adjust the strength and weakness dimensions in the node capability dynamic information. When the node storage load decreases and the processing speed increases, strengthen the corresponding capability strength dimension. When the load is too high and the processing speed decreases, update the weakness dimension label and simultaneously correct the capability assessment description.

[0191] Based on the resource impact analysis report, adjustments were made to the strengths and weaknesses dimensions in the node capability dynamic information. When a node's storage load decreases (e.g., due to data shard migration) and its processing speed increases, it indicates that the node's advantages in storage and processing capabilities are more pronounced. These strengths are highlighted, for example, "Storage Capability: Strength" is changed to "Storage Capability: Strong Strength." When a node's processing speed decreases due to excessive load, it indicates that the node's processing capability may become a weakness. The weakness dimension labeling is updated, for example, "Processing Capability: Strength" is changed to "Processing Capability: Weakness." The capability assessment descriptions were also revised simultaneously, updating the textual descriptions of each node's capability dimensions to match the adjusted strengths and weaknesses dimensions, accurately reflecting the node's current capability status.

[0192] Step S1524: Correct the node capability assessment parameters. Based on the real-time data of node hardware resources and the results of resource impact analysis, adjust the assessment indicators of each capability dimension, including storage capacity threshold, read / write rate benchmark and bandwidth adaptation range, and correct the node capability assessment parameters so that they are calculated based on the real-time data of node hardware resources.

[0193] Node capability assessment parameters are specific indicators and standards used to evaluate various dimensions of a node's capabilities. These parameters need to be adjusted based on real-time data of node hardware resources and resource impact analysis results. The storage capacity threshold is a critical value used to determine whether a node's storage capacity is sufficient, and it is adjusted according to changes in the node's remaining storage capacity. The read / write rate benchmark is a reference standard for evaluating the node's read / write processing speed, and it is adjusted according to changes in the node's actual read / write processing speed. The bandwidth adaptation range is the bandwidth range used to evaluate the node's transmission capabilities, and it is adjusted based on network bandwidth utilization and resource impact analysis results. By revising these node capability assessment parameters, node capability assessments can be calculated based on the latest real-time data of node hardware resources, ensuring the accuracy and timeliness of the assessment results.

[0194] Step S1525: Configure differentiated early warning standards for the weak and strong dimensions of each capability dimension. When the node hardware resource parameters exceed the node capability early warning threshold, a capability adjustment prompt is triggered, combining dynamic information of node capabilities and real-time data of hardware resources.

[0195] Configure differentiated early warning standards for the weak and strong dimensions of each node's capabilities. For weak dimensions, set a lower early warning threshold. When the node's hardware resource parameters approach or exceed this threshold, a capability adjustment prompt will be triggered in a timely manner so that measures can be taken to improve this capability dimension as early as possible. For strong dimensions, set a higher early warning threshold. When the hardware resource parameters exceed this threshold, a prompt will be triggered to prevent the strong dimension from becoming a weak dimension due to excessive resource consumption.

[0196] The node capability warning threshold is set based on dynamic node capability information and real-time hardware resource data. For example, for a node where storage capacity is a bottleneck, the warning threshold can be set to have remaining storage capacity less than 20% of the total capacity; for a node where storage capacity is an advantage, the warning threshold can be set to have remaining storage capacity less than 10% of the total capacity. When the node's hardware resource parameters exceed the set warning threshold, the system automatically triggers a capability adjustment prompt, notifying operations and maintenance personnel to handle the situation.

[0197] Step S1526: Construct a capability update feedback link to synchronize the updated content of node capability dynamic information to the node capability adaptation matrix, the cross-node sharding identifier association link, and the terminal and node request response link, and synchronize the updated content of node capability dynamic information to relevant components so that each component can perform operations based on the latest information.

[0198] A capability update feedback loop is established to ensure that updates to node capability dynamic information are promptly synchronized to other relevant components of the storage architecture. When the strengths, weaknesses, or evaluation parameters in the node capability dynamic information are updated, this feedback loop synchronizes the updated content to the node capability adaptation matrix, ensuring that the data in the matrix matches node capabilities based on the latest node capabilities; it also synchronizes to the cross-node sharding identification association link, adapting the link's node collaboration and access rules to changes in node capabilities; and it synchronizes to the terminal and node request response link, adjusting the link's transmission parameters and load balancing strategies according to node capabilities. Through this synchronization mechanism, all components of the storage architecture can perform operations based on the latest node capability dynamic information, ensuring the coordinated and efficient operation of the entire storage architecture.

[0199] Step S1527: Record the dynamic update process of node capabilities, including data collection, impact analysis, dimension adjustment, parameter correction and feedback synchronization details, to form a capability update log, combined with the node's unique identifier and update timestamp.

[0200] The entire process of dynamically updating node capabilities is recorded. The data collection phase records the collection time and values ​​of real-time hardware resource data; the impact analysis phase records the main conclusions of the resource impact analysis report; the dimension adjustment phase records the adjustments made to the strengths and weaknesses dimensions; the parameter correction phase records the modifications made to the node capability assessment parameters; and the feedback synchronization phase records the time and results of synchronizing the updated content to each component. These records, combined with the node's unique identifier and update timestamp, form a capability update log. This log details the historical process of each node's dynamic capability updates, facilitating traceability and auditing.

[0201] Step S1528: Combine capability update logs with the intelligent operation and maintenance closed loop of storage architecture, and mark the update triggering reasons, execution process and impact on the operation and maintenance closed loop.

[0202] By combining capability update logs with the intelligent operation and maintenance closed loop of storage architecture, the logs are marked with the triggering reasons for each dynamic update of node capabilities, such as changes in hardware resources, execution of optimization schemes, etc.; the execution process, that is, the steps and operations experienced during the update process; and the impact on the operation and maintenance closed loop, that is, the impact of the update on other links in the operation and maintenance closed loop of storage architecture (such as the prediction of changes in demand, the iteration of shard identification association rules, etc.).

[0203] Step S1529: Regularly calibrate the node capability assessment framework, and adjust the capability assessment dimensions and parameter standards in combination with changes in the customized requirements of terminal applications and the optimization effect of terminal-adaptive unstructured data storage architecture to adapt to continuously changing storage needs.

[0204] To ensure the node capability assessment framework adapts to continuously changing storage needs, it is calibrated periodically. This involves adjusting the capability assessment dimensions based on long-term trends in end-application customization requirements (such as generally higher security requirements and continuously increasing access frequency) and the optimization effects of end-adaptive unstructured data storage architectures (such as which capability dimensions have a greater impact on adaptation). This includes adding new assessment dimensions (such as energy consumption capacity) or adjusting the weights of existing dimensions. Parameter standards are also adjusted, i.e., updating the assessment metrics and thresholds for each capability dimension to align with current storage needs and technological advancements.

[0205] Step S15210: Output the completed node capability dynamic update rules, simultaneously organize resource collection rules, early warning threshold configurations, update feedback links and capability update logs, and combine them with the terminal-adaptive unstructured data storage architecture to achieve dynamic adaptation of node capabilities.

[0206] The system outputs a completed dynamic update rule for node capabilities. This rule includes real-time node hardware resource acquisition rules, resource impact analysis methods, capability dimensions and parameter update processes, early warning standard configurations, and update feedback mechanisms. It also simultaneously organizes resource acquisition rules (clarifying acquisition frequency and dimensions), early warning threshold configurations (early warning thresholds for each capability dimension), update feedback mechanisms (synchronization paths and methods), and capability update logs (update history). These elements are then integrated into a terminal-adaptive unstructured data storage architecture, enabling the architecture to dynamically adapt node capabilities. This allows for real-time adjustments to node capability information based on changes in node hardware resources and storage requirements, ensuring the adaptability and efficiency of the storage architecture.

[0207] Step S153: Iterate the fragment identification association rules, combine the changes in the requirements and the adaptation experience in the optimization trajectory, update the identification correspondence logic between fragments, data connection rules and node collaboration rules, and improve the rules' adaptability to changes in terminal requirements.

[0208] Based on changes in end-application customization requirements and adaptation experience accumulated during optimization, the shard identifier association rules are iteratively updated. Updating the identifier correspondence logic involves adjusting the identifier correspondence method between different shards of the same data to adapt to new shard storage locations or identifier systems. Updating the data connection rules involves modifying the connection nodes, integration order, or cross-node connection methods between shards to improve the efficiency and accuracy of data integration. Updating the node collaboration rules involves optimizing the division of labor, data synchronization methods, and fault tolerance processes between primary storage nodes and candidate storage nodes to adapt to changes in node capabilities and new requirements. By iteratively updating these rules, the adaptability of the shard identifier association rules to changes in end-application requirements is improved, ensuring that the cross-node shard identifier association link can continue to work effectively.

[0209] Step S154: Optimize the performance of the terminal and node request response link. Based on the response performance requirements in the operation and maintenance requirement list, adjust the link transmission parameters, bandwidth allocation and synchronization timing to improve the response rate and stability of the link to changes in requests.

[0210] Based on the link response performance requirements (such as transmission rate, stability, and latency) in the operation and maintenance requirements list, the performance of the terminal and node request response links is optimized. Link transmission parameters, such as packet size, transmission protocol options, and retransmission mechanism parameters, are adjusted to improve data transmission efficiency and reliability. Bandwidth allocation is optimized by dynamically adjusting the bandwidth usage of each link according to its load and data transmission priority, preventing some links from affecting response performance due to insufficient bandwidth. Synchronization timing is adjusted by shortening the synchronization interval for frequently accessed or important data fragments to ensure data timeliness and consistency. With the goal of improving the link's response rate and stability to changes in requests, link parameters are continuously adjusted to better meet the needs of terminal applications.

[0211] Step S155: Construct a closed-loop intelligent operation and maintenance system for the storage architecture, connecting the monitoring of changes in demand, execution of storage optimization, updating of node capabilities and iteration of rules, and setting the closed-loop operation logic, including triggering conditions, execution process and feedback rules, to adapt to terminal demands.

[0212] Define the closed-loop operation logic, including triggering conditions (i.e., under what circumstances a certain link in the closed loop is activated, such as triggering the storage optimization execution link when a change in terminal demand is detected); execution flow (i.e., the specific operation steps and sequence of each link); and feedback rules (i.e., how the execution results of each link are fed back to other links to guide subsequent operations, such as feeding back the storage optimization execution results to the node capability update link to update the node capability dynamic information). By defining these operational logics, the intelligent operation and maintenance closed loop of the storage architecture can automatically adapt to changes in terminal demand, achieving continuous optimization and self-adjustment of the storage architecture.

[0213] Step S156: Integrate the optimized node capability dynamic information, the iterated sharding identification association rules, the optimized demand response link and intelligent operation and maintenance closed-loop rules, construct a terminal-adaptive unstructured data storage architecture, and mark the corresponding relationships and operating rules of each component.

[0214] This paper integrates optimized and iterated key elements, including dynamic node capability information (reflecting the current capability status of nodes), shard identifier association rules (guiding shard association and collaborative access), request response links (enabling data transmission), and intelligent operation and maintenance closed-loop rules (ensuring continuous operation and maintenance and optimization of the architecture). Based on these elements, a terminal-adaptive unstructured data storage architecture is constructed. The architecture annotates the correspondence between components, such as the correspondence between dynamic node capability information and the node capability adaptation matrix, and the correspondence between shard identifier association rules and cross-node shard identifier association links; it also annotates the operational rules of each component, such as the update rules for dynamic node capability information, the transmission rules for request response links, and the triggering and execution rules for the intelligent operation and maintenance closed loop.

[0215] Step S157: Start the operation and maintenance log generation process, set the log structure, including node capability update records, rule iteration content, link optimization details, request adaptation effect and closed-loop operation trajectory modules. The log structure contains records of the entire operation and maintenance process.

[0216] The log structure includes multiple modules: the node capability update record module records the update status of dynamic information on node capabilities (such as a summary of the capability update log); the rule iteration content module records the iterative update content of the rules associated with shard identifiers; the link optimization details module records the parameter adjustments and effects of link performance optimization in response to requests; the request adaptation effect module records the adaptation effect of the storage architecture to changes in terminal requests (such as a summary of the adaptation effect analysis table); and the closed-loop operation trajectory module records the operation process and triggering conditions of the intelligent operation and maintenance closed loop of the storage architecture.

[0217] Step S158: Extract the specific content and basis of node capability updates, rule iterations, and link optimizations, fill them into the corresponding log module, and record the parameter changes, adaptation and upgrade points, and accumulated operation and maintenance experience before and after optimization.

[0218] Extract the specific content of node capability updates (such as adjustments to advantage dimensions and parameter corrections) and their basis (such as changes in hardware resources and resource impact analysis reports) from the capability update logs and fill them into the node capability update record module; extract the iteration content of shard identifier association rules (such as modifications to identifier corresponding logic and optimization of node collaboration rules) and their basis (such as changes in requirements and adaptation experience) from the rule iteration document and fill them into the rule iteration content module; extract the specific parameter adjustments for link optimization (such as transmission rate limits and synchronization intervals) and their basis (such as the operation and maintenance requirement list and response performance test results) from the link optimization record and fill them into the link optimization details module. Record the parameter changes before and after optimization (such as changes in node storage load and changes in link transmission rate), adaptation and upgrade points (such as newly added capability assessment dimensions and optimized collaborative access rules), and accumulated operation and maintenance experience (such as effective optimization methods and solutions to common problems) in each module.

[0219] Step S159: Organize the terminal request adaptation effect information, including the degree of request satisfaction, storage performance improvement, link response rate changes and operation and maintenance closed-loop status, and fill it into the corresponding log module.

[0220] Organize information on the adaptation effect of the storage architecture to terminal requirements. The degree of requirement satisfaction refers to the extent to which the storage architecture meets the new requirements after changes in terminal application customization (e.g., fully or partially). Storage performance improvement refers to the increase in various performance indicators of the storage system (e.g., storage capacity utilization, data read / write speed) after optimization. Link response rate changes refer to the changes in transmission rate, latency, and other indicators of the request response link before and after optimization. Operational closed-loop status refers to whether the intelligent operation and maintenance closed loop of the storage architecture is running normally and whether each link is working collaboratively.

[0221] Step S1510: Output terminal-adaptive unstructured data storage architecture and operation and maintenance logs, and synchronously combine the information of each component of the architecture and the log content.

[0222] The terminal-adaptive unstructured data storage architecture includes key information such as shard identifier storage location, terminal correspondence rules, and node collaboration rules. It can be dynamically adjusted and optimized according to changes in the customized requirements of terminal applications, achieving efficient storage and management of unstructured data. The operation and maintenance logs record the details of the entire process, including information construction, shard storage, and optimization iteration.

[0223] Furthermore, Figure 2 A schematic diagram of the hardware structure of an unstructured data storage system 100 based on a distributed architecture for implementing the methods provided in the embodiments of this application is shown. Figure 2As shown, the unstructured data storage system 100 based on a distributed architecture may include at least one processor 102 (the processor 102 may be, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, a transmission device 106 for communication functions, and a controller 108. Those skilled in the art will understand that... Figure 2 The structure shown is for illustrative purposes only and does not limit the structure of the unstructured data storage system 100 based on a distributed architecture. For example, the unstructured data storage system 100 based on a distributed architecture may also include... Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown.

[0224] The memory 104 can be used to store software programs and modules of application software, such as the program instructions corresponding to the method embodiments described above in this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-described unstructured data storage method based on a distributed architecture. The transmission device 106 is used to acquire or send data via a network.

[0225] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

Claims

1. A method for storing unstructured data based on a distributed architecture, characterized in that, The method includes: Based on the customized application requirements of data acquisition terminals, unstructured data terminal perception information is constructed. According to the original characteristics of data, terminal hardware parameters, scene acquisition attributes and application function requirements, a set of terminal perception information is generated. The set of terminal perception information includes the three-dimensional correspondence identifiers of data, terminals and applications and feature adaptation logic. Based on the terminal perception information set, a distributed node capability dynamic information is constructed. The storage requirements corresponding to the information characteristics are matched with the node hardware resource information to generate a node capability adaptation matrix and a terminal and node demand response link. Based on the node capability adaptation matrix, terminal identification fragmentation is performed on unstructured data. By combining the fragmented data with the terminal perception information identification, a cross-node fragment identification association link is constructed to obtain the fragment identification storage set and association link log. Monitor changes in terminal application customization requirements, optimize the shard identifier storage set according to the changes, adjust the node capability adaptation matrix and shard identifier association link structure accordingly, and generate a demand-driven storage optimization scheme and optimization trajectory. Based on the demand-driven storage optimization solution, a closed loop for storage architecture operation and maintenance is constructed, node capability dynamic information and shard identification association rules are updated, and terminal-adaptive unstructured data storage architecture and operation and maintenance logs are output. The terminal-adaptive unstructured data storage architecture includes shard identification storage location, terminal corresponding rules and node collaboration rules, and operation and maintenance logs record the details of the entire process of information construction, shard storage, optimization and iteration.

2. The unstructured data storage method based on a distributed architecture according to claim 1, characterized in that, The process of constructing distributed node capability dynamic information based on terminal perception information sets, matching storage requirements corresponding to information features with node hardware resource information, and generating a node capability adaptation matrix and a terminal-node demand response link includes: Extract the storage requirement dimension corresponding to each piece of data in the terminal sensing information set, covering storage capacity requirements, access response requirements, security protection requirements, fault tolerance and redundancy requirements, and life cycle requirements, to form a data storage requirement list, and mark the correspondence between each requirement dimension and the terminal sensing information identifier. Collect hardware resource information of all distributed nodes, including node physical storage capacity, read and write processing speed, security protection configuration, geographical location, network bandwidth resources and hardware operating status, classify and record them according to the node's unique identifier, and form a node hardware resource list. The node hardware resource list is broken down into dimensions to construct a node capability assessment framework, covering storage capacity, processing capacity, transmission capacity, security capacity and stability capacity. Each dimension is combined with specific hardware resource parameters to form the basic content of node capability. Based on the basic content of node capabilities, a distributed node capability dynamic information is constructed. Combining node hardware resource parameters and capability evaluation dimensions, the capability advantages and disadvantages of each node are marked to form a dynamic information set of node capabilities, combined with the unique identifier of the node. The data storage requirement list is matched with the dynamic information set of node capabilities. The requirement dimension of each data item is matched with the capability advantage dimension of the node. Nodes with matching capabilities are selected as candidate storage nodes for the corresponding data, and the correspondence between candidate storage nodes and data identifiers is marked. Combining terminal hardware parameters and geographical location information from the terminal perception information, candidate storage nodes are sorted, and nodes that are geographically close to the terminal and have compatible hardware interfaces are prioritized as primary storage nodes, generating a node priority sorting table; Construct a basic link for responding to demands between terminals and nodes, connect the main storage node with the corresponding data acquisition terminal, configure the link transmission parameters according to the data transmission characteristics of the terminal and the network bandwidth capacity of the node, and mark the correspondence between the link and the terminal and node; Extend the request response link to the candidate storage node, construct a full-link response structure between the terminal, the master node, and the candidate node, and set the link data flow logic according to the access response requirements in the data storage requirement list. The link data flow logic includes data synchronization direction, response timing, and load distribution method to match the access response requirements in the data storage requirement list. Based on the node capability dynamic information set, node priority ranking table, and demand response link configuration, a node capability adaptation matrix is ​​constructed to determine the adaptation relationship between data and nodes, capability matching dimensions, and link response rules. The matrix adaptation logic is labeled, and the node capability adaptation matrix and the terminal and node demand response link are output. The node capability dynamic information set, candidate storage node list, and link configuration information are also compiled simultaneously.

3. The unstructured data storage method based on a distributed architecture according to claim 1, characterized in that, The node capability adaptation matrix performs terminal identifier sharding on unstructured data. Combining the sharded data with terminal perception information identifiers, a cross-node shard identifier association link is constructed, resulting in a shard identifier storage set and association link logs, including: Based on the original data features and volume parameters in the terminal sensing information set, terminal identification fragmentation processing is performed on unstructured data to split it into multiple data fragments. Each data fragment carries the terminal sensing identification fragment of the corresponding data and marks the fragment's unique identifier, the data to which it belongs, and the identification fragment information. Each data shard is deeply integrated with the corresponding terminal perception information identifier, and the terminal identifier information, the data attribution identifier, the unique identifier of the shard, and the corresponding storage requirement dimension of the shard data are labeled to form a shard and information binding set, so that each data shard contains its corresponding terminal perception attribute information. Based on the node capability adaptation matrix, the data shards in the shards and information binding sets are allocated one by one to the corresponding main storage nodes. The storage location of the shard data, the unique identifier of the node, the content of the binding information and the capability adaptation dimension are recorded to generate a shard allocation record table. After receiving the data shards, the primary storage node configures the corresponding storage strategy based on the advantageous dimensions in the node's dynamic capability information. Combining the sharded data, terminal perception information identifiers, and node capability information, it generates primary storage records and sharded storage indexes, and marks the storage strategy adaptation logic. Through the terminal and node demand response link, the data fragments and corresponding binding information are synchronized to the candidate storage nodes. The candidate storage nodes perform backup storage according to the storage policy of the primary storage node, generate candidate storage records, and combine the primary storage node identifier and synchronization timestamp. Construct cross-node fragment identification association links, establish the identification correspondence relationship and data connection logic of different fragments of the same data based on terminal perception information identification and fragment identification fragments, and set fragment collaborative access rules; Configure node collaboration rules in the cross-node sharding identification association link to determine the division of labor between the primary storage node and the candidate storage node, including data read and write permissions, update synchronization responsibilities and fault tolerance processing procedures, and match the advantageous dimensions in the dynamic information of node capabilities. Extract the operation trajectory of the entire process of data sharding, storage recording, binding and combination and link construction, record the complete path of data from splitting to storage and combination, mark the node participation, link usage information and corresponding details of each link, and form a related link log; Integrate the main storage record, candidate storage record, shard storage index and shard identifier correspondence to form a preliminary set of shard identifier storage, and mark the storage status, identifier integrity and node collaboration information of each shard; Supplement key information in the initial set of fragmented identifier storage, including fragmented storage time, terminal identifier matching degree and node capability adaptation, improve the information in the set, and output the fragmented identifier storage set and associated link logs.

4. The unstructured data storage method based on a distributed architecture according to claim 1, characterized in that, The monitoring terminal application's customized requirements change. Based on the changes, the shard identifier storage set is optimized, the node capability adaptation matrix and shard identifier association link structure are adjusted accordingly, and a requirement-driven storage optimization scheme and optimization trajectory are generated, including: Establish rules for monitoring changes in customized terminal application requests, collect data in real time to collect information on adjustments to the application function requirements of terminals, including changes in access frequency, security level adjustments, storage cycle changes, and sharing scope modifications, and form a list of request changes, combined with the terminal's unique identifier. The storage information of each data fragment is extracted from the fragment identification storage set, including storage location, node identifier, terminal perception information binding relationship and fragment identification information. Combined with the operation trajectory in the associated link log, fragment storage traceability data is formed. The list of changes in demands is compared and adapted with the fragmented storage traceability data in reverse. The changes in demands corresponding to the terminal identifier are matched with the fragmented storage information. The existing storage architecture is analyzed to see if it meets the changed demands. The items that are not met are identified, recorded as adaptation gaps, and optimization directions are determined. The causes of the adaptation gaps were analyzed, including the mismatch between node capabilities and new requirements, the sharding logic not conforming to the changes in requirements, the link response rate not meeting the adjusted access frequency, and the insufficient adaptability of storage strategies, resulting in a gap cause analysis report. Based on the gap cause analysis report and optimization direction, a sharded storage optimization strategy was formulated, which determined the data shards to be adjusted, the target storage nodes, the node capability adaptation adjustment direction, and the shard identifier association link structure optimization scheme, combined with the strategy and the changes in requirements and adaptation gaps. Generate demand-driven storage optimization solutions based on the sharded storage optimization strategy, marking the optimization objects, target storage nodes, adjustment methods, timing arrangements and adaptation standards, and combining the solutions with terminal identifiers, a list of demand changes and a gap cause analysis report; Based on the demand-driven storage optimization solution, adjust the node capability adaptation matrix, update the data and node adaptation relationship, capability matching dimensions and priority ranking, and synchronously correct the link response rules in the matrix to adapt to the changed terminal demands. Adjust the cross-node shard identifier association link structure, update the identifier correspondence between shards, data connection logic and node collaboration rules based on the new storage location and changes in requirements, and optimize the shard collaborative access rules; Migrate the data shards that need to be adjusted to the target storage node, update the shard storage location, storage record and shard storage index, and record the migration details and adaptation effect in conjunction with the migration process and demand-driven storage optimization solution. Record information throughout the entire process of monitoring changes in demand, reverse adaptation, optimization plan formulation and execution, form an optimization trajectory, mark the operation content and basis of each optimization step, and output demand-driven storage optimization plan and optimization trajectory.

5. The unstructured data storage method based on a distributed architecture according to claim 1, characterized in that, The aforementioned demand-driven storage optimization scheme constructs a closed-loop storage architecture operation and maintenance system, updates dynamic information on node capabilities and shard identifier association rules, and outputs a terminal-adaptive unstructured data storage architecture and operation and maintenance logs, including: Based on demand-driven storage optimization solutions and optimization trajectories, we sort out the storage architecture operation and maintenance requirements, covering dynamic updates of node capabilities, iteration of shard identifier association rules, optimization of link response performance, and prediction of changes in demands, forming an operation and maintenance requirements list. Build dynamic update rules for node capabilities, combine changes in node hardware resources and the execution effect of demand-driven storage optimization solutions, update the advantages and disadvantages in the dynamic information of node capabilities in real time, correct node capability evaluation parameters, and synchronize the actual status of nodes. Iterate the fragment identification association rules, combine the changes in requirements and the adaptation experience in the optimization trajectory, update the identification correspondence logic between fragments, data connection rules and node collaboration rules, and improve the rules' adaptability to changes in terminal requirements. Optimize the performance of the terminal and node request response link. Based on the response performance requirements in the operation and maintenance requirement list, adjust the link transmission parameters, bandwidth allocation and synchronization timing to improve the response rate and stability of the link to changes in requests. Build an intelligent operation and maintenance closed loop for storage architecture, connecting the monitoring of changes in demand, execution of storage optimization, updating of node capabilities and iteration of rules, and setting the closed loop operation logic, including triggering conditions, execution process and feedback rules, to adapt to terminal demands; By integrating and optimizing the dynamic information of node capabilities, the iterative fragmentation identification association rules, the optimized request response link and intelligent operation and maintenance closed-loop rules, a terminal-adaptive unstructured data storage architecture is constructed, and the corresponding relationships and operating rules of each component are marked. Initiate the operation and maintenance log generation process, set the log structure, including node capability update records, rule iteration content, link optimization details, request adaptation effect and closed-loop operation trajectory modules. The log structure contains records of the entire operation and maintenance process. Extract the specific content and basis of node capability updates, rule iterations, and link optimizations, fill them into the corresponding log module, and record the parameter changes, adaptation and upgrade points, and operation and maintenance experience accumulation before and after optimization. Organize information on the effectiveness of terminal requirement adaptation, including the degree of requirement fulfillment, storage performance improvement, changes in link response rate, and operation and maintenance closed-loop status, and fill it into the corresponding log module; Output terminal-adaptive unstructured data storage architecture and operation and maintenance logs, synchronously combining information from each component of the architecture and log content.

6. The unstructured data storage method based on a distributed architecture according to claim 2, characterized in that, The extended request response link to the candidate storage node constructs a full-link response structure between the terminal, the master node, and the candidate nodes, and sets the link data flow logic, including data synchronization direction, response timing, and load distribution method, matching the access response requirements in the data storage requirement list, including: Collect the communication interface parameters, protocol types, and transmission capabilities between the primary storage node and candidate storage nodes; determine the range of transmission rates, data encapsulation formats, and connection stability that can be supported between nodes; form a list of communication capabilities between nodes; and combine this with the unique identifier of each node. Based on the configuration parameters of the basic link for responding to demands from terminals and nodes, the link is extended to each candidate storage node to build a hierarchical full-link response structure. A unique identifier is assigned to each extended link, and the extended link identifier is combined with the identifiers of the main storage node, candidate storage node, and terminal. Based on the access response requirements in the data storage requirement list, the data synchronization direction in the link data flow logic is set, and a flow mode is adopted in which the main storage node synchronizes core shard data unidirectionally to the candidate storage node and the candidate storage node feeds back storage status and load information bidirectionally to the main storage node, and the flow rules are marked. Set the link response timing, set differentiated synchronization timing according to the access frequency of different data shards, configure the corresponding synchronization interval for the corresponding access frequency range, and match the response time requirements in the data storage requirement list; Develop a link load distribution method. Based on the processing capacity and network bandwidth resources in the dynamic information of node capabilities, allocate the data transmission load of each link and formulate a load distribution strategy. When the link load reaches a preset threshold, some data transmission tasks will be reallocated to preset low-load links. Configure link fault-tolerant flow rules, set flow adjustment rules when link failure, node abnormality and overload occur, determine the conditions for enabling backup links, data switching sequence and fault recovery process, with the goal of achieving data flow continuity and response stability, configure link fault-tolerant flow rules; By combining the link data flow logic with the node capability adaptation matrix, ensure that the synchronization direction, response timing, and load distribution method are consistent with the data and node capability adaptation dimensions. When setting the link data flow logic, ensure that it is consistent with the requirements in the data storage requirement list. A link status monitoring module is constructed to collect the transmission rate, load, connection status and fault information of each link in real time. Combined with the unique link identifier and dynamic node capability information, it provides data support for load distribution adjustment and fault tolerance processing. Simulate the link operation status under different access scenarios, verify the feasibility of data flow logic, response timing and load distribution methods, and adjust link parameters based on monitoring data to optimize the adaptability of the entire link response structure. Output the configured end-to-end response structure of the terminal, master node, and candidate nodes, and simultaneously organize the link identifier, correspondence, data flow logic, and monitoring module configuration information to supplement the node capability adaptation matrix.

7. The unstructured data storage method based on a distributed architecture according to claim 3, characterized in that, The construction of cross-node fragment identifier association links, based on terminal-perceived information identifiers and fragment identifier fragments, establishes the identifier correspondence relationship and data connection logic for different fragments of the same data, and sets fragment collaborative access rules, including: Extract all data fragments and terminal perception identifier fragments corresponding to the same data identifier in the fragment and information binding set, classify them into a single data fragment cluster, and label the unique data identifier, total number of fragments and core features of terminal perception information. Based on terminal perception information identifiers and segment identifier fragments, analyze the identifier correspondence of each segment of the same data, extract common identifier features and unique identifier fragments, determine the identifier correspondence relationship between segments, establish an identifier correspondence index, and record the corresponding dimension and corresponding segment identifier. By combining the original characteristics and content logic of the data, establish the data connection logic between the segments, determine the content connection nodes, data integration order and cross-node connection rules of each segment, and achieve the data integrity after segment integration is consistent with the original data. Define the cross-node call logic in the shard collaborative access rules, determine the call order, data transmission path and node collaboration method of associated shards when the terminal accesses a certain shard, and adapt to the access requirements in the customized requirements of terminal applications; Define read and write permission allocation in the sharded collaborative access rules. Based on the security capabilities in the dynamic information of node capabilities and the security level in the terminal requirements, allocate read and write permissions for each node to the sharded data. Configure full read and write permissions for the primary storage node and limited read, write and backup permissions for the candidate storage node. By combining the corresponding index of the identifier, the data connection logic and the sharding collaborative access rules, a core framework for cross-node sharding identifier association links is constructed, which marks the link nodes, corresponding links and access rules, and combines the unique data identifier and the terminal perception information identifier. The core framework of cross-node shard identifier association link is synchronized to all corresponding storage nodes to ensure that each node knows the identifier correspondence between shards, the data connection logic and the shard collaborative access rules, and supports cross-node collaborative operation. Configure dynamic link adjustment rules, and set link structure adjustment trigger conditions based on changes in terminal application customization requirements and node capability updates, including changes in requirements, node anomalies and shard migration, to adapt to storage architecture optimization needs. Output the completed cross-node shard identifier association link, and simultaneously organize the corresponding index of the identifier, data connection logic, shard collaborative access rules and dynamic adjustment rules.

8. The unstructured data storage method based on a distributed architecture according to claim 4, characterized in that, The migration requires adjusting data shards to the target storage node, updating shard storage locations, storage records, and shard storage indexes, and recording migration details and adaptation effects in conjunction with the migration process and demand-driven storage optimization solutions, including: Analyze demand-driven storage optimization solutions, extract the data shard identifiers to be migrated, the original storage node identifiers, the target storage node identifiers, the migration sequence and adaptation standards, and form a shard migration execution list. Combine the list with the optimization objects and adaptation gaps in the solution. Based on the fragmented migration execution list, a dedicated migration channel is built through the terminal and node request response link. The channel transmission parameters are configured to match the size of the data fragments and the terminal perception identification fragments and the node network bandwidth capabilities. The migration priority is set and the data fragments are migrated in order of access frequency. The original storage node reads the data fragments to be migrated and their corresponding binding information, performs data encapsulation processing according to the adaptation standard, and the encapsulation content includes data fragments, terminal perception identification fragments, binding information content and storage requirement dimensions, generates a migration data packet, and marks the data packet with the data packet ownership identifier and identifier integrity information; The migration data packet is transmitted to the target storage node through a dedicated migration channel. After receiving the data packet, the target storage node performs unpacking and storage operations according to the advantageous dimensions and adaptation standards in the node capability dynamic information. Combined with data fragmentation, terminal perception information identification and target storage node capability information, a target storage record is generated. The original storage node deletes the data fragments that have been migrated, updates its own storage records and remaining hardware resource status, marks the data migration completion identifier, target storage node information and migration timestamp, generates a migration completion record, and synchronously updates the storage load parameters in the node's dynamic capacity information. Synchronously update the shard storage index, correct the data storage location, node identifier, binding information content and identifier correspondence of the migrated shards, and update the shard storage index to reflect the actual storage status after data migration; By combining the migration process with demand-driven storage optimization solutions, the optimization objects, adaptation gaps and optimization goals corresponding to the migrated shards are recorded, and the parameter adjustments, link usage and node collaboration during the migration process are marked to form a migration detail record table. Analyze the data sharding storage adaptation effect after migration, compare the access response rate, node capability adaptation degree and terminal demand satisfaction before and after migration, form an adaptation effect analysis table, and combine it with the migration detail record table; By synchronizing the migration results and adaptation effects of the terminal and node request response links to all corresponding storage nodes, the shard storage locations and corresponding relationships in the cross-node shard identifier association links are updated, and the relevant information in the cross-node shard identifier association links is updated to maintain the consistency of collaborative operations among nodes. Output the data sharding migration execution results, and simultaneously organize the migration details record table, adaptation effect analysis table, sharding storage index update information and node capability adjustment data, and supplement them to the optimization trajectory.

9. The unstructured data storage method based on a distributed architecture according to claim 5, characterized in that, The aforementioned dynamic update rules for node capabilities, combined with changes in node hardware resources and the execution effect of demand-driven storage optimization solutions, update the strengths and weaknesses in the dynamic information of node capabilities in real time, correct node capability evaluation parameters, and synchronize the actual status of nodes, including: Establish real-time data collection rules for node hardware resources, set the collection frequency and collection dimensions. The collection dimensions cover the node's remaining storage capacity, read and write processing rate, network bandwidth usage, security protection status and hardware operating parameters, forming real-time data of node hardware resources, combined with the node's unique identifier. By combining real-time data on node hardware resources with the execution effect of demand-driven storage optimization solutions, the impact of the optimization solutions on node hardware resources is analyzed, including changes in storage load, processing capacity usage, and bandwidth consumption, resulting in a resource impact analysis report. Based on the resource impact analysis report, the strength and weakness dimensions in the node capability dynamic information are adjusted. When the node storage load decreases and the processing speed increases, the corresponding capability strength dimension is strengthened. When the load is too high and the processing speed decreases, the weakness dimension label is updated and the capability assessment description is corrected simultaneously. The node capability assessment parameters were revised. Based on real-time data of node hardware resources and the results of resource impact analysis, the assessment indicators of each capability dimension were adjusted, including storage capacity threshold, read / write speed benchmark and bandwidth adaptation range. The node capability assessment parameters were revised so that they are calculated based on real-time data of node hardware resources. Differentiated early warning standards are configured for the weak and strong dimensions of each capability dimension. When the node hardware resource parameters exceed the node capability early warning threshold, a capability adjustment prompt is triggered, which is combined with the dynamic information of node capabilities and real-time data of hardware resources. Construct a capability update feedback link to synchronize the updated dynamic information of node capabilities to the node capability adaptation matrix, the cross-node sharding identifier association link, and the terminal and node request response link, and synchronize the updated dynamic information of node capabilities to relevant components, so that each component can perform operations based on the latest information; Record the dynamic update process of node capabilities, including data collection, impact analysis, dimension adjustment, parameter correction and feedback synchronization details, to form a capability update log, combined with the node's unique identifier and update timestamp; By combining capability update logs with the intelligent operation and maintenance closed loop of storage architecture, the reasons for update triggering, execution process and impact on the operation and maintenance closed loop are marked. Regularly calibrate the node capability assessment framework, and adjust the capability assessment dimensions and parameter standards in combination with changes in the customized requirements of terminal applications and the optimization effect of terminal-adaptive unstructured data storage architecture to adapt to continuously changing storage needs. Output the completed node capability dynamic update rules, synchronously organize resource collection rules, early warning threshold configurations, update feedback links and capability update logs, and combine them with the terminal-adaptive unstructured data storage architecture to achieve dynamic adaptation of node capabilities.

10. An unstructured data storage system based on a distributed architecture, characterized in that, It includes a processor and a readable storage medium storing a program that, when executed by the processor, implements the unstructured data storage method based on a distributed architecture as described in any one of claims 1-9.

Images