Node time processing method and video storage system

Abstract

The invention discloses a node time processing method and a video storage system. The processing method comprises the following steps: respectively detecting time differences corresponding to a plurality of nodes from at least two dimensions to obtain at least two time differences; wherein the at least two dimensions comprise at least two of a system layer dimension, a business layer dimension and a metadata dimension; performing node time anomaly detection based on the at least two time differences; and when the target node is detected to have the time abnormality, performing time restoration on the target node. By means of the mode, higher-precision time synchronization anomaly detection can be achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of node time processing, and particularly relates to a node time processing method and a video storage system. BACKGROUND

[0002] Distributed storage stores data in multiple nodes to achieve high scalability and fault tolerance. In the field of video cloud storage, it combines object storage to process massive video files, uses block storage, multiple copies and erasure code to ensure the reliability and read-write performance of videos, and uses metadata indexing to accelerate timestamp positioning and intelligent retrieval (such as security scene second-level retrieval), balance low-latency access and storage costs, and meet the high-concurrency demand in live streaming and playback scenarios.

[0003] In the related art, distributed cluster time synchronization synchronizes local clocks of multiple nodes to ensure global time consistency to support file storage, log tracking and other key operations. When a single node time synchronization in the cluster fails, it will cause cluster authentication failure and affect the availability of the faulty node. The traditional solution relies on single-dimensional time verification or manual intervention, which has high missed detection rate, long isolation delay and low recovery efficiency. SUMMARY

[0004] The present application provides a node time processing method and a video storage system, which can realize higher precision time synchronization anomaly detection.

[0005] To solve the above technical problems, one technical solution adopted by the present application is to provide a node time processing method, which comprises: detecting time differences corresponding to a plurality of nodes from at least two dimensions respectively to obtain at least two time differences; wherein the at least two dimensions include at least two of system layer dimension, business layer dimension and metadata dimension; performing node time anomaly detection based on the at least two time differences; and performing time repair on a target node when detecting that the target node has time anomaly.

[0006] The at least two dimensions include system layer dimension, business layer dimension and metadata dimension; detecting time differences corresponding to a plurality of nodes from at least two dimensions respectively to obtain at least two time differences comprises: detecting time differences corresponding to a plurality of nodes from system layer dimension to obtain first time differences; detecting time differences corresponding to a plurality of nodes from business layer dimension to obtain second time differences; and detecting time differences corresponding to a plurality of nodes from metadata dimension to obtain third time differences; and performing node time anomaly detection based on the at least two time differences comprises: performing node time anomaly detection based on the first time differences, the second time differences and the third time differences.

[0007] The process of detecting the time difference between several nodes from the system layer to obtain the first time difference includes: periodically acquiring the time difference between each node and the clock source to obtain a preset number of time differences; calculating the standard deviation based on the preset number of time differences, and using the standard deviation as the first time difference.

[0008] The second time difference is obtained by detecting the time difference corresponding to several nodes from the business layer dimension. This includes: for each node, extracting the timestamp of each video frame from its corresponding video stream, and calculating the time difference using the timestamps between adjacent video frames to obtain the second time difference.

[0009] The process involves detecting the time difference between several nodes from the metadata dimension to obtain the third time difference. This includes: recording the current timestamp when each node initiates a write operation, and obtaining the global timestamp updated after the transaction is committed to ETCD; calculating the time difference based on the current timestamp and the global timestamp to obtain the third time difference.

[0010] Among them, the node time anomaly detection based on the first time difference, the second time difference and the third time difference includes: in response to the first time difference being greater than the first time threshold, the second time difference being greater than the second time threshold, and the third time difference being greater than the third time threshold, determining that the target node corresponding to the first time difference, the second time difference and the third time difference has a time anomaly.

[0011] Specifically, when a time anomaly is detected in a target node, time repair is performed on the target node, including: weighted summation of the first time difference, the second time difference, and the third time difference to obtain the total time difference; soft synchronization correction of the target node's time is performed in response to the total time difference being less than a fourth time threshold; and isolation and time repair of the target node are performed in response to the total time difference being greater than or equal to the fourth time threshold.

[0012] The process of performing soft synchronization correction on the target node includes: setting a baseline correction rate based on the total time difference; performing soft synchronization correction using the total time difference and the baseline correction rate, detecting the system load during the correction process, correcting the baseline correction rate based on the system load, and continuing to perform soft synchronization correction using the corrected baseline correction rate until the soft synchronization correction result meets the preset requirements.

[0013] The process of isolating and repairing the target node includes: isolating the target node for business operations and network operations; and repairing the target node for time after isolation is completed.

[0014] The process of isolating the target node for business and network includes: calling the ETCD lease management interface to revoke the target node's lease permissions, terminating all incomplete transaction operations of the target node, removing the target node from the Raft protocol's voting member list, and freezing the target node's permissions; after the permissions are frozen, all outgoing traffic of the target node is discarded.

[0015] The process of time-based repair of the target node includes: conducting stability tests on the target node; and, after the stability tests are passed, gradually restoring the business traffic of the target node using a progressive, segmented, incremental algorithm.

[0016] The process involves using a progressive, segmented, incremental algorithm to gradually restore the service traffic of the target node. This includes: setting an initial weight for the target node; distributing service traffic to the target node according to the initial weight and detecting the node load of the target node; adjusting the initial weight according to the node load and continuing to distribute service traffic to the target node according to the initial weight until the initial weight is 1 and the node load is stable.

[0017] To solve the above-mentioned technical problems, another technical solution adopted in this application is to provide a video storage system, which includes several nodes, and to perform node time processing on each node using the processing method provided by the above technical solution.

[0018] Among them, several nodes include storage nodes and management nodes.

[0019] The node time processing method and video storage system provided in this application obtain at least two time differences by detecting the time differences corresponding to several nodes from at least two dimensions. The at least two dimensions include at least two of the system layer dimension, business layer dimension, and metadata dimension. Node time anomaly detection is performed based on the at least two time differences. When a time anomaly is detected in a target node, time repair is performed on the target node. This method overcomes the limitations of a single detection dimension and reduces the false negative rate by cross-validating detection through at least two dimensions, thus achieving higher accuracy in time synchronization anomaly detection. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1 This is a flowchart illustrating an embodiment of the node time processing method provided in this application; Figure 2This is a flowchart illustrating another embodiment of the node time processing method provided in this application; Figure 3 yes Figure 2 A flowchart of an embodiment of step 25; Figure 4 yes Figure 3 A flowchart illustrating an embodiment of step 252; Figure 5 yes Figure 3 A flowchart of an embodiment of step 253; Figure 6 yes Figure 5 A flowchart of an embodiment of step 51; Figure 7 This is a flowchart illustrating an embodiment of time repair of the target node provided in this application; Figure 8 yes Figure 7 A flowchart illustrating an embodiment of step 72; Figure 9 This is a schematic diagram of an embodiment of the video storage system provided in this application. Detailed Implementation

[0021] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only for explaining this application and not for limiting it. Furthermore, it should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, not all structures. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] Distributed storage achieves high scalability and fault tolerance by distributing data across multiple nodes. In the field of video cloud storage, it combines object storage to process massive video files, utilizes block storage, multiple replicas, and erasure coding to ensure video reliability and read / write performance; it accelerates timestamp positioning and intelligent retrieval through metadata indexing (such as second-level access in security scenarios), balances low-latency access with storage costs, and meets the high-concurrency requirements of live streaming and playback scenarios.

[0024] In related technologies, distributed cluster time synchronization ensures global time consistency by coordinating the local clocks of multiple nodes to support critical operations such as file storage and log tracing. When the time synchronization of a single node in the cluster fails, cluster authentication will fail, impacting the availability of the faulty node. Traditional solutions rely on single-dimensional time verification or manual intervention, which suffers from drawbacks such as high false negative rates, long isolation delays, and low recovery efficiency.

[0025] Based on this, this application proposes a method to obtain at least two time differences by detecting the time differences corresponding to several nodes from at least two dimensions; wherein the at least two dimensions include at least two of the system layer dimension, business layer dimension, and metadata dimension; node time anomaly detection is performed based on the at least two time differences; when a time anomaly is detected in a target node, time repair is performed on the target node. This method overcomes the limitations of a single detection dimension and reduces the false negative rate by cross-validating detection through at least two dimensions, thus achieving higher accuracy in time synchronization anomaly detection. See any of the following embodiments for details.

[0026] See Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the node time processing method provided in this application. Applied to a storage system comprising several nodes, the processing method includes: Step 11: Detect the time difference corresponding to several nodes from at least two dimensions to obtain at least two time differences; wherein, the at least two dimensions include at least two of the system layer dimension, business layer dimension and metadata dimension.

[0027] In some embodiments, the storage system may be a distributed storage system consisting of several nodes.

[0028] In some embodiments, the storage system comprises a plurality of storage nodes. Based on this, the processing method of this application is executed by the storage nodes. For example, a master node is determined from the plurality of storage nodes, and the remaining nodes are slave nodes. The master node takes the lead, and the slave nodes cooperate to complete the node time processing. For example, each storage node can detect the time differences corresponding to the plurality of nodes from at least two dimensions, obtain at least two time differences, and then send these time differences to the master node so that the master node can perform subsequent operations. That is, the master node can obtain the time differences corresponding to the plurality of nodes.

[0029] In some embodiments, the storage system comprises several storage nodes and several management nodes, with one management node corresponding to at least one storage node. Based on this, the processing method of this application is executed by the management node. For example, the management node takes the lead, and the storage nodes cooperate to complete the node time processing. For instance, each storage node can detect the time differences corresponding to several nodes from at least two dimensions, obtain at least two time differences, and then send these time differences to the management node so that the management node can perform subsequent operations. That is, the management node can obtain the time differences corresponding to several storage nodes.

[0030] In some embodiments, the storage system may be a data storage system such as a video storage system or a file storage system.

[0031] In some embodiments, at least two dimensions include a system-level dimension and a business-level dimension. The time differences corresponding to several nodes are detected from both the system-level and business-level dimensions to obtain two time differences.

[0032] In some embodiments, at least two dimensions include a system-level dimension and a metadata dimension. The time difference corresponding to several nodes is detected from both the system-level dimension and the metadata dimension to obtain two time differences. Metadata refers to data that describes data, such as a description of a video.

[0033] In some embodiments, at least two dimensions include a business layer dimension and a metadata dimension. The time difference corresponding to several nodes is detected from both the business layer dimension and the metadata dimension to obtain two time differences.

[0034] In some embodiments, at least two dimensions include at least two of the following: system layer dimension, business layer dimension, and metadata dimension. The time differences corresponding to several nodes are detected from the system layer dimension, business layer dimension, and metadata dimension respectively, resulting in three time differences.

[0035] In some embodiments, NTP (Network Time Protocol) time offset technology can be used to detect the time difference between several nodes at the system layer level, video frame timestamp continuity verification can be used to detect the time difference between several nodes at the business layer level, and file modification time can be used to detect the time difference between several nodes at the metadata level.

[0036] In some embodiments, for each node in the storage system (video storage system), the time difference corresponding to several nodes can be detected from at least two dimensions to obtain at least two time differences.

[0037] Step 12: Detect node time anomalies based on at least two time differences.

[0038] In some embodiments, node time anomaly detection can be performed on the corresponding time difference according to the time anomaly detection logic of its corresponding dimension. That is, at least two detection results can be obtained.

[0039] For example, at least two detection results are included: a first detection result and a second detection result. If both the first and second detection results indicate that the target node exhibits a time anomaly, then the target node exhibits a time anomaly. If the first detection result indicates that the target node exhibits a time anomaly, but the second detection result indicates that the target node does not exhibit a time anomaly, then the target node does not exhibit a time anomaly. If both the first and second detection results indicate that the target node does not exhibit a time anomaly, then the target node does not exhibit a time anomaly.

[0040] For example, at least two detection results are required, including a first detection result, a second detection result, and a third detection result. If the first detection result indicates that the target node has a time anomaly, the second detection result also indicates that the target node has a time anomaly, and the third detection result also indicates that the target node has a time anomaly. Otherwise, it indicates that the target node does not have a time anomaly.

[0041] Step 13: When a time anomaly is detected in the target node, perform time repair on the target node.

[0042] In some embodiments, when a time anomaly is detected in the target node, time repair is performed on the target node so that the time of the target node can be synchronized with the system time.

[0043] In some embodiments, the storage system consists of a plurality of storage nodes. Therefore, steps 12 and 13 are executed by the master node among the storage nodes.

[0044] In some embodiments, the storage system comprises a plurality of storage nodes and a plurality of management nodes. Therefore, steps 12 and 13 are performed by the management nodes.

[0045] In this embodiment, at least two time differences are obtained by detecting the time differences corresponding to several nodes from at least two dimensions. The at least two dimensions include at least two of the system layer dimension, business layer dimension, and metadata dimension. Node time anomaly detection is performed based on the at least two time differences. When a time anomaly is detected in a target node, time repair is performed on the target node. This method overcomes the limitations of a single detection dimension and uses cross-validation through detection from at least two dimensions to reduce the false negative rate and achieve higher accuracy in time synchronization anomaly detection. This solves the problem of high time sensitivity of storage services in the video acquisition field.

[0046] See Figure 2 ,Figure 2 This is a flowchart illustrating another embodiment of the node time processing method provided in this application. Applied to a video storage system comprising several nodes, the processing method includes: Step 21: Detect the time difference corresponding to several nodes from the system layer dimension to obtain the first time difference.

[0047] In some embodiments, for each node, the time difference between it and the clock source is periodically acquired to obtain a preset number of time differences; the standard deviation is calculated based on the preset number of time differences, and the standard deviation is used as the first time difference.

[0048] In some embodiments, each node periodically (every 1 second) obtains the time difference (offset) with the NTP time source (clock source) via the NTP protocol. The system maintains a circular queue of fixed length 10 to store the 10 most recent time differences. When the window (circular queue) is full, the standard deviation σ of the data within the window (circular queue) is calculated in real time and used as the first time difference. If σ is greater than 50 ms for three consecutive calculations, a primary alarm is triggered. In some embodiments, the length of the circular queue can be set according to actual conditions, such as a fixed length of 8, 12, or 15. In some embodiments, a primary alarm can be triggered if σ is greater than 50 ms for four or five consecutive calculations.

[0049] In some embodiments, since the comparison is performed on continuously calculated σ, the count is reset to zero when the current σ is less than or equal to 50ms.

[0050] Step 22: Detect the time difference corresponding to several nodes from the business layer dimension to obtain the second time difference.

[0051] In some embodiments, for each node, the timestamp of each frame of video is extracted from its corresponding video stream, and the time difference is calculated using the timestamps between adjacent video frames to obtain a second time difference. The node stores the video. The timestamp can be the decoding timestamp or the rendering timestamp of each video frame.

[0052] In some embodiments, the decoding timestamp (DTS) or presentation timestamp (PTS) of each frame of video is extracted from the video stream with millisecond-level precision. The timestamp of the first frame is ignored because there is no previous frame for comparison. When extracting subsequent frames, the time difference with the previous frame is calculated. Implement anomaly detection. Among them, This represents the timestamp of the nth frame of the video. This represents the timestamp of the (n-1)th frame of the video.

[0053] In some embodiments, the theoretical frame interval can be calculated based on the video frame rate (FPS, frames per second). Dynamically set the allowed fluctuation range When detected When the threshold is exceeded, a secondary alarm is triggered and a compensation mechanism is initiated, generating a timestamp for the missing frame through linear interpolation. This ensures the continuity of subsequent verifications.

[0054] Step 23: Detect the time difference corresponding to several nodes from the metadata dimension to obtain the third time difference.

[0055] In some embodiments, when each node initiates a write operation, the current timestamp is recorded, and the global timestamp updated after the transaction is committed to ETCD is obtained; the time difference is calculated based on the current timestamp and the global timestamp to obtain a third time difference.

[0056] In some embodiments, ETCD is used as a distributed key-value store to maintain a global modification timestamp for files. When a node initiates a write operation, it records the current timestamp. And commit the transaction to ETCD to obtain the updated global timestamp. Calculate the absolute deviation If a node transaction fails to commit, the global timestamp updated by other nodes will be used. calculate ,when If an alarm is triggered, the node will be marked as abnormal.

[0057] ETCD is a distributed key-value store that uses the Raft algorithm to achieve high availability and strong consistency. It is commonly used to store and synchronize configuration data in distributed systems.

[0058] The core features of ETCD are as follows: Distributed storage: Utilizes the Raft algorithm to ensure data consistency across multiple nodes, supporting cluster mode (typically 3 or 5 nodes) to guarantee data reliability. Consistency Protocol: The Raft algorithm is used to elect a leader node. All data write operations are handled uniformly by the leader node, ensuring strong data consistency. Application scenarios: Primarily used for configuration management, service discovery, distributed locks, and other similar scenarios, such as Kubernetes cluster state storage.

[0059] Step 24: Detect node time anomalies based on the first time difference, the second time difference, and the third time difference.

[0060] In some embodiments, in response to a first time difference being greater than a first time threshold, a second time difference being greater than a second time threshold, and a third time difference being greater than a third time threshold, it is determined that a time anomaly has occurred in the target node corresponding to the first time difference, the second time difference, and the third time difference.

[0061] Step 25: When a time anomaly is detected in the target node, perform time repair on the target node.

[0062] In some embodiments, time repair of the target node can be performed manually.

[0063] In some embodiments, time repair of the target node can be performed through automatic repair.

[0064] In some embodiments, see Figure 3 Step 25 can be the following process: Step 251: Perform a weighted summation of the first time difference, the second time difference, and the third time difference to obtain the total time difference.

[0065] In some embodiments, since the time anomaly of the target node is determined by at least two time differences, there may be different first, second, and third time differences, which cannot be quantified. Based on this, the first, second, and third time differences can be weighted and summed to obtain the total time difference.

[0066] In some embodiments, a first weight can be assigned to a first time difference, a second weight can be assigned to a second time difference, and a third weight can be assigned to a third time difference. The third weight is the largest. The first and second weights can be the same or different. For example, the first weight is 0.2, the second weight is 0.2, and the third weight is 0.6. For example, the first weight is 0.1, the second weight is 0.2, and the third weight is 0.7. For example, the first weight is 0.2, the second weight is 0.1, and the third weight is 0.7. For example, the first weight is 0.3, the second weight is 0.3, and the third weight is 0.4.

[0067] Step 252: In response to the total time difference being less than the fourth time threshold, perform soft synchronization correction on the time of the target node.

[0068] In this embodiment, different time correction or time repair strategies can be applied based on the total time difference. For example, in response to the total time difference being less than the fourth time threshold, soft synchronization correction is performed on the time of the target node. In some embodiments, see Figure 4 Step 252 can be the following process: Step 41: Set the baseline correction rate based on the total time difference.

[0069] In some embodiments, a baseline correction rate can be set based on the total time difference and the planned correction time.

[0070] For example, the reference correction rate , This indicates the planned correction time, which can be set as needed. This represents the total time difference.

[0071] Step 42: Perform soft synchronization correction using the total time difference and the baseline correction rate, and detect the system load during the correction process.

[0072] In some embodiments, soft synchronization continuously adjusts the system clock through micro-steps, avoiding the time jumps of traditional hard synchronization that could cause certain time-sensitive services to become unavailable. The core correction principle of smoothly adjusting the system clock through micro-steps is as follows: .

[0073] in This represents the remaining clock offset at time t. Indicates the initial time deviation (total time difference) (unit: ms). Indicates the correction rate.

[0074] Step 43: Adjust the baseline correction rate according to the system load, and continue to perform soft synchronization correction using the adjusted baseline correction rate until the soft synchronization correction result meets the preset requirements.

[0075] In some embodiments, the correction process can be dynamically adjusted based on the current CPU / memory load of the system. By setting a load threshold, the correction rate is increased when the current system load exceeds the threshold, and vice versa. That is, the following formula can be used for correction:

[0076] in This indicates the corrected baseline correction rate. This is the damping coefficient, typically set to 0.5 to suppress oscillations. Indicates the current CPU / memory load. This is the load threshold. When... The soft synchronization time correction ends when the time is less than 5ms. That is, during the correction process, it can be utilized... As mentioned above .

[0077] Step 253: In response to the total time difference being greater than or equal to the fourth time threshold, isolate and time-repair the target node.

[0078] In some embodiments, in response to a total time difference greater than or equal to a fourth time threshold, i.e., when an abnormal time of the target node is detected, the system will trigger an isolation decision process. Through the Raft protocol, the master node initiates an isolation request for the abnormal node (target node), refusing it participation in subsequent decision voting and write permissions, marking the node as "untrusted", and simultaneously triggering subsequent service and network isolation.

[0079] In some embodiments, see Figure 5 Step 253 can be the following process: Step 51: Perform service isolation and network isolation on the target node.

[0080] In some embodiments, see Figure 6 Step 51 can be the following process: Step 61: Call the ETCD lease management interface to revoke the lease permissions of the target node and terminate all incomplete transaction operations of the target node.

[0081] In some embodiments, after the cluster detects a node abnormality, the system immediately revokes the lease permissions of the abnormal node (target node) by calling the ETCD lease management interface, and forcibly terminates all its currently incomplete transaction operations.

[0082] Step 62: Remove the target node from the Raft protocol's voting member list and freeze the target node's permissions.

[0083] In some embodiments, the cluster member configuration is modified to remove the abnormal node from the Raft protocol's voting member list, prohibiting it from participating in subsequent data writing or consistency negotiation. To ensure operational reliability, the permission freeze process is implemented using atomic transactions, and after the operation is completed, the ETCD member list and lease status are queried to ensure that the abnormal node can no longer initiate or respond to any requests involving data consistency.

[0084] Step 63: After the permissions are frozen, discard all outgoing traffic from the target node.

[0085] In some embodiments, after the permission freeze command takes effect, the system dynamically distributes OpenFlow flow table rules to all switches in the network through the SDN controller. The rule content matches the IP address and port information of the abnormal node, and sets the action to drop all its outbound traffic within milliseconds, completely blocking the abnormal node from communicating with other members of the cluster. Simultaneously, the load balancing service updates the backend node list in real time, switching the service traffic originally directed to the abnormal node to predefined backup nodes. The SDN controller is an application in Software-Defined Networking (SDN) responsible for flow control to ensure intelligent networking. The SDN controller is based on protocols such as OpenFlow, allowing servers to tell switches where to send data packets.

[0086] In some embodiments, the collaborative operation of SDN network isolation can be frozen based on database permissions, which can achieve millisecond-level blocking of abnormal nodes. By calling the ETCD interface to freeze distributed transaction permissions and blocking the outbound traffic of abnormal nodes by the SDN controller, dual isolation of the business layer and network layer is achieved respectively, reducing the isolation latency to less than 5ms, reducing the risk of dirty data spread, and effectively avoiding the pollution of cluster data by erroneous timestamps.

[0087] Step 52: Perform time repair on the target node after isolation is completed.

[0088] In some embodiments, the root cause of a target node (such as an NTP service failure or a hardware clock malfunction) is located through log analysis, and then time repair is performed on the target node based on the root cause.

[0089] In some embodiments, see Figure 7 After performing time repair on the target node, the process can be as follows: Step 71: Perform a stability test on the target node.

[0090] In some embodiments, after the target node is repaired, it must pass a stability test before it can reconnect to services. The target node sends 10 consecutive timestamp requests to the reference time source, requiring a single error... Overall variance Then you can pass the verification.

[0091] Step 72: After the stability test is passed, the service traffic of the target node is gradually restored using a progressive segmented incremental algorithm.

[0092] For example, in the video capture industry, the early morning hours are generally the off-peak hours for pedestrian traffic on various streets. Performing data migration tasks at this time can minimize the impact on real-time services. A progressive, segmented, incremental algorithm is used to gradually restore node service traffic, avoiding the impact on the network when the full video stream is restored.

[0093] In some embodiments, seeFigure 8 Step 72 can be the following process: Step 81: Set the initial weights for the target node.

[0094] Step 82: Distribute service traffic to the target node according to the initial weight and detect the node load of the target node.

[0095] Step 83: Adjust the initial weight according to the node load, and continue to distribute service traffic to the target node according to the initial weight until the initial weight is 1 and the node load is stable.

[0096] The dynamic weighted traffic redirection strategy achieves seamless service recovery through a phased, load-sensitive, and gradual traffic takeover mechanism. After a node is repaired, the load balancer sets differentiated initial weights based on service priority. The weights are then gradually increased in 30-second intervals using the following method: .

[0097] in Let the weights be a function of time. This is the initial weight (ranging from 10% to 20%). Simultaneously, node resource load is monitored in real time; when a node load becomes too high, a rate-down rule is triggered. When the weight reaches 100% and the load remains stable below the threshold for three consecutive cycles, the tagging process is complete.

[0098] See Figure 9 , Figure 9 This is a schematic diagram of an embodiment of the video storage system provided in this application. The video storage system 90 includes several nodes, and each node is processed using the processing method provided in the above technical solution.

[0099] This includes several nodes, including storage nodes and management nodes. Management nodes are used to store the metadata of the storage nodes and to manage them accordingly.

[0100] In some embodiments, each node is equivalent to an electronic device, which is equipped with a processor and a memory. The memory is used to store relevant data, such as video data, corresponding management programs, and computer programs. The processor performs node time processing on each node based on the processing method provided by the above technical solution, according to the computer program. If the node is a storage node, its memory capacity can be very large, such as consisting of several hard disks.

[0101] In summary, the node time processing method and video storage system provided in this application obtain at least two time differences by detecting the time differences corresponding to several nodes from at least two dimensions. These at least two dimensions include at least two of the following: system layer dimension, business layer dimension, and metadata dimension. Node time anomaly detection is performed based on these at least two time differences. When a time anomaly is detected in a target node, time repair is performed on the target node. This approach overcomes the limitations of a single detection dimension by using cross-validation through detection from at least two dimensions, reducing the false negative rate and achieving higher accuracy in time synchronization anomaly detection. This addresses the high time sensitivity of storage services in the video acquisition industry.

[0102] Furthermore, this application generates a composite alarm chain based on a multi-dimensional collaborative detection mechanism. In video acquisition scenarios, it achieves cross-validation by integrating the system layer (NTP offset sliding window statistics), the business layer (video frame interval verification), and metadata (file modification time), overcoming the limitations of a single detection dimension. An alarm is triggered when three consecutive offsets at the system layer exceed 50ms, a secondary alarm is triggered when metadata differences exceed 30ms, and a composite alarm chain is generated when the frame interval at the business layer suddenly increases by more than 200ms, accurately capturing hidden faults such as clock drift.

[0103] Furthermore, this application achieves dual isolation of services and the network through database permission freezing and SDN-based collaborative isolation technology. Database permissions are frozen, disabling data writing by abnormal nodes, and flow table rules are dynamically issued through the SDN controller to discard outgoing traffic from abnormal nodes. The load is synchronously switched to backup nodes, effectively improving the system's high availability and data consistency.

[0104] Furthermore, this application employs a dynamic, graded self-healing mechanism based on deviation classification. For minor deviations less than 100ms, soft synchronization adjustments are triggered, dynamically adjusting the recovery rate according to system load and other factors. When the deviation exceeds 100ms, the root cause is immediately isolated and located. After node recovery, it must pass 10 challenge tests (error < 10ms, variance < 5ms²). Upon verification, traffic is automatically redirected, and the synchronization rate is dynamically adjusted according to business load to ensure stability under resource contention scenarios.

[0105] Furthermore, this application achieves secure regression of abnormal nodes through a dynamic load weight strategy. After a node passes the challenge-response test, a traffic redirection mechanism is triggered. According to preset rules (load weight linearly increases from 20% to 100%), service traffic is restored in stages, while node stability (such as timestamp error and resource load rate) is monitored in real time, and the recovery rate is dynamically adjusted. Its innovation lies in gradual traffic control, avoiding instantaneous load shocks that could cause network congestion.

[0106] Furthermore, this application overcomes the limitations of traditional single-dimensional detection by integrating joint verification of the system layer, business layer, and metadata, achieving higher-precision time synchronization anomaly detection. During the detection process, this method uses a sliding window to statistically calculate the NTP time offset and video frame timestamp consistency of each node in real time, ensuring a significantly reduced false negative rate for anomaly detection and keeping the detection latency within 10 milliseconds, thereby significantly improving the system's timely response to abnormal states. Combined with a dynamic correction strategy, based on graded processing of time deviation severity and progressive adjustment, node self-healing recovery is achieved. For minor time deviations, rate limiting adjustments are implemented through NTP soft synchronization measures, significantly reducing service disturbances; while for severe deviations, immediate isolation and log analysis are performed. After node time recovery, rigorous challenge-response testing ensures its stability, and automatic release from isolation upon successful verification, thus achieving dynamic linkage between resource scheduling and business load, guaranteeing service continuity and reliability.

[0107] Based on actual project testing, the system can generally meet node-level time synchronization and fault isolation requirements when time synchronization anomalies occur in the cluster. In large-scale deployment scenarios of distributed systems (video storage systems), where cluster time synchronization anomalies are inevitable, the time anomaly detection and dynamic correction technologies involved in this application can accurately identify and resolve server-level faults, greatly reducing the risk of data corruption and loss, and effectively improving system availability and data consistency.

[0108] Furthermore, the fault isolation mechanism proposed in this application achieves millisecond-level blocking of abnormal nodes through collaborative operations, fully ensuring system stability and data consistency. By combining freezing the database permissions of abnormal nodes with SDN network isolation, it not only quickly prevents interference from abnormal nodes but also dynamically updates the load balancing strategy, switching traffic to backup nodes. The isolation latency is reduced to less than 5 milliseconds, significantly reducing the risk of dirty data spread.

[0109] Based on real-world project testing, the fault isolation mechanism was designed to quickly freeze the operational permissions of an abnormal node and implement network isolation upon detection. This mechanism effectively improves the stability of the entire cluster, ensuring that abnormal nodes do not affect the data consistency and business continuity of other normal nodes. In actual projects, when fault isolation operations are detected, quickly marking abnormal nodes as "untrusted" and updating traffic forwarding policies in real time effectively prevents the spread of dirty data, thereby optimizing the reliability of the entire system.

[0110] Furthermore, this application is applicable to the field of video acquisition and storage. By introducing multi-dimensional collaborative detection, fault isolation mechanisms, and dynamic correction strategies, a complete closed-loop management process from anomaly detection to self-healing recovery is formed. This solution not only effectively improves the time synchronization reliability of nodes in the cluster but also enhances the system's ability to respond quickly to abnormal situations, ultimately achieving the goal of improving the overall performance and stability of video storage services.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of circuits or units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0112] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0113] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0114] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for processing node time, characterized in that, The processing method includes: The time difference corresponding to several nodes is detected from at least two dimensions to obtain at least two time differences; wherein, the at least two dimensions include at least two of the system layer dimension, business layer dimension and metadata dimension; Node time anomaly detection is performed based on the at least two time differences; When a time anomaly is detected in a target node, time repair is performed on the target node.

2. The processing method according to claim 1, characterized in that, The at least two dimensions include system layer dimension, business layer dimension, and metadata dimension; the time difference corresponding to the plurality of nodes is detected from the at least two dimensions respectively to obtain at least two time differences, including: The first time difference is obtained by detecting the time difference corresponding to the aforementioned nodes at the system layer level. The second time difference is obtained by detecting the time difference corresponding to the aforementioned nodes from the business layer perspective; The third time difference is obtained by detecting the time difference corresponding to the aforementioned nodes from the metadata dimension; The node time anomaly detection based on the at least two time differences includes: Node time anomaly detection is performed based on the first time difference, the second time difference, and the third time difference.

3. The processing method according to claim 2, characterized in that, The step of detecting the time difference corresponding to the plurality of nodes from the business layer dimension to obtain the second time difference includes: For each node, the timestamp of each video frame is extracted from its corresponding video stream, and the time difference is calculated using the timestamps between adjacent video frames to obtain the second time difference.

4. The processing method according to claim 2, characterized in that, The step of detecting the time difference corresponding to the plurality of nodes from the metadata dimension to obtain the third time difference includes: When a write operation is initiated on each node, the current timestamp is recorded, and the global timestamp updated after the transaction is committed to ETCD is obtained. The third time difference is obtained by calculating the time difference between the current timestamp and the global timestamp.

5. The processing method according to claim 2, characterized in that, When a time anomaly is detected in a target node, time repair is performed on the target node, including: The first time difference, the second time difference, and the third time difference are weighted and summed to obtain the total time difference; In response to the total time difference being less than the fourth time threshold, the time of the target node is soft-synchronized and corrected. In response to the total time difference being greater than or equal to the fourth time threshold, the target node is isolated and time-repaired.

6. The processing method according to claim 5, characterized in that, The soft synchronization correction of the target node's time includes: Set the baseline correction rate based on the total time difference; Soft synchronization correction is performed using the total time difference and the reference correction rate. During the correction process, the system load is detected, and the reference correction rate is adjusted according to the system load. Soft synchronization correction is then performed again using the adjusted reference correction rate until the soft synchronization correction result meets the preset requirements.

7. The processing method according to claim 5, characterized in that, The isolation and time-based repair of the target node includes: Perform service isolation and network isolation on the target node; After isolation is completed, time repair is performed on the target node.

8. The processing method according to claim 7, characterized in that, The process of isolating the target node for services and networks includes: Call the ETCD lease management interface to revoke the lease permissions of the target node, terminate all incomplete transaction operations of the target node, remove the target node from the voting member list of the Raft protocol, and freeze the permissions of the target node; After permissions are frozen, all outgoing traffic from the target node is discarded.

9. A video storage system, characterized in that, The video storage system includes several nodes, and each node is processed using the processing method described in any one of claims 1-8.

10. The video storage system according to claim 9, characterized in that, The nodes include storage nodes and management nodes.

Images