Intelligent data recycling management method and system for storage system

Abstract

The application relates to the technical field of security monitoring storage systems, in particular to an intelligent data recovery management method for a storage system, which is applied to a security monitoring storage system and comprises the following steps: acquiring historical monitoring data in a security monitoring area, and generating a security area baseline model according to the historical monitoring data; acquiring deleted monitoring data, comparing the deleted monitoring data with the security area baseline model, and screening effective monitoring data segments from the deleted monitoring data; establishing an associated mapping relationship between the effective monitoring data segments and the security area baseline model, and recovering and storing the effective monitoring data segments in the security monitoring storage system according to the associated mapping relationship. The application can periodically clean up invalid monitoring data, release storage resources, retain important information, and maximize the storage safety of monitoring data.

Description

Technical Field

[0001] This application relates to the field of security monitoring storage system technology, and in particular to an intelligent data recycling management method and system for storage systems. Background Technology

[0002] In existing technologies, the data monitored in the security surveillance industry is large in volume due to the need for real-time monitoring, putting significant pressure on storage systems. Long-term storage of large amounts of data not only consumes storage resources but also increases the wear and tear and maintenance costs of storage devices. Furthermore, since most regions have legal or industry regulations regarding the retention period of security surveillance data, such as retaining it for 7 or 30 days before deletion, data needs to be retained for a certain period before deletion. However, deleting data when the retention period reaches the limit may result in the deletion of important information. Therefore, it is necessary to reclaim the monitoring data when it is deleted. Summary of the Invention

[0003] Therefore, it is necessary to provide an intelligent data recycling management method and system for storage systems that can periodically clean up invalid monitoring data, release storage resources, retain important information, and ensure the storage security of monitoring data to the greatest extent possible, in order to address the above-mentioned technical problems.

[0004] The technical solution of this invention is as follows: An intelligent data recycling management method for storage systems, applied to a security monitoring storage system, the method comprising: Acquire historical monitoring data within the security monitoring area, and generate a baseline model of the security area based on the historical monitoring data; Obtain deleted monitoring data, compare the deleted monitoring data with the security area baseline model, and filter out valid monitoring data segments from the deleted monitoring data; Establish an association mapping relationship between the effective monitoring data segment and the security area baseline model, and reclaim and store the effective monitoring data segment in the security monitoring storage system according to the association mapping relationship.

[0005] Optionally, deleted monitoring data can be acquired, compared with the security area baseline model, and valid monitoring data segments can be selected from the deleted monitoring data, including: Obtain deleted monitoring data and generate a fused feature vector based on the deleted monitoring data; Randomly sample each of the fused feature vectors to obtain the first point set; The baseline model of the security area is sampled to obtain a second set of points; Calculate the cost matrix based on the first point set and the second point set; Based on the cost matrix, the first point set, and the second point set, perform entropy regularization optimal transmission calculation and generate entropy regularization 2-Wasserstein distance; Valid monitoring data segments are selected based on the entropy regular 2-Wasserstein distance.

[0006] Optionally, valid monitoring data segments are filtered based on the entropy regular 2-Wasserstein distance, including: The monitoring data difference degree is generated based on the entropy regular 2-Wasserstein distance; Valid monitoring data segments are selected based on the differences in the monitoring data.

[0007] Optionally, valid monitoring data segments can be filtered based on the differences in the monitoring data, including: Calculate the logarithmic mean and standard deviation based on the degree of difference in the monitoring data; A first threshold and a second threshold are generated based on the mean and standard deviation of the numbers; Valid monitoring data segments are selected based on the first threshold and the second threshold.

[0008] Optionally, an association mapping relationship is established between the effective monitoring data segment and the security area baseline model, and the effective monitoring data segment is reclaimed and stored in the security monitoring storage system according to the association mapping relationship, including: Obtain the corresponding baseline sub-cluster identifier based on the monitoring data difference degree corresponding to the effective monitoring data segment; A residual vector is generated based on the fusion feature vector corresponding to the effective monitoring data segment, and a binary mask is generated based on the residual vector. The binary mask is aggregated into a sparse tensor according to the time dimension; Generate an association mapping relationship based on the baseline sub-cluster identifier and the sparse tensor; The valid monitoring data segments are retrieved and stored in the security monitoring storage system according to the aforementioned association mapping relationship.

[0009] Optionally, historical monitoring data within the security monitoring area is acquired, and a baseline model of the security area is generated based on the historical monitoring data, including: Acquire historical monitoring data for a preset time from a preset camera angle within a security monitoring area; Initial keyframes are extracted from the historical monitoring data according to a preset time interval, and after denoising and slight anti-shake processing of the initial keyframes, preprocessed keyframes are generated. The preprocessed keyframes are input into the first preset learning model to generate the first feature vector; Optical flow difference is calculated for the preprocessed keyframes of three adjacent frames, and a second feature vector is generated; The first feature vector and the second feature vector are concatenated to form a joint feature vector; the joint feature vector is written into a preset vector database, and the vector set is density-clustered based on the incremental HDBSCAN clustering algorithm to generate a security area baseline model.

[0010] Optionally, the data in the deleted monitoring data other than the valid monitoring data segment is the monitoring data segment to be evaluated, and the method further includes: Generate a fragment summary for the monitoring data segment to be evaluated; The monitoring data segment to be evaluated is compressed and stored in a preset mode.

[0011] Optionally, an intelligent data recycling management system for storage systems is also provided, the system comprising: The baseline model generation module is used to acquire historical monitoring data within the security monitoring area and generate a baseline model of the security area based on the historical monitoring data. The effective data filtering module is used to obtain deleted monitoring data, compare the deleted monitoring data with the security area baseline model, and filter out effective monitoring data segments from the deleted monitoring data. The data recycling and storage module is used to establish the association mapping relationship between the effective monitoring data segment and the security area baseline model, and to recycle and store the effective monitoring data segment in the security monitoring storage system according to the association mapping relationship.

[0012] Optionally, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps described in the intelligent data recycling management method for storage systems.

[0013] Optionally, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps described in the intelligent data recycling management method for storage systems.

[0014] The technical effects achieved by this invention are as follows: The aforementioned intelligent data recycling management method and system for storage systems acquires historical monitoring data within a security monitoring area and generates a security area baseline model based on the historical monitoring data; acquires deleted monitoring data, compares the deleted monitoring data with the security area baseline model, and filters out valid monitoring data segments from the deleted monitoring data; establishes an association mapping relationship between the valid monitoring data segments and the security area baseline model, and recycles and stores the valid monitoring data segments in the security monitoring storage system according to the association mapping relationship. The security monitoring area is a pre-defined area that needs to be monitored in real time, and the historical monitoring data is data acquired through advance monitoring of the security monitoring area. Each security monitoring area has clearly defined target objects that need to be monitored. For example, if the security monitoring area is used to monitor a production line in a factory, then the production line and the personnel using the production line are the target objects that need to be monitored. For the production line, most of the monitoring data contains a large number of similar images in different monitoring cycles, such as the same conveyor belt, basically the same worker actions, and the same lighting arrangement. To facilitate subsequent data storage and recycling management, a security area baseline model is first generated based on the historical monitoring data to represent the basic operational behavior and status of target objects within the security monitoring area. Next, to meet industry standards and data security management regulations, monitoring data needs to be stored for a period of time. Afterward, to prevent the security monitoring storage system from running out of storage resources due to a large amount of identical data stored for the same monitoring period, the monitoring data needs to be deleted. However, direct deletion could easily lead to the deletion of currently undiscovered important information, causing data storage security issues. Therefore, when deleting monitoring data, it is necessary to reclaim the monitoring data. Specifically, this involves acquiring the deleted monitoring data, comparing it with the security area baseline model, filtering out valid monitoring data segments from the deleted data, establishing a mapping relationship between the valid monitoring data segments and the security area baseline model, and reclaiming and storing the valid monitoring data segments in the security monitoring storage system based on this mapping relationship. This setup allows for the retrieval of previously deleted valid monitoring data segments when data needs to be accessed later, and the mapping relationship can be used to view the data in conjunction with the security area baseline model to obtain monitoring data that more comprehensively reflects the monitored events. Therefore, this application first generates a security area baseline model, then filters out valid monitoring data segments from the deleted monitoring data, and finally performs data recycling and storage management based on the correlation mapping relationship between the two. This achieves the goal of regularly cleaning up invalid monitoring data and releasing storage resources, while also retaining important information and ensuring the storage security of monitoring data to the greatest extent possible. Attached Figure Description

[0015] Figure 1This is a flowchart illustrating an intelligent data recycling management method for a storage system in one embodiment. Figure 2 This is a block diagram of an intelligent data recycling management system for a storage system, as shown in one embodiment. Detailed Implementation

[0016] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0017] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0018] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0019] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0020] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0021] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0022] In one embodiment, a terminal is provided, the terminal being configured to: acquire historical monitoring data within a security monitoring area, and generate a security area baseline model based on the historical monitoring data; acquire deleted monitoring data, compare the deleted monitoring data with the security area baseline model, and filter out valid monitoring data segments from the deleted monitoring data; establish an association mapping relationship between the valid monitoring data segments and the security area baseline model, and reclaim and store the valid monitoring data segments in a security monitoring storage system based on the association mapping relationship.

[0023] The terminal may be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices.

[0024] In one embodiment, such as Figure 1 As shown, an intelligent data recycling management method for storage systems is provided, applied to a security monitoring storage system. The method includes: Step S100: Obtain historical monitoring data within the security monitoring area, and generate a baseline model of the security area based on the historical monitoring data; Step S200: Obtain deleted monitoring data, compare the deleted monitoring data with the security area baseline model, and filter out valid monitoring data segments from the deleted monitoring data; Step S300: Establish the association mapping relationship between the effective monitoring data segment and the security area baseline model, and reclaim and store the effective monitoring data segment in the security monitoring storage system according to the association mapping relationship.

[0025] In this embodiment, the security monitoring area is a pre-defined area that needs to be monitored in real time, and the historical monitoring data is data obtained by monitoring the security monitoring area in advance. Each security monitoring area clearly defines the target objects that need to be monitored. For example, if the security monitoring area is used to monitor the production line in a factory, then the production line and the personnel using the production line are the target objects that need to be monitored. For the production line, most of the monitoring data contains a large amount of similarity in different monitoring periods, such as the same conveyor belt, basically the same worker actions, and the same lighting arrangement. To facilitate subsequent data storage and retrieval management, a security area baseline model is first generated based on the historical monitoring data to represent the basic operating behavior and status of the target objects within the security monitoring area. Next, to meet industry standards and data security management regulations, monitoring data needs to be stored for a period of time. Afterward, to prevent the security monitoring storage system from running out of storage resources due to a large amount of identical data stored for the same monitoring period, the monitoring data needs to be deleted. However, direct deletion could easily lead to the deletion of currently undiscovered important information, causing data storage security issues. Therefore, when deleting monitoring data, it is necessary to reclaim the monitoring data. Specifically, this involves acquiring the deleted monitoring data, comparing it with the security area baseline model, filtering out valid monitoring data segments from the deleted data, establishing a mapping relationship between the valid monitoring data segments and the security area baseline model, and reclaiming and storing the valid monitoring data segments in the security monitoring storage system based on this mapping relationship. This setup allows for the retrieval of previously deleted valid monitoring data segments when data needs to be accessed later, and the mapping relationship can be used to view the data in conjunction with the security area baseline model to obtain monitoring data that more comprehensively reflects the monitored events. Therefore, this application first generates a security area baseline model, then filters out valid monitoring data segments from the deleted monitoring data, and finally performs data recycling and storage management based on the correlation mapping relationship between the two. This achieves the goal of regularly cleaning up invalid monitoring data and releasing storage resources, while also retaining important information and ensuring the storage security of monitoring data to the greatest extent possible.

[0026] In one embodiment, step S100: acquiring historical monitoring data within the security monitoring area and generating a security area baseline model based on the historical monitoring data, including: Step S110: Obtain historical monitoring data for a preset time from a preset camera angle within the security monitoring area; In this step, the preset monitoring camera viewing angle is pre-set, with a fixed position and data acquisition angle, thus ensuring the reliability of the obtained historical monitoring data. The preset time is set to at least one week and up to two months to ensure that the historical monitoring data has a certain time span and is therefore representative.

[0027] Step S120: Extract initial keyframes from the historical monitoring data according to a preset time interval, and generate preprocessed keyframes after denoising and slight anti-shake processing of the initial keyframes. In this step, initial keyframes are extracted at pre-set time intervals to achieve fixed-interval frame extraction, avoiding redundant computation problems caused by full-frame parsing. Furthermore, the time interval is not always constant; it can be adaptively fine-tuned based on inter-frame similarity, ensuring information integrity while reducing computational load. Denoising employs a combined temporal and spatial domain approach to reduce false detections caused by high-ISO nighttime noise. Slight image stabilization is achieved through sub-pixel-level video stabilization technology, using high-precision motion compensation to offset minor camera vibrations, thereby reducing the noise tolerance requirements in subsequent processing. Finally, the pre-processed keyframes are generated, reducing invalid data input for subsequent feature extraction and clustering, lowering GPU usage, and ensuring consistent lighting and structure in the extracted frames. This provides a more stable input distribution for the subsequent Transformer network, reducing false alarms.

[0028] Step S130: Input the preprocessed keyframe into the first preset learning model and generate the first feature vector; In this step, the first preset learning model is a self-supervised visual Transformer network, which is a model combining self-supervised learning and the visual Transformer architecture. By setting up the self-supervised visual Transformer network, robust global features to structure and texture can be obtained without manual annotation, making it suitable for monitoring security surveillance areas where annotation is difficult in this application, and solving the problem of high feature drift caused by CNNs or pixel histograms in the prior art.

[0029] Step S140: Calculate the optical flow difference for the three adjacent preprocessed keyframes and generate a second feature vector; In this step, micro-movements in the monitoring are captured by differential capture of three adjacent frames, so that the micro-movements in the monitoring data have significant feature increments, better preserve the dynamic and effective data in the monitoring, and at the same time solve the problem of missed detection in the pure static background model in the existing technology.

[0030] Step S150: Concatenate the first feature vector and the second feature vector to form a joint feature vector; In this step, the first feature vector is a spatial feature, and the second feature vector is a temporal feature. The joint feature vector is generated by scaling and normalizing the first and second feature vectors before fusing and concatenating them. This allows for the utilization of Transformer global features in conjunction with optical flow information to improve adaptability to changes in external conditions.

[0031] Step S160: Write the joint feature vector into a preset vector database, perform density clustering on the vector set based on the incremental HDBSCAN clustering algorithm, and generate a security area baseline model.

[0032] In this step, the joint feature vector is written into a preset vector database, and the vector set is density-clustered based on the incremental HDBSCAN clustering algorithm to extract the k centroids of the highest density clusters. and its covariance ; as described The dataset serves as a baseline model for the security area, and at least one representative image is synchronously saved for each cluster, forming a "graph-vector" dual index. Furthermore, based on the incremental HDBSCAN clustering algorithm, it can automatically identify clusters of different densities, suitable for multimodal data in security monitoring scenarios, specifically including modeling day shift, night shift modes, and equipment operating status. This is achieved by extracting the centroids of the k highest-density clusters. This is to achieve the selection of the highest density clusters, thereby focusing on high-frequency modes, reducing anomalous interference, and improving the stability of the baseline model. Regarding its covariance... It is used to describe the distribution characteristics of data within a cluster, providing a statistical basis for subsequent data processing and enhancing the sensitivity to deviations in multidimensional features.

[0033] In one embodiment, step S200: obtaining deleted monitoring data, comparing the deleted monitoring data with the security area baseline model, and filtering out valid monitoring data segments from the deleted monitoring data, including: Step S210: Obtain deleted monitoring data and generate a fusion feature vector based on the deleted monitoring data; In this step, in this embodiment, the deleted monitoring data refers to data that has been stored for a period of time and needs to be deleted. To avoid deleting important data, the data that needs to be deleted is recycled. After obtaining the deleted monitoring data, a fusion feature vector is generated based on the deleted monitoring data. The step of obtaining the fusion feature vector is the same as the data processing steps in steps S110-S150. Those skilled in the art should know and master how to generate the fusion feature vector, so this application will not elaborate further.

[0034] Step S220: Randomly sample each of the fused feature vectors to obtain the first point set; Randomly and uniformly downsample each of the fused feature vectors to obtain a point set Pi = {p1, ..., pn} of size n (≤512). Specifically, randomly sample n fused feature vectors to represent the discrete representation of the probability distribution Pi on d-dimensional real feature vectors.

[0035] Step S230: Sample the baseline model of the security area and obtain the second point set; Each Gaussian subcluster in the security area baseline model Through covariance Sample m pseudo-points to form the second point set Q. j ={q1, ..., qm}, j=1…k, m takes values ​​from 128 to 256.

[0036] Step S240: Calculate the cost matrix based on the first point set and the second point set; In this step, the formula for calculating the cost matrix is ​​as follows: , in, The cost matrix value, For and Let be the two feature vectors to be matched. In this embodiment, represents the feature vectors in the first point set and the second point set, where 'a' takes the value 1-m and 'b' takes the value 1-k. For feature dimension, This is for calculating the Euclidean squared distance. During calculation, it is divided by the feature dimension. This scales the distance to an average of approximately 1, making it easier to unify the regularization and thresholding in the future.

[0037] Step S250: Perform entropy-regularized optimal transport calculation based on the cost matrix, the first point set, and the second point set, and generate the entropy-regularized 2-Wasserstein distance; In this step, the cost matrix is ​​calculated to ensure the normal iteration of the Sinkhorn-Knopp matching algorithm. During the entropy-regularized optimal transport calculation, the regularization strength is... Below, the Sinkhorn-Knopp iteration is used to solve for Pi and Q. j The entropy regular 2-Wasserstein distance between them is calculated, and the number of such entropy regular 2-Wasserstein distances generated is multiple.

[0038] Step S260: Generate the monitoring data difference degree based on the entropy regular 2-Wasserstein distance; In this step, the minimum value among the generated entropy regularized 2-Wasserstein distances is taken as the difference degree between the deleted monitoring data and the security area baseline model, i.e., the monitoring data difference degree D. i .

[0039] Step S270: Filter out the valid monitoring data segments based on the difference in the monitoring data.

[0040] In this step, instead of performing frame-by-frame point-to-point matching, the feature distribution of the entire monitoring data segment is compared using the optimal transmission-SinKhorn distance. By treating each segment of the deleted monitoring data as a probability distribution, the security area baseline model is simplified to k Gaussian sub-distributions. The Sinkhorn-Knopp matching algorithm is then used to measure Pi and Q using entropy-regularized 2-Wasserstein distance. j The data transportation costs between the two are used to determine the differences and finally filter out the effective monitoring data segments. The effective monitoring data segments can be understood as data segments that are significantly different from the security area baseline model and exceed the threshold. This means that not all segments with differences are retained, but rather conditional screening and data retention are carried out to further avoid storing some redundant monitoring data, thereby achieving the goal of saving storage while retaining key information.

[0041] In one embodiment, step S270: filtering out valid monitoring data segments based on the monitoring data difference includes: Step S271: Calculate the logarithmic mean and standard deviation based on the difference in the monitoring data; Step S272: Generate a first threshold and a second threshold based on the mean and standard deviation of the numbers; Step S273: Select valid monitoring data segments based on the first threshold and the second threshold.

[0042] In this embodiment, the difference D of the monitoring data within the preset time window is first determined. i Find the logarithmic mean with standard deviation And generate a dual threshold, which includes a first threshold and a second threshold. and .

[0043] First threshold and second threshold and The formula is as follows: , .

[0044] D i Less than At that time, the corresponding monitoring data segment is a dissimilar data segment, D i In and During this period, management personnel will verify the data. After verification, a portion of the data will be marked as dissimilar data segments or as valid monitoring data. When D... i Greater than When the specified threshold is reached, the corresponding monitoring data segment is set as valid monitoring data. The setting of the first and second thresholds is not limited to the methods described above; those skilled in the art can also set values ​​different from the examples above based on actual needs and testing. This application will not elaborate on this further.

[0045] Preferably, the regularization intensity is 1-10% of the mean of the elements of the cost matrix.

[0046] In one embodiment, step S300: establishing an association mapping relationship between the effective monitoring data segment and the security area baseline model, and reclaiming and storing the effective monitoring data segment in the security monitoring storage system according to the association mapping relationship, including: Step S310: Obtain the corresponding baseline sub-cluster identifier based on the monitoring data difference degree corresponding to the effective monitoring data segment; Step S320: Generate a residual vector based on the fusion feature vector corresponding to the effective monitoring data segment, and generate a binary mask based on the residual vector; Step S330: Aggregate the binary mask into a sparse tensor according to the time dimension; Step S340: Generate an association mapping relationship based on the baseline sub-cluster identifier and the sparse tensor; Step S350: Reclaim the valid monitoring data segment and store it in the security monitoring storage system according to the association mapping relationship.

[0047] In this embodiment, the monitoring data difference W corresponding to the effective monitoring data segment ij As can be obtained from step S260, the monitoring data difference degree acquisition corresponds to the baseline sub-cluster identifier (cidi). Next, the fusion feature vector corresponding to the valid monitoring data segment is... Calculate the residual vector The formula for calculating the residual vector is: .in, The centroid representing the nearest valid monitoring data segment can be understood as the mean vector of the cluster CID, derived from a pre-trained security area baseline model. Obtain from the middle. Next, obtain the residual vector. Threshold clipping is performed to obtain the binary mask m. t The binary mask is aggregated into a sparse tensor Di={(p,t)|m according to the time dimension. t,p=1}, where p represents the feature dimension and t is the frame number. Then, based on the baseline subcluster identifier and the sparse tensor, an association mapping relationship is set, specifically by setting the association triple ARi=<seg_id_i,cidi,Di> Then, the associated triplet is written to a key-value store or graph database, where seg_id_i is the time primary key time_id of the valid monitoring data segment. This configuration, based on the association mapping relationship, allows for quick identification of the cluster cidi and its difference descriptor D between the valid monitoring data segment and the security area baseline model when searching for the valid monitoring data segment by reading ARi based on seg_id_i. i Finally, based on the aforementioned association mapping relationship, the valid monitoring data segments are retrieved and stored in the security monitoring storage system. Therefore, by setting up a complete link including reference cluster determination, residual masking, spatiotemporal descriptors, and triplet storage, the valid monitoring data segments can be quickly located, and users can be informed of where the valid monitoring data segments differ from the monitoring data in the security area baseline model.

[0048] In one embodiment, the data in the deleted monitoring data other than the valid monitoring data segment is the monitoring data segment to be evaluated, and the method further includes: Step S021: Generate a fragment summary for the monitoring data segment to be evaluated; Step S022: Compress and store the monitoring data segment to be evaluated in a preset mode.

[0049] In this embodiment, a fragment summary is generated for the monitoring data segment to be evaluated. The fragment summary includes at least a 1fps JPEG thumbnail sequence, a motion vector histogram, and OCR / object detection text. The preset mode is to encode the monitoring data segment to be evaluated with AV1 encoding (CRF≥48) and downsample to a resolution less than or equal to QVGA, with a compression bitrate less than or equal to 0.05Mbps, and then write it into a low-cost cold archive bucket (ColdTier) after Reed-Solomon (14,10) erasure coding. When a user requests access to the monitoring data segment to be evaluated, a fragment summary is returned immediately for quick preview. If the user confirms that the original fragment is needed, a retrieval command is issued to object storage based on the shard_id. After retrieval, the fragment is temporarily warmed up to the hot layer cache. After 30 days, it is automatically re-archived and the quota_offset is restored.

[0050] Of course, if it is detected that no one queries the monitoring data segment to be evaluated within the preset storage period, the monitoring data to be evaluated and related data will be deleted directly.

[0051] In one embodiment, the method further includes: First, the monitored data segment V_raw to be evaluated is written to the cold archive bucket to obtain the archive object V_arc. Then, the byte size S_raw is recorded, and a placeholder file Stub with a size not exceeding 1 KB is generated in the hot layer storage bucket. Next, after capturing the PutObject(Stub) event, the quota control module returns the same number of bytes of storage quota to the available quota of the hot layer storage bucket according to the negative value of the quota-offset field. Finally, the actual used capacity of the hot layer is periodically calculated and summed with the quota-offset field of all Stub files. If the algebraic sum is inconsistent with the billed capacity, the difference is recorded and a correction is triggered. Therefore, by combining space-saving data migration with real-time quota replenishment, disk and storage costs are saved, continuous business writes are not affected by capacity limits, and billing consistency is automatically maintained.

[0052] In one embodiment, the intelligent data recycling management method for storage systems further includes: Key monitoring targets are pre-defined. Historical characteristics of these targets are obtained from historical data. Based on these historical characteristics, target monitoring data matching these characteristics is filtered from deleted monitoring data, where each target monitoring data corresponds to one deleted monitoring data. The target monitoring data are then concatenated to form the target object's activity trajectory, which is then collected and stored. This setup enables data retrieval for specific monitoring targets.

[0053] It should be noted that the hardware equipment required for the intelligent data recycling management method for storage systems described in this application shall be set by those skilled in the art when implementing it, and this application does not impose any specific restrictions.

[0054] In one embodiment, such as Figure 2 As shown, an intelligent data recycling management system for storage systems is also provided, the system comprising: The baseline model generation module is used to acquire historical monitoring data within the security monitoring area and generate a baseline model of the security area based on the historical monitoring data. The effective data filtering module is used to obtain deleted monitoring data, compare the deleted monitoring data with the security area baseline model, and filter out effective monitoring data segments from the deleted monitoring data. The data recycling and storage module is used to establish the association mapping relationship between the effective monitoring data segment and the security area baseline model, and to recycle and store the effective monitoring data segment in the security monitoring storage system according to the association mapping relationship.

[0055] In another embodiment, the effective data filtering module is further configured to: acquire deleted monitoring data and generate a fusion feature vector based on the deleted monitoring data; randomly sample each of the fusion feature vectors to obtain a first point set; sample the security area baseline model to obtain a second point set; calculate a cost matrix based on the first point set and the second point set; perform entropy regularization optimal transmission calculation based on the cost matrix, the first point set, and the second point set to generate an entropy regularization 2-Wasserstein distance; generate a monitoring data difference degree based on the entropy regularization 2-Wasserstein distance; and filter out effective monitoring data segments based on the monitoring data difference degree.

[0056] In another embodiment, the effective data filtering module is further configured to: calculate the logarithmic mean and standard deviation based on the difference in the monitoring data; generate a first threshold and a second threshold based on the logarithmic mean and standard deviation; and filter out effective monitoring data segments based on the first threshold and the second threshold.

[0057] In another embodiment, the data recycling and storage module is further configured to: obtain the corresponding baseline sub-cluster identifier based on the monitoring data difference degree corresponding to the effective monitoring data segment; generate a residual vector based on the fusion feature vector corresponding to the effective monitoring data segment, and generate a binary mask based on the residual vector; aggregate the binary mask into a sparse tensor according to the time dimension; generate an association mapping relationship based on the baseline sub-cluster identifier and the sparse tensor; and recycle and store the effective monitoring data segment in the security monitoring storage system based on the association mapping relationship.

[0058] In another embodiment, the baseline model generation module is further configured to: acquire historical monitoring data for a preset time from a preset camera angle within the security monitoring area; extract initial keyframes from the historical monitoring data according to a preset time interval, and generate preprocessed keyframes after denoising and slight image stabilization processing of the initial keyframes; input the preprocessed keyframes into a first preset learning model and generate a first feature vector; calculate the optical flow difference for the preprocessed keyframes of three adjacent frames and generate a second feature vector; concatenate the first feature vector and the second feature vector into a joint feature vector; write the joint feature vector into a preset vector database, perform density clustering on the vector set based on the incremental HDBSCAN clustering algorithm, and generate a security area baseline model.

[0059] In another embodiment, the baseline model generation module is further configured to: generate a fragment summary of the monitoring data segment to be evaluated; and compress and store the monitoring data segment to be evaluated in a preset mode.

[0060] In another embodiment, the baseline model generation module is further configured to: write the monitoring data segment V_raw to be evaluated into a cold archive bucket to obtain an archive object V_arc. Then, record the byte size S_raw, and generate a placeholder file Stub with a size not exceeding 1 KB in the hot layer storage bucket. Next, after capturing the PutObject(Stub) event, the quota control module returns the same number of bytes of storage quota to the available quota of the hot layer storage bucket according to the negative value of the quota-offset field. Finally, it periodically calculates the actual used capacity of the hot layer and sums it with the quota-offset field of all Stub files. If the algebraic sum is inconsistent with the billed capacity, the difference is recorded and a correction is triggered.

[0061] In another embodiment, the baseline model generation module is further configured to: pre-identify key monitoring targets; obtain historical characteristics of the key monitoring targets based on historical data; filter target monitoring data matching the historical characteristics from the deleted monitoring data based on the historical characteristics, wherein one target monitoring data corresponds to one deleted monitoring data; concatenate the target monitoring data to form the target object's activity trajectory; and recycle and store the target object's activity trajectory. This configuration enables data recycling for special monitoring targets.

[0062] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0064] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0065] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0066] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0067] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0068] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the various method embodiments above.

[0069] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0070] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0075] One embodiment of this application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above-described methods.

[0076] The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the above description is an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than described above, or a combination of certain components, or different components, such as input / output devices, network access devices, etc.

[0077] The processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0078] In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory may include both internal and external storage units of the computer device. The memory is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.

[0079] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An intelligent data recycling management method for storage systems, applied to security monitoring storage systems, characterized in that, The method includes: Acquire historical monitoring data within the security monitoring area, and generate a security area baseline model based on the historical monitoring data to represent the basic operational behavior and status of target objects within the security monitoring area, including: Acquire historical monitoring data for a preset time from a preset camera angle within a security monitoring area; Initial keyframes are extracted from the historical monitoring data according to a preset time interval, and after denoising and slight anti-shake processing of the initial keyframes, preprocessed keyframes are generated. The preprocessed keyframes are input into the first preset learning model to generate the first feature vector; Optical flow difference is calculated for the preprocessed keyframes of three adjacent frames, and a second feature vector is generated; The first feature vector and the second feature vector are concatenated to form a joint feature vector; the joint feature vector is written into a preset vector database, and the vector set is density-clustered based on the incremental HDBSCAN clustering algorithm to generate a security area baseline model. Obtain deleted monitoring data, compare the deleted monitoring data with the security area baseline model, and filter out valid monitoring data segments from the deleted monitoring data, including: Obtain deleted monitoring data and generate a fused feature vector based on the deleted monitoring data; Randomly sample each of the fused feature vectors to obtain the first point set; The baseline model of the security area is sampled to obtain a second set of points; Calculate the cost matrix based on the first point set and the second point set; Based on the cost matrix, the first point set, and the second point set, perform entropy regularization optimal transmission calculation and generate entropy regularization 2-Wasserstein distance; Valid monitoring data segments are selected based on the entropy regularization 2-Wasserstein distance. Establish an association mapping relationship between the effective monitoring data segment and the security area baseline model, and reclaim and store the effective monitoring data segment in the security monitoring storage system according to the association mapping relationship.

2. The intelligent data recycling management method for storage systems according to claim 1, characterized in that, Valid monitoring data segments are selected based on the entropy regularization 2-Wasserstein distance, including: The monitoring data difference degree is generated based on the entropy regular 2-Wasserstein distance; Valid monitoring data segments are selected based on the differences in the monitoring data.

3. The intelligent data recycling management method for storage systems according to claim 2, characterized in that, Based on the differences in the monitoring data, valid monitoring data segments are selected, including: Calculate the logarithmic mean and standard deviation based on the degree of difference in the monitoring data; A first threshold and a second threshold are generated based on the logarithmic mean and standard deviation; Valid monitoring data segments are selected based on the first threshold and the second threshold.

4. The intelligent data recycling management method for storage systems according to claim 1, characterized in that, Establishing a mapping relationship between the effective monitoring data segment and the security area baseline model, and reclaiming and storing the effective monitoring data segment in the security monitoring storage system according to the mapping relationship, including: Obtain the corresponding baseline sub-cluster identifier based on the monitoring data difference degree corresponding to the effective monitoring data segment; A residual vector is generated based on the fusion feature vector corresponding to the effective monitoring data segment, and a binary mask is generated based on the residual vector. The binary mask is aggregated into a sparse tensor according to the time dimension; Generate an association mapping relationship based on the baseline sub-cluster identifier and the sparse tensor; The valid monitoring data segments are retrieved and stored in the security monitoring storage system according to the aforementioned association mapping relationship.

5. The intelligent data recycling management method for storage systems according to claim 1, characterized in that, The data in the deleted monitoring data other than the valid monitoring data segment is the monitoring data segment to be evaluated. The method further includes: Generate a fragment summary for the monitoring data segment to be evaluated; The monitoring data segment to be evaluated is compressed and stored in a preset mode.

6. An intelligent data recycling management system for storage systems, employing the intelligent data recycling management method for storage systems as described in any one of claims 1-5, characterized in that, The system includes: The baseline model generation module is used to acquire historical monitoring data within the security monitoring area and generate a security area baseline model based on the historical monitoring data to represent the basic operating behavior and status of target objects within the security monitoring area. The effective data filtering module is used to obtain deleted monitoring data, compare the deleted monitoring data with the security area baseline model, and filter out effective monitoring data segments from the deleted monitoring data. The data recycling and storage module is used to establish the association mapping relationship between the effective monitoring data segment and the security area baseline model, and to recycle and store the effective monitoring data segment in the security monitoring storage system according to the association mapping relationship.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images