1749results about "File system functions" patented technology

The invention relates to the technical field of archive management, in particular to an archive full life cycle management system and method based on cloud computing. The method specifically comprises an intelligent classification module based on a format adaptation degree function and semantic density analysis, a five-stage life cycle management module, and an authority scheduling module fusing RBAC / ABAC authority control and block chain auditing, and based on a matrix decomposition archiving static coding mechanism, document security storage and analysis are realized through an orthogonal matrix and pseudo-inverse operation, and the file security is realized. The data security in a long-term archiving scene is improved; cloud storage resources are dynamically optimized according to the document life cycle state and the access frequency, and the storage cost is reduced in combination with a cold and hot layering strategy; and an AI algorithm is introduced, the archive retention value is automatically evaluated according to laws and regulations and semantic analysis, and a compliant destruction decision is realized. According to the method, through multi-model fusion metadata extraction, dynamic strategy binding and automatic state transition, the whole-process management efficiency and safety are remarkably improved.

The invention relates to the technical field of information management, and comprises a distribution automation operation and maintenance work log intelligent recording platform based on a CS architecture, and the platform comprises a log field analysis module, a task chain construction module, a task logic verification module, and a log storage tracing module. According to the method, missing and abnormal fields are effectively marked and corrected through meticulous analysis and comparison of uploaded log records, the accuracy and integrity of data are improved, task scheduling is optimized and the operation and maintenance management efficiency is improved through association of device numbers and task types and adjustment of execution time in sorting and grouping of task sequences, and the task scheduling efficiency is improved. Task chains are recombined to solve conflicts and adjust priorities, smooth execution of operation and maintenance tasks is guaranteed, delay is reduced, logic verification enhances task reliability through rule verification and error marking, unassociated tasks are deduced, analyzed and corrected, the exception processing capacity is enhanced, the data tracing and storage efficiency is improved, and the tracing capacity of data management is enhanced.

The invention relates to the technical field of intelligent manufacturing, discloses a papermaking equipment fault tracing method and system based on a process knowledge graph, and discloses the papermaking equipment fault tracing method and system based on the process knowledge graph. The method and the system comprise data acquisition and preprocessing, papermaking process knowledge graph construction, fault event detection and matching, fault tracing and propagation path reasoning, and maintenance scheme recommendation and optimization. The method overcomes the limitation that the traditional method is difficult to capture the cross-equipment, cross-process and cross-time sequence deep causal association and fault propagation path of the papermaking equipment. By constructing a comprehensive papermaking process knowledge graph, equipment operation state data, process parameter data, production quality data, equipment structure principle, process flow knowledge, fault mode knowledge, maintenance experience and other heterogeneous knowledge are subjected to deep fusion and semantic association, so that the system can exceed the correlation of the data surface; and an internal mechanism and a propagation chain of the fault are deeply excavated.

The invention provides an integrated metadata management method and system based on data source expandability, and relates to the technical field of data sources, and the method comprises the steps: executing real-time detection on a plurality of heterogeneous data sources through a PF4J plug-in framework, collecting connection information, and establishing a dynamic connection pool; obtaining a table structure, a field attribute and an association relationship list, and assembling the table structure, the field attribute and the association relationship list into JSON format data; performing analysis and standardization processing on the data, and constructing an association degree calculation matrix; generating a standardized data structure description file; establishing a three-layer index directory, and detecting structure change and updating; and calculating a data table reference dependency relationship, and generating a DAG data flow path. According to the method, efficient integration and management of heterogeneous data sources are realized, and the metadata processing efficiency is improved.

The invention discloses an intelligent resource scheduling and dynamic data priority management method and system, and the method comprises the steps: setting priorities for data files or data blocks based on the business importance of the data files or the data blocks, setting independent queues for tasks with different priorities, and dynamically adjusting the resource distribution according to the priorities; the task type, the access frequency and the flow condition of the current task are monitored and analyzed, the high-load task is automatically identified, and bandwidth allocation is dynamically adjusted; labels are set for different data attributes in the HDFS, the use conditions of data are monitored in real time, the use conditions comprise the access frequency, the modification frequency and the data size, and the labels and the priorities of the data are automatically adjusted. According to the method and the system, through fine-grained priority management, self-adaptive bandwidth allocation and a data intelligent label mechanism, flexible resource scheduling is realized, and the access efficiency of key data is improved, so that the response speed of the system is optimized, and the problems of resource allocation and access challenges under high-concurrency and multi-priority services are solved.

The invention relates to the technical field of data processing, in particular to a cross-unit data management method, which comprises the following steps of: performing format standardization processing on data in a data pool of each organ and unit, converting the data into a uniform standard format, extracting semantic features to construct a semantic association network, and recording a semantic relationship of data items and a dynamic authorization and authority mapping mechanism. Then, according to the inquirer identity information, the unit attributes, the business requirements and the data sensitivity level, dynamically generating inquiry authorization and establishing an authority mapping table; and when a query condition is received, analyzing by utilizing a natural language processing technology, and generating a query route and optimizing a statement in combination with the semantic association network and the permission mapping table. And finally, calling data from each data pool according to the route, performing fusion processing, displaying the data to a querier in real time, and recording a query log. The invention aims to solve the problems of data semantic isolation, authority management staticization, low query efficiency and insufficient cross-unit data conflict resolution capability in the existing cross-unit data management.

The invention discloses a data synchronization method and system, a storage medium and electronic equipment, and relates to the field of data processing. The method comprises the steps that incremental data generated when full data of an old database is written into a new database is obtained, the new database is in a running state, and the old database is in a to-be-run state; writing the incremental data into the new data and the old database, and judging the consistency of the data in the new database and the old database; and when the consistency of the data reaches a preset standard, gradually replacing the use states of the old database and the new database. By implementing the technical scheme provided by the invention, the data consistency can be ensured under the condition of not interrupting business operation in the database migration process.

The embodiment of the invention provides a memory recovery method and device, electronic equipment and a storage medium, and relates to the technical field of data storage. Under the condition that both the file system and the storage device support the perceptual garbage collection function, the device state of the storage device is obtained in response to the starting of the perceptual garbage collection function; according to the equipment state, determining whether an execution condition for sensing a garbage collection function is met or not; and if the execution condition of the garbage collection sensing function is met, the file system is linked to perform memory collection on the storage device, so that the file system can know the data blocks moved by the storage device, the file system is prevented from moving the data blocks moved by the storage device again, and the efficiency of the file system is improved. In addition, invalid blocks in the file system and the storage device can be synchronized, the situation that invalid data blocks in the file system are moved when the storage device executes memory recovery is avoided, and therefore the service life expenditure of the storage device and the write-in amplification value of the storage device are reduced.

The invention discloses a transaction processing mechanism optimization method and system for time series data, and belongs to the technical field of databases, and the method realizes a differentiated ACID support mechanism, strengthens atomicity and durability guarantee, and weakens consistency and isolation requirements; comprising the following steps: designing a collaborative architecture of a transaction system and a pre-write log WAL, and establishing a binding relationship between a transaction state and WAL writing; generating a WAL record only for the successfully submitted transaction operation, and performing persistent storage by adopting a batch group submission strategy; transaction scheduling is realized through a lightweight transaction manager, and the life cycle of a transaction request is managed; the functions of efficient writing of transaction granularity, log storage and rapid recovery are realized through a pre-writing log manager; through intelligent Checkpoint background service, based on a periodic incremental check point mechanism, compression merging and space recovery of the WAL file are realized. According to the invention, the performance breakthrough of the time sequence database in a high-concurrency write-in scene can be realized.

The invention provides a data conflict processing method. The data conflict processing method comprises the following steps: monitoring files under a file synchronization path; when the change of the file is monitored, executing file conflict detection to determine whether conflicts which cannot be merged exist or not; performing priority evaluation on the conflicts which cannot be merged to obtain priority scores; performing priority ranking on the conflicts which cannot be merged based on the priority scores, and generating a priority ranking result; according to the priority ranking result, selecting a proper version to carry out conflict resolution on the conflict which cannot be merged; generating version numbers for the unselected conflicts which cannot be merged, and storing the unselected conflicts which cannot be merged and related detailed information into a database or recording the unselected conflicts which cannot be merged and the related detailed information in a persistent file mode. The disclosure also provides an electronic device, a computer readable medium, and a computer program product. According to the method, manual intervention is reduced in the file synchronization process, and important data is ensured to be processed preferentially.

Systems and processes disclosed herein perform an automatic remediation process. The automatic remediation process may be a log-based remediation process. Systems disclosed herein may obtain log files from an information management system and determine the occurrence of errors at the information management system based on error codes included in the logs. Further, the systems may access a knowledgebase to determine whether solutions for the errors have been previously generated. The solutions may include patches or hotfixes that can be applied to the information management system without removing user-access or stopping execution of the information management system. The systems may automatically update the information management system to address the errors. Alternatively, or in addition, the systems may alert a user, such as an administrator, of the existence of a solution to the error, and whether the solution may be applied without interrupting service or access to the information management system.

The invention discloses a log template post-processing method and system based on semantic screening and similarity clustering. The method comprises the steps that log information is input into LLM to generate a log template; inputting the new log template and the cache log template into a semantic model, and respectively generating semantic embedding vectors; calculating a similarity distance between the new log template and the cache log template, and screening to obtain a candidate template; according to the generation confidence score, calculating the weighted structure similarity between the new log template and each candidate template, and clustering and combining the candidate templates exceeding a threshold value to generate a final template; and updating and replacing the final template into the cache structure. According to the technical scheme, the semantic similar templates can be recognized and combined more reasonably, efficiently and accurately, a semantic enhanced, controllable and stable template post-processing mechanism can be constructed on the premise that LLM calling burden is not increased, and automatic cache cleaning, template number control and structural consistency maintenance are achieved.

The invention relates to the field of computers, and provides a method and a device for pre-aggregating time series data. The method comprises the following steps: acquiring at least one time sequence data from a first device; the first device is a generation device of at least one time sequence data, or the first device is a node device of a time sequence database, and any time sequence data in the at least one time sequence data comprises an identifier; determining a pre-polymerization method according to the identifier of the at least one time sequence data, wherein the pre-polymerization method and the identifier of the at least one time sequence data have a preset binding relationship; processing the at least one time sequence data by using a pre-aggregation method to generate pre-aggregated data; and when the occupancy rate of the memory meets a trigger condition, writing the pre-aggregated data and the at least one time sequence data into a disk. According to the method, the time sequence data and the pre-aggregated data can be uniformly distributed, and the utilization rate of the storage space is improved.

The invention discloses an unstructured storage hierarchical strategy optimization method based on machine learning, and relates to the technical field of data storage management, and the method comprises the steps: monitoring a file access event in real time, generating an access log, and extracting a multi-dimensional feature data set; utilizing the trained multi-dimensional decision tree model to distribute a corresponding storage hierarchy for the storage object to obtain a storage hierarchy decision; according to a storage level decision, distributing the storage objects to different storage layers, and carrying out resource configuration and storage operation; monitoring the access condition of the storage object in the new storage hierarchy in real time, and collecting file access performance, storage cost and response time to obtain feedback data; by monitoring the file access event in real time and extracting the multi-dimensional feature data set containing the basic attribute, the access behavior and the context information, efficient response of hotspot data and reasonable utilization of resources are ensured, and the overall performance utilization rate and the cost effectiveness of storage are remarkably improved.

In one embodiment, a system for managing a virtualization environment includes a set of host machines, each of which includes a hypervisor, virtual machines, and a virtual machine controller, and a first virtualized file server configured to receive a request to access a storage item located at a second virtualized file server, determine that the storage item is designated as being accessible by other virtualized file servers, identify an FSVM of the second virtualized file server at which the storage item is located, and forward the request to the FSVM of the second virtualized file server. The storage item may be designated as being accessible by other virtualized file servers when the storage item is associated with a predetermined tag value indicating that the storage item is shared among virtualized file servers. The predetermined tag value may be stored in a sharding map in association with the storage item.

Systems and methods are described for scheduling a query for execution. The system receives and parses a query to identify one or more portions of the query. The system determines a resource allocation for each portion of the query, and determines an availability of compute resources for the different portions of the query. Based on the resource allocation and the availability of compute resources, the system schedules the query.

Methods, systems, and devices for data management are described. A backup cluster may store incremental or base snapshots of computing objects. When a snapshot expires, a data management system (DMS) that manages the backup cluster may in some cases merge or consolidate the expired snapshot with a non-expired snapshot to create a new merged snapshot. In some cases, however, consolidation may be deferred until a chain of multiple expired snapshots satisfies one or more heuristic thresholds, to conserve resources. Example heuristic thresholds may be a length of the chain, an amount of space reclaimable by consolidating the snapshots, an age of the expired snapshots in the chain of incremental snapshots, or an amount of free space on the backup cluster.

A defragmentation method may include: acquiring at least one chunk information of data in response to a writing request for the data, wherein the at least one chunk information may include an identification of at least one chunk and a stream identification assigned to data of the at least one chunk; updating a fragmentation degree of each of the at least one chunk and a stream to which the each of the at least one chunk belongs in a chunk index structure to obtain an updated chunk index structure, based on the at least one chunk information, wherein the chunk index structure may include identifications of a plurality of chunks, stream identifications corresponding to the plurality of chunks, and fragmentation degrees of the plurality of chunks; and determining a first chunk to be defragmented, based on the fragmentation degrees of the plurality of chunks in the updated chunk index structure.

Systems and methods are provided for generating metadata for an improved data structure that enables improvements to generative AI and LLM processing and use of the data. A system may receive a log file generated by an entity associated with a device and generate metadata of the log file that comprises origination data associated with the entity and location of the device. The system may store the metadata and content of the log file as a document object. A data structure may be generated using various data objects. For example, the system can generate a vector embedding object enriched with a temporal embedding, a positional embedding, or a hierarchy of the metadata. The system may initiate an action based on the vector embedding object.

The invention discloses a configuration management method of a network file system, electronic equipment, a medium and a product, and relates to the technical field of network file systems.The method comprises the steps that a configuration file is analyzed, configuration information obtained through analysis is stored according to a preset format, and a first file is obtained; the preset format is used for indicating memory layouts of different configuration information, and the memory layouts of the different configuration information are continuous, so that the configuration file is converted into a pre-compiled first file, and the configuration information in the first file has the continuous memory layouts, and thus when the content of the first file is loaded, the content of the first file is loaded. In this way, the memory for storing the loaded content is continuous, the memory fragmentation rate is reduced, moreover, due to the fact that the first file is obtained by analyzing the configuration file in advance, the configuration file does not need to be analyzed when the configuration loading instruction is received, time consumed for analyzing the configuration file is avoided, the loading efficiency is improved, and therefore the time consumption of analyzing the configuration file is reduced. The technical problems of low configuration file loading efficiency and high memory fragmentation rate can be solved.

The invention discloses a one-time plot rule mining method and device for a process event log, and belongs to the field of plot mining. The distance between two events is limited by adopting a time interval constraint, and repeated use of the events is avoided by adopting a one-time condition so as to avoid mining excessive meaningless plots. Moreover, when a strong one-time plot rule (i.e., a one-time plot rule with confidence greater than or equal to a minimum confidence threshold) is mined, a triple mechanism of candidate plot generation, support degree calculation and rule generation is provided. In a candidate plot generation stage, eliminating redundant candidate plot extension by adopting a plot connection strategy; in the support degree calculation stage, the support degree is calculated based on the position index; in a rule generation stage, firstly, all frequent one-time plots are mined, and then strong one-time plot rules are screened according to a minimum support degree threshold value and a minimum confidence coefficient threshold value. The technical effect of efficiently and accurately mining the strong one-time plot rule is achieved.

The invention discloses a text log management method and device based on an embedded system, equipment and a medium. According to the text log management method based on the embedded system, a storage medium is provided, the storage medium is divided into storage areas, and the storage areas are divided according to partitions; the management method comprises a writing method, a quick retrieval method and a data reading method. The method can be conveniently applied to an embedded system with relatively few hardware resources, the contradiction between embedded system development and large-capacity log data storage requirements is solved, meanwhile, the requirement that field maintenance is simple and easy to use can be met, and the method can be suitable for various storage media, occupies few system hardware resources, can be rapidly deployed, and has high practicability. And the method also has the advantages of quick retrieval, sequential writing, data reading, automatic coverage, few occupied resources and the like.

The invention discloses a file integrity monitoring method, electronic equipment and a storage medium. The method comprises the steps of sending a verification request to a scheduling node through a user node when a file integrity verification condition is triggered; determining an edge node group through the scheduling node according to the edge node load condition, and distributing an audit task corresponding to the verification request to each edge node in the edge node group; searching corresponding integrity evidence according to the auditing task through the edge node, and sending the integrity evidence to the scheduling node; consistency verification is carried out through the scheduling node according to the integrity evidence fed back by each edge node, and if the consistency verification is passed, the integrity evidence is sent to the user node; and performing local verification through the user node according to the integrity evidence, and determining a corresponding file verification result. By means of the method, the safety, traceability and credibility of file integrity monitoring in the domestic terminal environment are effectively improved, resource occupation is low, and compatibility is high.

A method for file system destinations includes obtaining events for storage on one or more of the storage systems. For each event, the method includes extracting at least one field value from the event, comparing the at least one field value to configurations of the storage systems to identify at least one storage system of the plurality of storage systems having a matching configuration, transmitting the event to an ingest module queue for the at least one storage system, selecting a partition for the event based on the at least one field value to obtain a selected partition, mapping the selected partition to a file using a partition mapping, and appending the event to the file on the at least one storage system.

The invention discloses a database traffic auditing method, device and equipment and a storage medium, and relates to the field of databases, and the method comprises the following steps: deploying a target extension program in a kernel layer of an operating system, and capturing a target data packet by using a target probe in the target extension program; performing protocol analysis on the target data packet to extract target data, determining execution time of a target SQL statement based on the target data, and performing semantic analysis on the target SQL statement by using a target analyzer and a way of performing analysis based on syntax imported based on a bypass to obtain a corresponding semantic analysis result; the target data comprises a target SQL statement and target metadata; and storing the target data, the execution time and the semantic analysis result to a target log database, so as to audit the flow of the target database based on the data in the target log database by using a pre-configured target audit strategy. According to the method and the device, efficient, safe and real-time auditing of database traffic is realized.

The invention relates to the field of log data processing and analysis, and discloses a log analysis method, system and device and a storage medium, and the method comprises the steps: collecting log data from a multi-source device, and carrying out the cleaning and structural processing of the log data; generating a dynamic topology network in which nodes represent the equipment and edges represent the equipment interaction based on the interaction relationship between the equipment of the log data; extracting topological characteristics based on the dynamic topological network, wherein the topological characteristics comprise connectivity of a topological structure, a ring structure and complexity characteristics of the network; analyzing network topology characteristic changes in different time scales, and identifying an abnormal mode in the log data; and constructing a dynamic causal network between the devices, extracting an abnormal propagation chain, and positioning root cause devices or events which cause anomalies. The method can adapt to various log formats and multi-source data, supports real-time analysis and anomaly detection, improves the intelligence and efficiency of system log analysis, and provides powerful technical support for system troubleshooting and operation and maintenance management.

The method comprises the steps of obtaining a to-be-uploaded file and a corresponding file type, preprocessing the to-be-uploaded file, generating a to-be-audited file, calling a corresponding auditing model in an AI knowledge base for auditing according to the type of the to-be-audited file, and generating a corresponding auditing result; the multi-hop query request of the user is obtained according to the auditing result, a modification suggestion is generated, and finally the modification suggestion is returned to the user and is encrypted and stored, and more intelligent and accurate auditing is achieved by means of the powerful knowledge representation and reasoning ability of the knowledge base. The modification suggestions provided when the auditing is not passed can help the user to quickly understand the problem and perform targeted modification, so that the efficiency of the content creator is improved, the times of repeated submission and auditing are reduced, the auditing cost is reduced, and the quality and efficiency of the whole auditing process are improved.

The invention discloses an intelligent medical big data information management system, and the system comprises a data collection layer which serves as a sensing neural network, employs an Internet of Things gateway to be compatible with a medical equipment protocol, and deploys an edge calculation node for physiological signal preprocessing; the intelligent preprocessing layer is used for cleaning and standardizing the original data transmitted by the data acquisition layer; the informatization service layer is used for realizing fine-grained authority control based on an RBAC (Role Based Access Control) model, realizing inspection process automation by integrating an LIS (Line Information System) interface through a Hyperdger Fabric block chain evidence storage key operation hash, and constructing a zero-footprint DICOM (Digital Imaging and Communications in Medicine) viewer; the intelligent analysis layer is used for carrying out data life cycle management by adopting a mixed storage architecture, carrying out feature engineering on data, carrying out label matching and classified storage by using efficient neighbor search and similarity / distance measurement, quantifying contribution of each feature or feature combination to a single prediction result through an SHAP value, generating an interpretation result, and outputting the interpretation result; and presenting an interpretation result to a clinician or related personnel through a query interface.

Systems and methods are described for scheduling a query for execution. The system receives and parses a query to identify one or more portions of the query. The system determines a resource allocation for each portion of the query, and determines an availability of compute resources for the different portions of the query. Based on the resource allocation and the availability of compute resources, the system schedules the query.

The invention provides a snapshot generation method and device, electronic equipment and a storage medium, and the snapshot generation method comprises the steps: responding to any data partition update version, sending a version identification obtaining request to a version module, and taking the version identification returned by the version module as the version identification of the updated version of the data partition, the version module is used for generating a version identifier in a unique and incremental manner when receiving a version identifier acquisition request every time; and receiving a snapshot generation instruction, obtaining a version identifier newly generated by the version module, and generating a snapshot based on the snapshot version of each data partition, the snapshot version being a version to which the version identifier having the smallest difference with the newly generated version identifier belongs.