3123results about "File access structures" patented technology

The invention provides a meteorological metadata storage method and system based on machine learning, and the method comprises the steps: obtaining a meteorological metadata set of a target region, inputting the meteorological metadata set into a pre-trained time sequence feature extraction model, generating the time sequence distribution features of all meteorological file files, and inputting a spatial topology mapping model, generating spatial correlation topological features of the meteorological elements; based on a cross fusion result of the time sequence distribution features and the space correlation topological features, a hierarchical index structure of meteorological metadata is constructed, a meteorological storage optimization model is called to generate a storage node distribution strategy of a meteorological file, and the meteorological file is stored according to the storage node distribution strategy. And writing each meteorological file file in the meteorological metadata set into the corresponding distributed storage node, and dynamically adjusting the mapping relationship between the physical position and the index of the stored file based on a real-time updating mechanism of the hierarchical index. The meteorological metadata storage efficiency and retrieval accuracy can be improved, and storage resource waste and operation and maintenance cost are reduced.

The invention discloses an optical fiber data storage management system and method based on big data, and relates to the technical field of storage management. According to the optical fiber data storage management system and method based on big data, the spatial-temporal characteristics of the historical access records are extracted through the LSTM network, the data heat value is calculated in combination with the time decay function, and the thermal distribution diagram of the data blocks is generated. And based on the thermal distribution map, constructing a storage resource allocation model by using a deep reinforcement learning algorithm, and dynamically adjusting the optical path priority and copy distribution. A data prefetching strategy is optimized through the spatio-temporal joint index and the vector similarity index, and a high-frequency data cache index distribution diagram is generated. And comprehensively considering the data access delay, the index hit rate and the copy migration frequency, obtaining an optical fiber storage efficiency index, and adjusting a storage strategy according to the index. According to the method, the dynamic scheduling and utilization efficiency of data storage resources is effectively improved, the data access delay is reduced, the copy management is optimized, and the overall performance of a storage system is remarkably improved.

The invention provides an unstructured file parallel synchronization method and system, and relates to the technical field of file synchronization, and the method comprises the steps: carrying out the intelligent partitioning of a target file through a pre-training file partitioning model, and generating a double-layer cascade index number and a data hash value; and then, according to the data transmission priority model, transmitting the data sub-blocks to a target end database through a self-adaptive parallel transmission channel, and performing asynchronous verification. And finally, recombining the data sub-blocks based on the double-layer cascade index number and the data block association graph by utilizing a pre-training file recombining model, generating a synchronous target file, and submitting confirmation information to the block chain network. According to the method, the unstructured file synchronization efficiency can be improved, the data integrity and security are guaranteed, and the network transmission pressure and the storage cost are reduced.

The invention provides an AI agent memory management method and system based on cold and hot stratification, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining and preprocessing the historical information of an AI agent, and carrying out the vectorization storage of the historical information; calculating a score based on the access frequency and the semantic association; constructing a dual-threshold triggering mechanism to realize data cold and hot layered migration; establishing index mapping and dynamically monitoring data popularity for returning; and a storage water level dynamic adjustment migration strategy is monitored in real time. According to the invention, the problem of low memory management efficiency of the AI agent is solved, efficient access of memory content is realized, and the storage cost is reduced.

The invention provides an integrated metadata management method and system based on data source expandability, and relates to the technical field of data sources, and the method comprises the steps: executing real-time detection on a plurality of heterogeneous data sources through a PF4J plug-in framework, collecting connection information, and establishing a dynamic connection pool; obtaining a table structure, a field attribute and an association relationship list, and assembling the table structure, the field attribute and the association relationship list into JSON format data; performing analysis and standardization processing on the data, and constructing an association degree calculation matrix; generating a standardized data structure description file; establishing a three-layer index directory, and detecting structure change and updating; and calculating a data table reference dependency relationship, and generating a DAG data flow path. According to the method, efficient integration and management of heterogeneous data sources are realized, and the metadata processing efficiency is improved.

This disclosure describes a natural language question (NLQ) query service within a service provider network that provides row level security (RLS) for autocomplete during entry of NLQs and fuzzy matching in NLQ answering. The rules take the form of per-user predicates such as Tim can only see rows with region=US. In configurations a complex extraction and preprocessing pipeline to extract distinct combinations of values against RLS predicate “rule keys” is used. Those distinct values are indexed along with grouped rule keys to enable pushing down predicates at auto-complete time. This enables pushing part of RLS rule handling to ingestion time of a dataset rather than handling all RLS rule handling at query time, enabling meeting of latency goals. In configurations, a single logical document of unique cell values is split into multiple documents with a subset of rule keys to handle scalability limits.

The invention belongs to the technical field of data storage, and discloses a multi-source marine environment data storage method and system based on HDF5. According to the method, a hierarchical storage framework is constructed, a domain-type-source three-level directory architecture is constructed, a space-time dimension dynamic extension mechanism is executed, and space-time composite coding is performed; designing an HDF5 nested packet storage architecture, wherein grid topological structure definition, dynamic resolution self-adaptive storage, multi-source data processing and metadata management are carried out; constructing distributed data association and indexes, including data standardization preprocessing and cross-source data association; the storage performance is optimized and expanded, wherein density sensitive blocking and layered compression are carried out. According to the method, redundant storage of a high-latitude region is greatly reduced, and dynamic adaptation of a local encryption grid is supported. According to the invention, open circulation and collaborative utilization of interdisciplinary data resources are facilitated.

The invention discloses an enterprise-level knowledge management system based on a large language model. The enterprise-level knowledge management system comprises a knowledge storage module, a knowledge analysis module and a knowledge application module, the knowledge storage module is responsible for managing and storing knowledge from different sources, supporting uploading, updating, deleting and sharing of electronic documents and multimedia documents, and is provided with authority control; the knowledge analysis module is used for carrying out deep understanding and structured processing on text and multimedia contents through a special analyzer by utilizing an advanced large language model technology; the knowledge application module provides intelligent retrieval and intelligent question and answer functions and supports advanced applications such as behavior portrait analysis and standard process supervision. According to the method and the system, the acquisition, understanding and application efficiency of enterprise knowledge is remarkably improved by fusing a large language model technology and a traditional knowledge management method, an intelligent, automatic and safe knowledge management solution is provided for an enterprise, and value mining and innovative application of enterprise knowledge assets are effectively promoted.

The embodiment of the invention provides a file resource reuse library management method and device based on a database, and efficient management of project resources is realized by constructing a distributed column family database cluster and dividing the distributed column family database cluster into a plurality of special column families. A multi-dimensional index mechanism is innovatively designed and comprises reverse indexing, reverse indexing and version linking, and flexible resource retrieval is supported. A distributed version lock and a transaction manager are introduced to process concurrent access, and a multi-user cooperation scene is intelligently processed through a conflict detector. And the system automatically processes resource import and tool chain configuration, so that the consistency and reliability of resource reuse are ensured. According to the method, the limitation of traditional file resource management is broken through, and an efficient resource reuse solution is provided for large-scale project development.

The invention discloses a human resource archive filing management system, and relates to the technical field of data management, the system comprises an acquisition module, a processing module and a management module, the acquisition module acquires original archive data of employees, transmits the data to the processing module, classifies the original archive data according to a preset first rule, and stores the classified data to the management module; the first archive data is obtained, controlled and managed, a calling permission and a distribution strategy are generated, the first archive data is recognized through a preset second rule, second archive data is generated, corresponding processing is conducted according to the processing priority, a processing log is reserved, third archive data are obtained and transmitted to the management module, and the management module is used for managing the first archive data. A distributed cloud storage cluster and a local storage node are constructed, an overall storage architecture and a retrieval path of third archive data are optimized, archive management is refined through three-dimensional classification, data integrity is guaranteed through field-level verification and automatic completion, scene-based dynamic authority control safety is achieved, compliance risks are intelligently cleared and avoided, and efficiency is improved through mixed storage. And active management of archive management is promoted.

The invention discloses a smart city data sharing system based on a block chain technology, and relates to the crossing field of medical data processing and information technologies. The system comprises the following modules: 1, a data storage and block chain mapping module, which generates an abstract by using a salting SHA-256 algorithm, stores the abstract in a block chain, stores original data in an IPFS distributed manner, sets redundancy and cache, and regulates abstract storage according to data popularity; 2, a data sharing module driven by the intelligent contract, which is used for subdividing authorization levels, setting pre-request verification, dynamically updating rules and automatically executing functions to guarantee compliance and controllability; the privacy protection technology fusion module expands zero-knowledge proof application and optimizes multi-party security calculation by adopting hybrid encryption; and 4, a data tracing and compliance auditing module which records logs to form a historical chain, traces and outputs an auditing report. According to the invention, the security, efficiency and compliance of medical data sharing can be comprehensively improved, and the application prospect is wide.

The invention relates to the technical field of archive management, and provides an archive resource intelligent classification and association indexing method based on multi-modal ai analysis, which comprises the following steps: preprocessing collected archive data to form a multi-modal data set comprising text data, image data and audio data; extracting feature vectors of the text data, the image data and the audio data, and fusing the feature vectors into a unified archive representation vector through a cross-modal attention fusion mechanism; constructing a hierarchical classification model, and performing intelligent classification on the archives in combination with a domain knowledge graph; the method comprises the following steps: constructing an associated index of an archive from four dimensions of entities, semantics, time and space and events, establishing a feedback mechanism, dynamically updating and optimizing the associated index, and providing multi-modal retrieval and visual display based on a classification system and the associated index. According to the intelligent classification and association indexing method for the archive resources, the problems that in an existing archive classification and indexing method, the multi-modal processing capacity is insufficient, the association mining depth is insufficient, and the efficiency is low are solved.

The present invention relates to the technical field of computers, and provides a log processing method and system, a log management platform, and an electronic device. The method comprises: acquiring real-time log data of each service system, and storing the log data in a message queue; according to a preset time window, writing the log data in the message queue into a target file of a specified storage directory; and in response to completion of data writing of a current time window, on the basis of the storage directory of the target file, calling a log analysis interface of a log analysis service to perform log analysis on the target file. The log processing method in the present invention reduces the risk incident rate of service systems, and improves the localization accuracy of an anomaly condition of a service log.

The invention discloses an archive retrieval method and system based on cloud computing, and the method comprises the steps: receiving retrieval request information inputted by a user, and analyzing a retrieval intention according to the retrieval content in the retrieval request information, the historical retrieval behavior of the user and a retrieval field database by using a context perception semantic analysis model, generating a user retrieval semantic vector in combination with the retrieval intention; the method comprises the following steps: constructing an archive data index map for archive data stored in a distributed manner in a cloud storage environment; according to the user retrieval semantic vector, a search domain related to retrieval semantics is dynamically positioned in the archive data index map in a multi-layer projection calculation mode; and in the search domain, screening a candidate archive set according with the query intention by using a matching algorithm of multi-modal semantic fusion, and sorting and displaying the candidate archive set. By utilizing the embodiment of the invention, the accuracy and efficiency of file retrieval and the user experience can be improved.

The invention belongs to the technical field of data synchronization, and particularly relates to a real-time data synchronization method for cloud computing heterogeneous resources. In order to solve the problem of data synchronization among heterogeneous resources, the method comprises the following steps: aggregating a plurality of files in batches to generate a packaged file; performing metadata compression processing on the generated packaged file, and eliminating repeated data through a deduplication algorithm; dynamically selecting a lightweight data transmission protocol; establishing a multi-dimensional index structure; only retaining the file of the latest version through a version control mechanism; dynamically adjusting the allocation of synchronization tasks based on the load condition of available resources in the cloud computing environment; and rescheduling the uncompleted synchronization task. Through a multi-dimensional index structure, version control, self-adaptive transmission scheduling and a dynamic resource allocation mechanism, efficient and stable file synchronization among heterogeneous resources is realized, repeated transmission and storage waste are reduced, and real-time performance, reliability and resource utilization efficiency of synchronization are improved.

The invention relates to the technical field of information security, in particular to a computing method and system for information security. The method comprises the following steps of: partitioning and fragmenting input data, and distributing the data to a plurality of storage nodes through randomization, distributed storage and Hash mapping to ensure the safety of data storage and the distribution flexibility; establishing an index table for all the fragments; regularly triggering a data confusion mechanism, and adjusting the storage position of data fragments through a pseudo-random function; the integrity of the data is detected by generating a check code and an erasure code, it is ensured that the data is not tampered or damaged in the transmission or storage process, and the data is recovered through a redundant copy when the data is lost; the identity verification and the data integrity when the user accesses the data are ensured through authority authentication; and monitoring storage nodes and data access behaviors in real time, detecting potential threats and taking dynamic adjustment measures. According to the invention, through a multi-level security policy, the security and fault-tolerant capability of data processing are effectively improved.

The invention relates to the field of Web application, in particular to a tree structure data dynamic management method based on node hierarchical attributes and recursive updating. Comprising the following steps: S1, constructing a tree structure data system, and endowing each node with a hierarchical attribute which is used for identifying position information of the node in a tree structure; s2, receiving an operation instruction of a user for the tree structure data, wherein the operation instruction comprises node adding, node batch deleting, node moving, data exporting and importing, node querying and node quantity counting; and S3, according to different operation instructions, executing corresponding processing logic: based on node hierarchy attributes, applying a recursive update mechanism to realize dynamic management of tree structure data, and ensuring data consistency and accuracy.

The invention discloses a channel model dynamic association method based on a multi-source data index, and relates to channel engineering digital management. The channel model dynamic association method comprises the steps that component codes of channel engineering are constructed and used for identifying the same physical entity; establishing a three-layer index table structure according to the component codes; constructing a dynamic partition management mechanism of a three-level cache architecture, wherein the three-level cache architecture comprises a first-level cache stored in a memory, a second-level cache stored in a local disk and a third-level cache stored in a distributed manner; according to data access information in the three-layer index table structure, data migration is carried out among all levels of caches; according to the dynamic partition management mechanism, a data version management mechanism based on a timestamp concurrency control protocol is established, the data version management mechanism generates incremental data by comparing original data with updated data, and incremental updating or full updating is selected according to a threshold value. According to the method, the problem that data evolution tracking is difficult due to fragmentation of multi-source heterogeneous data and unbalanced access efficiency in navigation channel engineering is solved.

The invention relates to the technical field of data processing, and discloses a real-time data stream processing method and device, electronic equipment and a storage medium. The method comprises the following steps: distributing a received to-be-processed data stream to a first distributed node through a preset load balancer; fragmenting the received distribution data stream at the first distributed node according to a preset fragmentation rule, then writing the fragmented data stream into a message queue, and performing parallel processing on fragmented data in the message queue through multiple threads; storing the intermediate state data in the fragmented data processing process, generating snapshot data according to the fragmented data, and synchronizing the snapshot data to the distributed storage system; acquiring load data of the first distributed node, and dynamically updating a data flow distribution strategy of the load balancer according to the load data through the resource use prediction model; and writing a processing result of the to-be-processed data stream into persistent storage through an asynchronous batch mechanism, and returning the processing result through a preset real-time communication protocol. The method improves the data stream processing efficiency and stability.

The invention discloses an intelligent archive content evaluation system based on deep learning, which relates to the technical field of deep learning, and comprises the steps of inputting a standardized feature vector into a deep learning model, calculating a time sequence waveform factor and a semantic kurtosis factor in combination with a rule matching reference, and generating an evaluation confidence coefficient matrix; the cosine similarity and the Spearman correlation coefficient are included, and the comparison index threshold triggers abnormal marking and grading early warning. According to the method, a multi-model fusion deep learning evaluation model is constructed and comprises a semantic understanding model, a risk evaluation model and a value classification model, split attributes are dynamically optimized in combination with an evaluation rule decision tree, file contents can be comprehensively evaluated from multiple dimensions of semantic understanding, risk evaluation, value classification and the like, and the evaluation efficiency is improved. And an accurate comprehensive evaluation result and a rule matching reference are generated, so that the comprehensiveness and scientificity of file content evaluation are improved.

The invention provides a surveying and mapping result management method and system based on big data, and the method comprises the steps: obtaining geographic space surveying and mapping data through employing an unmanned plane, a vehicle-mounted laser radar and a sensor, and recording the time, position and equipment information of the surveying and mapping data in real time; a regularizer based on online linear optimization classifies the surveying and mapping data according to geographic positions, collection time and data types, and uploads the classified surveying and mapping data to a distributed storage system; performing data cleaning, format unification and quality verification on the stored surveying and mapping data to generate standardized surveying and mapping data; constructing a knowledge base based on a large language model (LLM), performing semantic analysis and feature labeling on the standardized surveying and mapping data by using geographic space operators in the knowledge base, and establishing a spatial index of the standardized surveying and mapping data; and receiving a user retrieval request, and performing privacy protection retrieval on the standardized surveying and mapping data based on the spatial index.

The invention discloses an intelligent document duplicate checking system and method based on a vector database and a large language model, and the method comprises the following steps: S1, collecting document data in various formats, and carrying out the preprocessing of the document data; s2, semantic coding is carried out through a large language model, and a document semantic vector is generated; s3, storing the document semantic vector into a vector database, constructing a vector index and recording historical query data; s4, carrying out preliminary candidate document retrieval, and carrying out approximate nearest neighbor retrieval based on outlier identification; s5, calculating the similarity between the candidate document and the document to be subjected to duplicate checking, and screening a final high-similarity document; s6, generating a duplicate checking report, and recording user operation behaviors; and S7, receiving user feedback, and dynamically updating the document semantic vector and the vector index. According to the method, efficient and accurate intelligent document duplicate checking is realized by utilizing the large language model and the vector database, the semantic matching capability is improved, the duplicate checking efficiency is optimized, and the intelligence and adaptability of a duplicate checking system are improved.

The invention discloses a document analysis and index construction method. The method comprises the following steps of layout detection, table image analysis and extraction, page recovery, content splitting and index construction and storage. According to layout detection, a document layout analysis model is used for carrying out area detection and classification on images, tables, titles and text elements of pdf document pages; the table image analysis extraction is to extract and process formats and contents of tables and images existing in each page of the pdf obtained based on layout detection; according to the method for analyzing the document and constructing the index, intelligent analysis is provided, texts, tables and images in the pdf document can be accurately extracted, and perfect index elements can be constructed.

The invention provides a data classification and grading method and device based on an artificial intelligence large model technology, and relates to the field of data security. The method comprises the following steps: constructing a data classification and grading catalog according to a preset classification rule, and storing each row of the classification and grading catalog as a document record in a target vector database; respectively carrying out first retrieval operation, second retrieval operation and third retrieval operation on the data classification and grading catalog by taking the business system name, the data table name and the field name as first query conditions so as to obtain a first matching document, a second matching document and a third matching document in the target vector database; sorting the first matching document, the second matching document and the third matching document according to a preset sorting mode, and obtaining a sorting result; and inputting a sorting result into the large language model, and performing data security early warning through the large language model. According to the method and the device, the security problems that corresponding security measures cannot be taken, sensitive data is leaked and the like when a data governance engineer identifies the metadata according to the data classification and classification catalog and judges the classification and security level of the metadata are solved.

The invention discloses an unstructured storage hierarchical strategy optimization method based on machine learning, and relates to the technical field of data storage management, and the method comprises the steps: monitoring a file access event in real time, generating an access log, and extracting a multi-dimensional feature data set; utilizing the trained multi-dimensional decision tree model to distribute a corresponding storage hierarchy for the storage object to obtain a storage hierarchy decision; according to a storage level decision, distributing the storage objects to different storage layers, and carrying out resource configuration and storage operation; monitoring the access condition of the storage object in the new storage hierarchy in real time, and collecting file access performance, storage cost and response time to obtain feedback data; by monitoring the file access event in real time and extracting the multi-dimensional feature data set containing the basic attribute, the access behavior and the context information, efficient response of hotspot data and reasonable utilization of resources are ensured, and the overall performance utilization rate and the cost effectiveness of storage are remarkably improved.

The invention provides a layout file geometric data processing method, a computer device and a storage medium. The method comprises the steps that a layout file of a three-dimensional chip is obtained, polygonal entities and layer information are extracted, the polygonal entities are distributed to corresponding layer containers, and an independent R tree index is created for each layer; performing connectivity analysis on the polygonal entity, and returning a connected component set and performance analysis statistical information; when an independent R-tree index is created for each layer, data of all polygonal entities of a single layer is obtained, a minimum boundary rectangle is generated for each polygonal entity, R-tree indexes are constructed for different layers in parallel by adopting a plurality of threads, whether the number of node entries of the R-tree indexes exceeds a splitting threshold value or not is judged, and if yes, splitting processing is carried out on nodes; and verifying the consistency of the created R tree index and the original data. The computer device and the storage medium can realize the method. The geometric data processing efficiency of the layout file can be improved.

Computer-implemented methods and systems are disclosed for receiving and indexing a plurality of files for later querying, for dynamically generating scripts to be executed during a query of a data store, and for horizontally distributing a query and aggregating results of the distributed query.

The invention provides a data weaving platform-oriented self-service data preparation method, which comprises the following steps of: accessing original data and associated structured metadata of a target data source into a data weaving platform to generate standardized accessed original data and associated structured metadata; based on a semantic model of the data weaving platform, performing classified semantic annotation on the associated structured metadata, and positioning a transfer node of the associated structured metadata to generate a data blood relationship record; performing bidirectional quality verification on the data consanguinity record according to semantics and path continuity, and matching with a federated governance rule of a data weaving platform to generate a metadata governance list; and generating a unified metadata directory based on the metadata governance list, so that a user of the data knitting platform can obtain key segments containing complete data consanguinity records. According to the method, the problem of insufficient accuracy and efficiency in self-service data preparation can be effectively relieved, and the efficient preparation requirement of a data weaving platform for multi-source data is met.

The present application relates to the field of cloud technology, and discloses a metadata access method and apparatus, a device, and a storage medium. According to the method, a global metadata cache scheme is used, and relates to a three-layer architecture, i.e., a global cache file, a cache description array, and a memory block; storage areas can be divided according to the data length of metadata in a shared memory; and when target metadata needs to be acquired for a target process, metadata can be queried layer by layer according to the three-layer architecture until target address information of target metadata that has been cached is obtained, and the target address information is returned to the target process for realizing the access of the target metadata. In this way, the metadata required by all the working processes can be accessed by means of the method, so that metadata caches of the working processes do not need to be respectively maintained by the working processes, thereby reducing the repeated metadata cache between the working processes, reducing the occupied memory resources, reducing the transmission resource occupation of the repeated metadata, and improving the resource utilization rate.

Methods and systems for decentralized storage are provided. To store a file in a decentralized network, a source node in the decentralized network first divides the file into a plurality of file portions. For each given file portion, a corresponding peer storage node is located in the decentralized network, where a distance between the corresponding peer storage node and the given file portion is below a given threshold, where the distance is computed based on an identifier of the corresponding peer storage node, and an identifier of the given file portion, and where the given threshold is associated with a storage capacity of the corresponding peer storage node. The source node then transmits each given file portion to the corresponding peer storage node, and generates a file identifier for the file, based on the identifiers of the plurality of file portions.