16986results about "Input/output to record carriers" patented technology

The invention discloses a geographic information big data-oriented multi-source data processing system, and relates to the technical field of data acquisition and sensors, and the system comprises a data acquisition module which accesses a remote sensing satellite, an unmanned aerial vehicle, an Internet of Things sensor and social media in real time through a multi-source heterogeneous interface, and carries out the adaptive analysis of a data format and metadata marking; the distributed storage module is used for performing partition storage on the geographic information data based on a space-time database and an object storage architecture, and establishing dynamic space-time index and version control; the data fusion module is used for realizing coordinate system conversion, time sequence calibration and semantic knowledge graph matching based on a multi-source data alignment method of dynamic weight distribution; and an intelligent analysis module and a security control module. According to the method, core pain points such as data splitting, low storage efficiency, extensive analysis and compliance risks in the geographic information field are systematically solved, and a full-stack type technical base is provided for scenes such as smart cities, emergency disaster relief and environment monitoring.

The invention relates to the technical field of data storage and processing, in particular to an RAID card static cache management method and device based on data popularity, and the method comprises the steps that the physical position of a data block needing to be accessed in an SSD array is acquired according to a read-write request; historical data access information of the RAID card is collected, a historical data set is generated, and a neural network model is trained by using the historical data set to obtain a cache management model; predicting and outputting data blocks which are possibly accessed in the future and the access probability of the data blocks through the cache management model; the RAID controller dynamically adjusts a cache strategy in combination with an LRU strategy according to a prediction result output by the cache management model and a read-write request of a file system, and optimizes a storage position and an updating mechanism of a data block in a cache space; and when the hit rate or the data access efficiency does not reach the preset value, adjusting the parameters of the cache strategy and updating the cache strategy. According to the method, cache resources can be more effectively distributed, the cache hit rate is improved, the cache replacement overhead is reduced, and therefore the performance of a storage system is optimized.

The invention discloses a multi-level storage control method based on access popularity, and relates to the technical field of information, the method comprises the following steps: building a storage resource monitoring module, and collecting data volume change, read-write frequency and capacity occupation proportion information of each node from a storage system in real time; the method comprises the following steps: collecting original data, carrying out preliminary cleaning and formatting processing on the collected original data to obtain a standardized resource state data set, comparing predicted data with operation parameters of a current storage system, and if a predicted demand exceeds a current capacity limit, automatically generating a capacity expansion scheduling task to obtain a final resource optimization configuration scheme; according to the multi-level storage control method and device based on the access popularity, intelligent monitoring, load balancing and capacity planning of storage resources are achieved, the resource utilization efficiency and expandability of a storage system are improved, and an effective solution is provided for stable operation and performance optimization of a large-scale storage system.

The invention relates to the technical field of edge calculation application and partial discharge detection, in particular to a partial discharge multichannel signal real-time synchronous acquisition method based on edge calculation. According to the method, the functions of multi-channel signal acquisition, time synchronization, feature analysis, hierarchical storage and the like are integrated at edge nodes, asynchronous acquisition, unified timestamp marking and data synchronous fusion of multiple types of partial discharge signals are realized, and extraction of multi-dimensional feature parameters such as time domain, frequency domain and energy and intelligent event discrimination are locally completed. For abnormal signals, the system realizes classified storage and remote uploading after encryption and compression processing; and for normal signals, dynamic management is carried out through circular caching. According to the method, the accuracy and efficiency of multichannel signal synchronous acquisition are remarkably improved, the data safety and the system adaptive capacity are enhanced, and the method is suitable for real-time online monitoring and intelligent diagnosis of partial discharge of power equipment.

The invention discloses a data read-write request processing method, electronic equipment, a storage medium and a program product, and relates to the technical field of redundant arrays of independent disks, and the data read-write request processing method comprises the following steps: if the category of a data read-write request is a random read request, if target data corresponding to the random read request does not exist in a target cache, executing the random read request; the random read request is directly issued to a corresponding redundant array of independent disks for processing, and pre-read processing is not carried out; if the category of the data read-write request is the random write request, the random write request is stored by using the random write cache and the random write backup cache, and when the proportion of the random read-write request is less than the preset proportion threshold, the random write request is issued to the corresponding redundant array of independent disks for processing, and the random write request does not need to be merged; and the number of the random write requests issued in a centralized manner is reduced, the technical problem that the performance of the storage system is reduced by a mode of processing the random read-write requests in related technologies is solved, and the technical effect of improving the performance of the storage system is achieved.

The invention relates to the technical field of data storage, can be applied to business scenes such as medical health and financial science and technology, and discloses a multi-Pool storage resource adaptive writing method which comprises the following steps: monitoring available storage space, read-write input / output rate, average response time and data access frequency of each Pool; dynamically adjusting the current write-in weight of the Pool according to the monitoring result; converting the current write-in weights of all Pool into probability values, and constructing a discrete probability distribution table; selecting a target Pool through a pseudo-random number generator according to the probability distribution table; and writing the to-be-stored data into the target Pool. According to the method, the running state of each Pool is continuously monitored, and the writing weight is adjusted in real time, so that the problem of resource allocation rigidity under a fixed weight strategy is avoided; in combination with a discrete probability distribution table constructed based on normalized weight and a pseudo-random selection mechanism, selection of the target Pool has certain load balancing elasticity, and data can be more effectively guided to be written into the Pool with light load and excellent performance.

The invention discloses a digital production plan scheduling method and system, and belongs to the technical field of optimal scheduling, and the method comprises the steps: constructing a distributed storage architecture based on edge computing nodes; a central coordinator is adopted to realize cross-node data synchronization through an improved Raft consensus algorithm, multi-version concurrency control is realized based on a vector clock, and a global consistent data view is established; a visual scheduling platform is built based on a Vue3 framework, and man-machine interaction is realized by adopting a Canvas and WebGL collaborative rendering framework; establishing a dynamic coordinate conversion model based on bilinear interpolation, designing a space mapping function containing distortion compensation, establishing a multi-thread coordinate service based on WebWorker, and realizing submillimeter-level bidirectional mapping of pixel coordinates and physical coordinates; constructing a three-dimensional space-time analysis model fused with the multi-dimensional features; and all the units are subjected to feature fusion through residual connection, and finally a scheduling scheme with a confidence coefficient weight is output. The method and the device have the effect of meeting various scheduling requirements.

The invention relates to the technical field of data processing, and discloses a cold and hot data exchange method and system based on optical storage and a storage medium. The method comprises the following steps: performing access feature acquisition and analysis on data in the optomagnetic hybrid storage system, calculating an access life cycle factor, generating a logic interval heat evaluation result, performing temperature grading, formulating a cold and hot data placement strategy, generating a data placement mapping table, executing batch migration according to priority, and updating a data position record. According to the method, the problem of how to realize logic interval hot degree accurate identification, temperature grade dynamic grading, storage mapping optimal matching and batch migration task efficient organization under the optomagnetic hybrid architecture is solved.

The invention discloses a data management method based on a distributed storage system, relates to the technical field of data management, and is used for solving the problem of storage management behavior strategy mismatch. On the basis of dynamic perception of data access behaviors in a distributed storage system, a behavior feature vector fusing an access mutation rate, periodicity and an access span is constructed, behavior pattern recognition and strategy structure generation are completed, optimal deployment and hierarchical storage of copies are achieved through node resource state perception and construction of a cost function, and the strategy structure generation efficiency is improved. And after compression and consistency configuration are executed and the strategy falls to the ground, the compressibility and the offset trend of the behavior path are analyzed, scheduling management information is extracted, and a stable execution or strategy adjustment signal is generated, so that the problem of strategy execution mismatching caused by behavior perception deficiency in the distributed storage system is reduced, and the strategy execution efficiency is improved. The strategy closed-loop control and the resource scheduling optimization under behavior driving are realized, so that the data management efficiency and the strategy adaptation capability of the distributed storage system are improved.

The invention discloses a load prediction-based solid state disk power consumption optimization method and system, and particularly relates to the technical field of solid state disks, and the method comprises the following steps: S1, data acquisition; s2, constructing and training a load prediction model; s3, load prediction; s4, power consumption optimization: optimizing the power consumption of the solid state disk by adopting a multi-objective optimized dynamic power consumption adjustment algorithm according to a load prediction result; and S5, feedback adjustment: monitoring actual power consumption and performance parameters of the solid state disk after optimization in real time, and adjusting the load prediction model and the dynamic power consumption adjustment algorithm in combination with a reinforcement learning algorithm. According to the solid state disk power consumption optimization method and system based on load prediction, a set of complete and efficient solid state disk power consumption optimization scheme is formed from load prediction model construction, dynamic power consumption adjustment algorithm design and feedback adjustment mechanism optimization; and a new technical thought and a new practice direction are expected to be provided for reducing energy consumption and improving performance of the solid state disk.

The invention relates to the technical field of geographic information systems, in particular to a map tile data efficient access method and system based on hybrid storage and intelligent layering, and the method comprises the following steps: constructing a multi-level storage architecture, carrying out dynamic heat analysis, carrying out intelligent layering scheduling, preloading and space prediction, and carrying out transmission and access optimization. The method has the beneficial effects that the multi-level storage pool is constructed by combining the advantages of object storage and a distributed file system; adopting a dynamic tile popularity analysis model to realize hot data cache acceleration and cold data hierarchical archiving; and meanwhile, a tile request preloading mechanism is introduced, and an access path is optimized in combination with a spatial locality prediction algorithm. According to the method, the storage cost can be remarkably reduced, the tile request delay is reduced, quick response in a high-concurrency scene is supported, and the method is suitable for application scenes such as Internet map services and smart cities.

The invention discloses a data storage control method and device, a storage medium and electronic equipment, and relates to the technical field of computers, and the data storage control method comprises the steps that the access frequency of multiple pieces of data accessed in a future time period is predicted according to historical access characteristics of the multiple pieces of data in a storage system; detecting whether the data access types of the data are changed or not according to the frequency range in which the access frequency falls; under the condition that it is detected that the data access type of the target data is changed, a storage strategy of the target data is adjusted according to the target data access type obtained after the target data is changed, and a target storage strategy is obtained; and executing the target change operation on the target data, the target change operation being used for changing the storage strategy of the target data to the target storage strategy, thereby solving the technical problem that it is difficult for a storage system to consider both storage overhead and storage performance during data storage. The technical effect that the storage cost and the storage performance are both considered when the storage system stores the data is achieved.

The invention relates to the technical field of data processing, can be applied to business scenes of financial science and technology, medical treatment and health and the like, and discloses a data hierarchical storage method, device, equipment and medium based on access characterization. Writing the data into a matched first target storage device according to a preset device performance corresponding relation; after the data are stored, continuously monitoring subsequent access characteristics of the data, and dynamically judging whether the heat state is changed or not; and if the popularity state is changed and a specific migration condition is met, redistributing the second target storage device and completing data migration operation so as to realize automatic hierarchical storage management of cold and hot data. By acquiring the data access representation and dynamically identifying the popularity state, hierarchical migration of data with different popularity among storage devices with different performances is realized, the resource use efficiency and access performance of a storage system are improved, and the storage cost is reduced.

The invention discloses a storage resource management method and system and a storage medium, and relates to the technical field of storage resource management, and the method comprises the steps: calculating the access popularity and popularity threshold of data according to a historical log file, and carrying out the popularity layering of the data in the historical log file according to the two; the data distribution mapping table is generated based on the historical log file and the access popularity, and finally multiple storage resources are allocated based on the layered data and the data distribution mapping table, so that the problems that application scenes of related technologies are limited, and the performance of a storage system is difficult to flexibly deal with complex and changeable business environments are solved; the technical problem that smooth expansion and rapid iteration of a storage system cannot be achieved is solved, and the technical effects of dynamic allocation, efficient utilization, flexible configuration and safety protection of storage resources and capability of leisurely coping with various complex application scenes are achieved.

The invention relates to the technical field of cloud computing and distributed storage, and discloses a distributed metadata management method and storage system based on cloud computing, and the method comprises the following steps: S1, collecting the access frequency of a plurality of metadata objects in a set time period; s2, performing weighting processing on the access frequency, and constructing access heat distribution under multiple time scales; s3, obtaining the available cache capacity of a plurality of cache nodes, and constructing cache resource weight distribution; s4, assessing the access cost and migration cost between each metadata object and each cache node; and S5, generating a scheduling strategy between the metadata object and the cache node based on a preset optimization target, wherein the optimization target is access cost minimization. According to the method, a heat, resource and cost joint modeling mechanism is constructed, and optimal scheduling and feedback adaptive control are combined, so that accurate decision, dynamic adjustment and migration cost controllability of copy deployment are realized, and the performance and stability of distributed metadata management are effectively improved.

The invention discloses a distributed data storage management system and method, and relates to the technical field of distributed storage, and the method comprises the steps: obtaining an access log, metadata and content features of each node; data object attributes are judged, cold data are marked, and cold data distribution nodes are positioned; the method comprises the following steps: dividing cold data into permanent, periodic and sudden basic access modes through first-layer classification; generating a composite classification label through second-layer classification; matching a predefined strategy matrix according to a classification result, dynamically selecting a compression algorithm and a storage medium, and allocating bandwidth; operation auditing, compliance verification and exception handling are realized through the smart contract; and collecting strategy execution indexes in real time, and adjusting classification model parameters and resource scheduling weights. Through an intelligent cold data judgment and classification mechanism, cold data is accurately identified and managed, occupation of storage resources is reduced, and the overall utilization rate of the storage resources is improved.

The invention discloses a GPU (Graphics Processing Unit) and parallel IO (Input / Output) collaborative optimization method based on mode heterogeneous computing, and particularly relates to the field of heterogeneous computing optimizing.The method comprises the following steps: monitoring mode characteristics during execution of a target computing task in real time through an instruction analyzer, and synchronously collecting dynamic parameters of a heterogeneous computing environment to generate a multi-dimensional resource situation matrix; based on the mode and the resource situation matrix, collaborative optimization operation of the binding equipment dimension and the scheduling time dimension is executed; and constructing a three-level adaptive transmission mechanism and periodically implementing feedback optimization according to an execution state of the three-level adaptive transmission mechanism. According to the GPU and parallel IO collaborative optimization method based on mode heterogeneous computing, the data prefetching hit rate is increased by dynamically sensing the matching relation between a heterogeneous computing mode and an I / O access mode; end-to-end delay is reduced and task contention conflicts are reduced through a dynamic matching strategy of an equipment capability portrait and a task type; and the heterogeneous storage bandwidth peak value utilization rate and the I / O throughput are improved by differentially utilizing the hierarchical characteristics of the multi-stage storage.

The invention discloses a data fragment storage index optimization method and system, and relates to the technical field of data fragment storage, and the method comprises the steps: determining an observation time period according to a data access mode and short-term fluctuation characteristics; monitoring data performance, and performing log data analysis; and performing data resource adjustment, and adjusting task allocation. According to the method, data loading is efficiently managed and adjusted, the user experience and reliability of a system are improved, the importance of a dynamic data management method based on the observation time period is highlighted, hot spot fragments can be recognized in time when the load is too high through setting of a dynamic threshold value, and therefore effective measures are taken, and the workload of a user is reduced. According to the method and the device, the waste of system resources is reduced, the use efficiency of the resources is improved, data operation can be completed under the condition that a main thread is not interfered, the data transmission efficiency and the storage utilization rate are improved, and the overall processing capacity and fluency of the system are improved.

The invention discloses a storage optimization method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.According to the storage optimization method and device, the electronic equipment and the storage medium, the occupied space of caches is compressed by adopting retention and selection strategies, and when the caches are allocated to each network layer, the cache allocation efficiency is improved. According to the method, key value caches from a low layer to a high layer are distributed in a gradually decreasing mode, so that keyword elements are prevented from being omitted in the low layer, unimportant lexical elements are prevented from being stored in the high layer, and then the target attention value is obtained by utilizing the attention focusing degree of the input lexical elements and the cached lexical elements. The keyword elements which have the most influence on the input lexical elements are selected from the cache lexical elements to be cached, then representative representative lexical elements are selected from the remaining lexical elements to be stored, and only the lexical elements which have the influence on the model processing data can be stored. And the technical effect of compressing the occupied space of the KV cache so as to provide the capability of processing more requests is achieved.

The invention provides an online document processing method and system based on a document processing model, and the method comprises the steps: obtaining an editing request of a user for a target data unit, generating a temporary locking identifier according to a user authority identifier and a request timestamp, limiting the modification operation of other users for the target data unit, and obtaining a conflict isolation result; constructing a data consistency module according to a conflict isolation result, extracting editing state information of the current data unit from the temporary locking identifier, and performing sequential recording on concurrent operation by adopting a distributed transaction processing technology; if it is detected that version conflicts exist in the sequenced records, a rollback mechanism is triggered, operation priorities are redistributed, and data versions after consistency verification are determined; and designing a hierarchical cache structure according to the data version after consistency verification, storing the high-frequency access data into a memory cache region, and storing the low-frequency access data into a disk cache region to obtain a hierarchical storage structure. The data consistency and security are good, and the storage and access efficiency is improved.

The present application relates to the technical field of data storage. Disclosed are a multi-source heterogeneous distributed system, a memory access method and a nonvolatile readable storage medium. The method comprises: a first unified interconnection bus unit receiving, by means of a protocol adaptation interface module, a read request sent by a first device, and setting the cache state of a first memory unit to be an invalid state; sending to a second unified interconnection bus unit of a second device a request message obtained by packaging the read request, the second unified interconnection bus unit reading data from a second memory unit; setting the cache state of the second memory unit to be a shared state, and sending to the first unified interconnection bus unit a response message, the first unified interconnection bus unit storing in the first memory unit data carried by the response message; and setting the cache state of the first memory unit to be a shared state. Thus, deploying a unified interconnection bus unit on each device can achieve a consistent memory communication between the devices, thereby achieving memory sharing, and further improving the performance of the multi-source heterogeneous distributed system.

The invention provides a real-time deep sea subsurface buoy monitoring system, which belongs to the technical field of deep sea measurement, and comprises a control chip, a multi-parameter sensor array, a data storage device and a power supply, high-precision monitoring is realized through the following steps: recording ocean data acquired by a multi-parameter sensor; correcting the sound wave propagation model and calculating the relative position of the subsurface buoy; constructing a three-dimensional data matrix and realizing data compression; identifying abnormal data by using a matrix disorder index; processing the time sequence data by adopting a sliding window weighted average method and eliminating jump; carrying out distributed preprocessing on the multi-source heterogeneous data and carrying out parallel processing by applying a matrix partitioning technology; accumulated errors are corrected by combining historical data and applying a Kalman filtering algorithm, and the problems of environment interference and sensor drift are solved through a deep sea environment disturbance compensation network model and a self-adaptive weighted error optimization function.

The invention discloses a solid state disk management system and a data processing method, and relates to the technical field of solid state disk management. Flexible task scheduling and resource management are provided through a lightweight operating system module, and a computing task program defined by a user is supported to be dynamically loaded; edge computing or machine learning operators are efficiently executed in combination with a programmable hardware processing unit and a DMA channel of the computing acceleration engine module, the data preloading module is utilized to predict and preload data to DDR based on LBA access history, access delay is reduced, an NVMe protocol is expanded by means of the task unloading interface module, host task issuing and result returning are achieved, and the data processing efficiency is improved. And hardware-level memory protection is ensured through the security isolation module, so that the data calculation processing capacity of the solid state disk is remarkably improved, localized calculation tasks such as edge calculation and machine learning are supported, mass data transmission is effectively reduced, and bus and network loads are relieved.

The invention provides a key value cache data scheduling method and device and a large model reasoning method and device, and relates to the technical field of artificial intelligence. The method comprises the following steps: predicting target key value cache data required by a large model for reasoning a subsequent token by utilizing a prediction model; the subsequent tokens refer to tokens which are not reasoned by the large model; judging whether the target key value cache data needs to be scheduled or not; if scheduling is needed, generating a scheduling strategy; obtaining target key value cache data from the first storage space according to a scheduling strategy, and storing the target key value cache data to a second storage space; wherein the target key value cache data is used for enabling the large model to infer subsequent tokens. According to the method and the device, frequent access to the first storage space is reduced, and only the required target key value cache data is transmitted every time instead of full-amount key value cache data, so that occupation of bandwidth resources caused by transmission of the key value cache data is reduced.

The invention provides a distributed storage method based on source code semantic partitioning, and particularly relates to the technical field of cloud data distributed storage. The method comprises the steps of performing semantic partitioning on a source code, and segmenting the source code into a plurality of semantic blocks according to dimensions such as functional semantics, an abstract syntax tree structure, author information and version information; generating metadata containing information such as grammar type tags, file paths, line number ranges, author identifiers, version identifiers and access popularity for each semantic block; constructing a weighted directed acyclic graph (DAG) based on the semantic chunks and the dependency relationship thereof; superposing a metadata layer in the DAG structure, and recording information such as function call dependency, inter-block reference relationship and version evolution chain; blocks with relatively high access frequency and close semantics are aggregated into super blocks, the traversal depth is reduced, and meanwhile, hot data and cold data are differentiated for hierarchical storage by adopting a cold and hot data management strategy; and evaluating a parent block aggregation degree through a BDS algorithm, determining a block sorting priority, and optimizing super block boundary division. Compared with the prior art, the method has the advantages that the semantic retrieval efficiency, the incremental updating capability and the distributed query performance of the source code storage system are improved.

The invention discloses a multi-level cache optimization method and system based on data popularity self-adaptive adjustment, relates to the technical field of data storage, and aims to solve the problems that a cache mechanism in an existing desktop cloud system is static, initialization is slow, and cold and hot data are difficult to recognize. The adopted scheme comprises the steps that a three-level cache structure of a local cache L1, a distributed cache L2 and a database L3 is set; constructing a data popularity scoring model based on the access frequency, the time decay and the user priority; dividing the data into hot data, temperature data and cold data according to the popularity score, and dynamically determining migration and elimination strategies of the data in L1, L2 and L3; the hot data are loaded to the upper-level cache T in advance through a prediction mechanism, and container-level cache preheating is supported; a cache preheating strategy is integrated to a Kubernetes life cycle, and hot data loading and cold data degradation are achieved; visual strategy configuration and monitoring during operation are supported, and cache strategies are managed in real time. The method is suitable for an access optimization scene of a large-scale desktop cloud system.

The invention discloses a computing platform management system based on artificial intelligence, particularly relates to the technical field of distributed storage resource dynamic allocation, and is used for solving the problem of resource allocation oscillation caused by hardware performance difference and dynamic attenuation in the prior art. Acquiring the hardware performance, the load state and the data path of the storage node through a real-time acquisition module; the entropy judgment module calculates a path topology entropy and triggers a reconstruction instruction; the resonance analysis module analyzes the group association strength of hardware performance degradation and marks a resonance node group; the weight generation module fuses the parameters to generate a dynamic allocation weight; the error compensation module optimizes a weight coefficient in combination with historical error data; the migration execution module executes migration according to the weight and limits the load of the resonance node; through cooperative regulation and control of topological entropy monitoring and hardware resonance analysis, local oscillation diffusion is inhibited, unnecessary migration operation is reduced, system stability is improved, and the service life of hardware is prolonged.

The invention discloses a data dynamic migration and storage method, device and equipment and a medium, and relates to the technical field of computers. The method comprises the steps that environment parameters of a current mechanical hard disk in a distributed cluster are collected; wherein the environment parameters comprise a vibration parameter, a temperature parameter and a noise parameter; quantifying the input and output delay performance of the current mechanical hard disk by using the environmental parameters of the current mechanical hard disk; if the input and output delay performance of the current mechanical hard disk is greater than a preset threshold value, searching a target mechanical hard disk combination in the distributed cluster; and migrating and storing the target data block of the current mechanical hard disk into the target mechanical hard disk combination based on an erasure code and a multi-copy mechanism. Through the scheme, the reliability of storing the data in the mechanical hard disk in the distributed cluster is remarkably improved.

The invention discloses a storage node load balancing method and device, equipment and a storage medium, and relates to the technical field of load balancing, and the method comprises the steps: carrying out the real-time monitoring of the state of each storage node, and collecting the real-time load information; predicting the load trend of each storage node based on the real-time load information and the historical load data of each storage node to obtain a corresponding prediction result; and adjusting each storage node and a corresponding data access strategy according to the real-time load information and the prediction result so as to respond to the related data request based on the data copy in the adjusted storage node. Therefore, the storage nodes and the data access strategy can be adjusted in combination with the real-time load information of each storage node and the prediction result of the corresponding load trend; the load change trend of each storage node can be accurately mastered, the storage nodes and access strategies are dynamically optimized, the load balancing capacity of the storage nodes can be improved, and diversified service requirements are met.