50results about How to "Reduce I/O operations" patented technology

The invention discloses a log data writing method and a log system. The method comprises the following steps of: obtaining generated log data and writing the log data into a memory; when the log data in the memory reach a preset number, writing all log data stored in the memory into a disk; or, after obtaining the generated log data and writing the log data into the memory and before writing all log data into the disk, writing all log data into the disk if the log data stored in the memory exist when a preset disk writing time is finished. Through the adoption of the log data writing method and the log system disclosed by an embodiment of the invention, the operation of immediately writing the generated log data into the disk once the log data are generated in the prior art can be reduced, I/O (Input/Output) operations of the disk are reduced, and the frequency of writing the log data into the disk for filing under a high-concurrency application of the log data is reduced, so that the time of writing the log into the disk is reduced; meanwhile, the power consumption of the system during writing the data into the disk is reduced and finally a purpose of quick response is achieved.

The invention discloses a data dimension reduction method based on a parallel principal component analysis (PCA) algorithm. The method comprises the steps of: S1, constructing a sample data matrix D<nxm> by high-dimensional data of which dimensions are to be reduced; S2, calculating a covariance matrix C<mxm> of the sample data matrix D<nxm>; S3, calculating m feature values of the covariance matrix C<mxm> and m corresponding feature vectors; S4, determining the number k of principal components according to the feature values and the feature vectors; and S5, utilizing the feature vectors, which correspond to the top-k greater feature values, to construct a transformation matrix, and utilizing the transformation matrix to calculate a principal component matrix, wherein the principal component matrix is data of which the dimensions are reduced. According to the method, the problem that according to a traditional stand-alone principal component analysis algorithm, the data cannot be loaded into a memory at once because a data size is too large is overcome, I/O operations are reduced, and the processing efficiency of data dimension reduction is improved.

The invention relates to a vector data preorder quadtree coding and indexing method based on the Key / Value type NoSQL (Not only SQL). According to the method, the data physical storage order and the spatial scale are consistent in continuity, the data major key order and the data physical storage order are consistent, and therefore input / output (I/O) operation is reduced during spatial query, and query efficiency is improved. The method comprises the steps of complete quadtree space division and preorder quadtree node coding, vector data prefix coding and index establishment, and preorder quadtree indexing.

The invention discloses an OWGA memory cache based mass remote sensing tile data quick releasing method. The method is characterized in that considering three attributes including the access time, theaccess frequency, and the key value of tile data; establishing an OWGA operator according to the access time and the access frequency of a requesting target tile, and using the key value as an uniqueidentifier of a tile, wherein the key value consists of the hierarchy, a line number and a row number of the tile; adopting a Map < the key value of the tile and a tile object > a chain table to manage the tile object in a memory; preferentially performing descending sorting according to the OWGA value of the tile object, and performing ascending sorting according to the key value so as to save the tile data which is frequently requested by a client recently and has high resolution in the memory all the time, and reduce time-consuming file I/O operations. The method provides a strategy that aplurality of tiles near the requesting target tile are preloaded in a memory cache so as to predetermine a purpose of a client request, can further improve the cache hit ratio, and can accelerate theresponse speed of a releasing server.

The invention provides a read-write solution for tens of millions of small file data. According to the solution, the mode of creating large-block continuous disk space is used for storing a large number of small files when the small files are stored, namely, logically continuous data are stored on the continuous space of a disk array as far as possible; the disk space is divided into a plurality of blocks, and the size of each block is 64 KB. According to the basic thought, each small file can only be stored in the single block and cannot be stored by crossing two blocks, each folder can be provided with one or more blocks, the blocks are only used for storing the data of the corresponding folder, and each piece of file data is stored on the continuous disk space. Compared with the prior art, the logically continuous data are stored on the continuous space of each physical disk as far as possible, the cache technology is used for playing the role of a metadata server, the cache utilization rate is improved through simplified file information nodes, and thus the access performance of the small files is improved.

The invention discloses a migration and erasure code-based reconstruction coupled rapid prediction repair method. The method comprises the steps of monitoring a node state by adopting a node failure prediction algorithm; organizing the blocks on the nodes about to fail into a plurality of reconstruction sets; and determining a coupling strategy of the migration and erasure code-based reconstruction according to the number of blocks contained in the reconstruction sets. According to the teaching of the above embodiment, the migration and erasure code-based reconstruction coupled rapid prediction repair can be realized by combining the failure prediction, thereby not only accelerating the data restoration, but also reducing the I/O operation needed by the repair compared with the reconstruction based on the erasure codes. The invention further discloses a migration and erasure code-based reconstruction coupled rapid prediction repair device.

The embodiment of the invention discloses an open platform and a system for processing reports and a report processing method. The open platform for processing the reports is accessed into an in-memory database to serve as a data source of an on-line report, so that the data of the in-memory database can be synchronized, visited and statistically processed, real-time of the data is ensured and meanwhile processing and generating efficiencies of the reports are greatly improved, the use ratio of computer resources is optimized, meanwhile consistency of the in-memory database and mainframe source data is maintained by a data synchronization module, the data of the in-memory database can be visited and processed by a report analysis engine, I/O operation when in data synchronization and processing is reduced, real-time and higher operation parallelism of the data are ensured, and performance of the whole system is improved.

The invention discloses a method used in the I/O of iSCSI target device block equipment. The method comprises the steps that page high speed cache is used and a Radix-Tree-based searching algorithm is adopted between interfaces of the iSCSI target device and the storage equipment. The method aims at enhancing the I/O performance of a storage system, exchanging time by using space, as well as using high-efficient searching algorithm to enhance executing efficiency. The invention adopts the method for improving a software system, basically has nothing to do with the characteristics of hardware equipment and has good universal external interfaces.

The invention discloses a data caching method and device, equipment and a storage medium, and relates to the technical field of computers, and the method comprises the steps that a data acquisition request initiated by a target client for target data is acquired, after the data acquisition request reaches a gateway, the gateway judges whether a first-level cache located in a local memory hits the target data or not according to a target caching rule, and if yes, the first-level cache in the local memory hits the target data; if the target data is hit, obtaining the target data from the first-level cache and returning the target data to the target client, if the target data is not hit, judging whether the target data exists in a second-level cache of the preset cache middleware or not, and if the target data exists in the second-level cache, obtaining the target data from the second-level cache and returning the target data to the target client, and if not, obtaining the target data from the service system and returning the target data to the target client. According to the method, network transmission and I/O operation of a disk are reduced in a multi-level cache mode, the client request efficiency, the CPU resource utilization rate and the query frequency processed by the system in unit time are improved, and meanwhile cache data can be prevented from being lost.

The invention discloses a construction method for local unequal codes. According to the solution, for an information symbol local unequal situation, a parity check bit matrix of maximum distance separable (MDS) codes is separated first, and then any separated subset is further separated according to a certain rule to obtain information symbol local unequal codes. For an all-symbol local unequal situation, Gabidulin coding is performed on a vector m of an F<k>qm domain first, then the coding result is divided twice, and MDS coding is performed on the second division result to obtain all symbol local unequal codes. The method effectively solves the problem that the minimum distance of existing local repair codes cannot reach the upper limit, correspondingly reduces the repair locality and the I/O operation in the repair process, and keeps a relatively low repair bandwidth in the repair process.

Embodiments of the invention disclose a data processing method and apparatus. The method comprises the steps of obtaining multiple pieces of to-be-processed data; defining the to-be-processed data in a plurality of preset intervals; determining approximate values of the to-be-processed data in the intervals; and according to the approximate values, determining quantiles corresponding to the to-be-processed data. By utilizing the method and the apparatus, the approximate values are used for replacing the numerical values of the to-be-processed data defined in any interval, so that the quantity of data participating in a calculation process can be reduced. Therefore, the traversal frequency during quantile calculation can be reduced; the I/O operation on a database can be further reduced; and the efficiency of the quantile calculation process can be improved to a certain extent.

The present invention discloses a real-time data report statistical method under the massive data in the field of data statistics. The method is characterized in that: while application data is storedin a database, statistical data is written into an in-memory database in real time according to statistical characteristics of the data; according to the statistical characteristics of the data, fromthe current statistical period, the statistical data of the previous statistical period is stored by the in-memory database in the database; and when a user views statistical report information, a data collection processor performs statistical analysis according to different statistical characteristics, extracts data from the database or the in-memory database, and returns a statistical result. According to the method disclosed by the present invention, the purposes of collecting statistics of real-time data and having relatively high performance results are achieved, and the user experienceis further improved.

The invention relates to a telemetering data access method for a ground station of an unmanned airship system, which comprises the following steps: saving the telemetering data of the unmanned airship system as an internal memory mapping data base comprising a plurality of data files, simultaneously establishing a corresponding external memory data base, and establishing worksheets corresponding to the data files of the internal memory mapping data base one to one in the external memory data base; then setting a historical variable and a current variable for each data file in the internal memory mapping data base, wherein the historical variables are used for marking recording positions in the external memory data base, and the current variables are used for marking the current recording position in the internal memory mapping data base; and finally, saving updated data to the internal memory mapping data base and saving the data files from the historical variables to the current variables in the internal memory mapping data base to the external memory data base at regular time to replace the complex and expensive internal memory mapping data base, realize the real-time monitoring on the unmanned airship and meet the demands on telemetering data access in the aspects of real-time performance, reliability and usability.

The invention discloses a method, device and system for allocating port rule information, which are applied in a network system including a server device, a port rule database and at least one client device, wherein the method includes the following steps: the server device Obtain all port rule information in the port rule database from the port rule database; the server device stores the obtained port rule information; the server device submits a query port rule request according to the client , sending the corresponding port rule information to the client device. By using the invention, the I/O operation from the server end to the port database is saved, and at the same time, high concurrency of data transmission can be achieved, and a large number of monitoring requests can be supported.

The invention relates to the technical field of cloud data, and provides a line migration elimination method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps of detecting a recording line with line migration, determining a line identifier corresponding to the recording line, searching a current data block where the recording line is located after line migration and an original data block where the recording line is located before line migration based on the line identifier, and performing line migration on the current data block and the original data block. And determining a residual free space of the original data block and a storage space occupied by the record line, when the residual free space is not smaller than the storage space, migrating the record line in the current data block into the original data block, and deleting a pointer pointing to the current data block in the original data block, the additional I/O operation of the database is reduced, and the data processing performance of the database is improved on the whole.

The invention discloses a buffering capacity allocation method for massive logs. The method comprises an eleventh step of reading the log in a sublist in real time; a twelfth step of counting citation times of an offset of a domain appears for the first time; a thirteenth step of establishing citation volume of each segment, and computing total citation volume of the sublist; a fourteenth step of performing linear fitting on each segment of citation volume; and a fifteenth step of allocating a preset total buffering capacity in the sublist for each segment. The method has the beneficial effects of reasonably allocating the buffering capacity, occupying less memory resources, and reducing input/output (I/O) operation due to citation of domain offset.