46results about How to "Reduce IO operations" patented technology

The invention discloses an EXCEL data export method in a multi-concurrence large data volume environment. The EXCEL data export method in the multi-concurrence large data volume environment comprises the following steps that (a) configuration information in a configuration file is set, the configuration information comprises data export format information, data table structure information, and the information about the data volume output every time, and the data export format is an xml format; (b) the configuration information is analyzed; (c) user inquiry conditions are obtained; (d) an executable statement of a database is generated, the executable statement of the database is generated according to the user inquiry conditions and the information about the data volume output every time; (e) data conversion and data output are carried out, the data needing exporting are converted and output in batches, and the memory is released every time output is finished. According to the EXCEL data export method and export device in the multi-concurrence large data volume environment, the output data format is defined as the xml format, the EXCEL files with a large data volume are exported in batches, the memory is released every time exportation is finished, and therefore occupation of the memory in the data downloading process can be largely reduced.

The present invention discloses a data block storage method and reading method, apparatuses thereof, and a block chain. The storage method comprises: before writing a new data block in the block chain, converting transaction record data in each row in a transaction record structure of the new data block to the processing of storing the data in a column structure; creating column structure storageidentification information in the new data block stored in a column structure; and storing the new data block with the column structure storage identification information into the block chain. Since the transaction data in each row is stored in a column structure, and the column structure storage identification information is created, when querying the transaction record data, instead of queryingthe entire transaction record data, only the transaction record data of the corresponding column needs to be queried according to the identification information; a large number of system IO operationsare reduced, large-capacity and high-concurrent query requirements can be supported, and efficient data processing can be facilitated; and in addition, by extending the columnar storage structure, changes in the future transaction record details can be handled.

The invention discloses a video file processing method, a video file processing device and a video file processing system. The video file processing method comprises the following steps of: receiving a transcoding request message from a client by using a metadata server, wherein the transcoding request message comprises the metadata information of a source file fragment and a target coding format; acquiring the metadata information of a target file fragment corresponding to the metadata information of the source file fragment according to the metadata information of the source file fragment; and if a coding format of the target file fragment is the same as the target coding format in the transcoding request message, transmitting the metadata information of the target file fragment to the client, so that the client can acquire the target file fragment corresponding to the metadata information of the target file fragment according to the metadata information of the target file fragment. Network input and output operation can be reduced, and the application performance of video transcoding can be improved.

The invention discloses a method for semantic query of a large-scale knowledge graph based on spark. The method comprises the steps that (1) entities and relations in each triple are replaced by corresponding id; (2) a layered sub-graph index is established based on categories and relations and the index is stored in a hdfs document; (3) operations related to sparql query are translated by an spark operation primitive language; (4) different score functions are distributed according to characteristics of each triple mode, and an execution sequence of each triple mode in the sparql query is determined; and (5) according to the execution sequences of the triple modes and the spark operation primitive language, query and linkage are executed, and then a linkage result is analyzed by a mapping list and then returned. According to the invention, efficient query of massive semantic data is supported; extendibility is very high; and the method has very high practical values for query and application based on large-scale semantic data.

The invention relates to a rapid retrieval method and system for mass website basic information. The method comprises the steps of collecting website basic information, and converting the basic information into a json formatted file of a fixed field to be used as a retrieval object; constructing a reverse index system; rapidly retrieving by use of a full-text retrieval method; and maintaining the reverse index system by use of a B tree structure. Through adoption of the method, working time is saved, and the retrieving efficiency is greatly improved.

The invention discloses a massive vehicle-mounted machine track data storage method which can meet the storage requirement of TB-grade track data generated every day under the condition of access of massive vehicle-mounted machine, and can search track data of a certain vehicle in a short time. The storage method takes a file system as a storage medium, the file system is divided into directory structures with three layers according to years, months and days, each vehicle-mounted machine generates a track file every day, the track file is stored in corresponding directories, and the track files in a previous day are combined and compressed at 0 o'clock every day.

The present invention relates to an optimized and accelerated disk monitoring method and system. The disk monitoring method comprises the following steps of (1) obtaining the type of disk related information to be queried, and determining whether the disk related information of the type is changed, if so, executing the step (2), and if not, executing the step (3); (2) executing the corresponding database update operation according to the type of the disk related information to be queried, and then executing the step (3); and (3) querying the disk related information in the database, and obtaining the current state of a disk. Compared with the prior art, the problems that the time for refreshing disk data is longer, and the data displayed in pages are different from actual data in a back-end system are solved, IO operation for querying the disk information is decreased, and the influence of monitoring information query on the system is reduced.

The invention discloses a synchronous log copying control method and system for a disaster recovery system. The method comprises presetting memory data block ID (identity) record sets for memory logs and disk data block ID record sets for disk logs, wherein the memory data block ID record set is identical to the disk data block ID record set, and defining all the disk data block ID record sets to be data block sets with data correction and requiring synchronous copying; in real time acquiring data correcting operation information of a production center and a disaster recovery data to acquire data blocks IDs to be synchronously copied; determining whether the corrected data blocks IDs belong to the memory data block ID record sets; if so, forbidding triggering of update of the disk logs, otherwise, writing the corrected data block IDs into the memory data block ID record sets and the disk data block ID record sets. The synchronous log copying control method and system for the disaster recovery system exploits the time and space limitation of data correction operation of storage systems, reduces write-in delay of the disaster recovery system and improves disaster recovery copying performance.

The invention discloses a low-latency file system address space management method and system and a medium. The method comprises the steps that a super block and a block group distribution table are generated from an address space of storage equipment, file system information and the distribution situation of a block group on a linear address space of the storage equipment are stored in the super block, and the block group distribution table is used for marking the distribution situation of data blocks in the corresponding block group; when the file is created, dynamically creating or selectinga corresponding block group according to the size of a specified data block, allocating the data block, writing file data into the allocated data block, and updating the information of a block groupallocation table and a super block. It can be guaranteed that which data blocks are occupied by one file on the storage device can be known only through one IO operation, the read-write delay of the file can be remarkably reduced, the addressing expenditure of file read-write can be remarkably reduced, the file read-write sequence can be remarkably improved, and therefore the read-write performance is improved.

The invention discloses a device and a method used for executing an Adam gradient descent training algorithm. The device includes a direct memory access unit, an instruction cache unit, a controller unit, a data cache unit and a data processing module. The method includes: firstly, reading a gradient vector and a to-be-updated value vector, and initializing first moment and second moment vectors and corresponding exponential decay rates at the same time; during each iteration, utilizing the gradient vector to update the first moment and second moment vectors, respectively calculating a first biased moment estimation vector and a second biased moment estimation vector, and utilizing the first biased moment estimation vector and the second biased moment estimation vector to update to-be-updated parameters, and continuing training until the to-be-updated parameter vector converges. By utilizing the device and the method, application of the Adam gradient descent algorithm can be realized, and the efficiency of data processing can be greatly improved.

The invention provides a block chain data processing method and device and a terminal device. The block chain data processing method comprises the steps of obtaining the block data to be written intoa block chain, performing column compression processing on the transaction record row data in the block data according to a transaction record column to obtain the block column compressed data, and writing the block column compressed data into the block chain. According to the scheme provided by the invention, by improving the structure of the block data written into the block chain, when the transaction record of the block is queried, the IO operation of the system can be greatly reduced, the resource waste is avoided, the high-concurrency query requirement is met, and the data is efficientlyprocessed.

The invention provides a recording method and device of ordered data and an access method and device of the ordered data. The recording method comprises the following detailed steps: creating a magnetic disk file for the ordered data; recording the ordered data in the magnetic disk file according to the block mode, and the quantity of blocks is two or more. The recording process comprises the step of recording the field interval of the corresponding ordered data and the recording number in the field interval in each block, wherein the ordered data in a database is stored in order according to the field. The invention has the advantages that data only needs to be read from the blocks and corresponding intervals covered by the records corresponding to the content of the specific pages, and the redundant data can be prevented from being read as much as possible, so that the IO (Input/Output) operation of a magnetic disk is reduced.

The invention discloses a rapid association classification and hierarchical storage method for network security logs, which comprises the following steps: defining an XML (Extensible Markup Language) format log template according to different application metadata, the log template comprising alarm log information, association flow log information, application metadata log information, message log information and file attachment log information; associating the alarm logs, the associated flow logs, the application metadata logs, the message logs and the file attachment logs together through flow and transaction information in the logs, wherein each alarm log outputs a complete log; performing hierarchical storage mechanism output on logs, including values in alarm log information, associated flow log information, application metadata log information, message log information and file attachment log information; the hierarchical storage mechanism adopts two levels, the first level adopts a memory file system, and the second level adopts an SSD/HD storage system with RAID. According to the technical scheme, the performance problem of Json log format output is solved.

The invention provides a data storage management method, device and equipment, and the method comprises the steps: responding to a data writing instruction, writing a data block into zone, taking a unique identifier of the data block as a key, taking the position information of the data block as a value, generating a data block index in a key-value form, and writing the data block index into a storage region corresponding to index information in a memory; when the data volume of the to-be-stored area reaches a set threshold value, packaging the data of the storage area into a file and storing the file in the CMR area; in response to the data reading instruction, retrieving a first target key matched with a target identifier in the instruction in the data block index; the corresponding first target value is positioned according to the first target key, and the target data block is read according to the position information of the data block in the first target value, so that efficient storage and management of the data block and the metadata index information thereof by using the SMR disk are realized.