Patents
Literature
Hiro is an intelligent assistant for R&D personnel, combined with Patent DNA, to facilitate innovative research.
Hiro

46results about How to "Reduce IO operations" patented technology

EXCEL data export method and export device in multi-concurrence large data volume environment

The invention discloses an EXCEL data export method in a multi-concurrence large data volume environment. The EXCEL data export method in the multi-concurrence large data volume environment comprises the following steps that (a) configuration information in a configuration file is set, the configuration information comprises data export format information, data table structure information, and the information about the data volume output every time, and the data export format is an xml format; (b) the configuration information is analyzed; (c) user inquiry conditions are obtained; (d) an executable statement of a database is generated, the executable statement of the database is generated according to the user inquiry conditions and the information about the data volume output every time; (e) data conversion and data output are carried out, the data needing exporting are converted and output in batches, and the memory is released every time output is finished. According to the EXCEL data export method and export device in the multi-concurrence large data volume environment, the output data format is defined as the xml format, the EXCEL files with a large data volume are exported in batches, the memory is released every time exportation is finished, and therefore occupation of the memory in the data downloading process can be largely reduced.
Owner:成都交大光芒科技股份有限公司

Metadata management method and system and medium

The invention discloses a metadata management method, system and a medium. All directories and files in a file system are organized by adopting a tree structure, and all the directories and sub-filesunder a single directory are used as a whole and are sequentially stored in a continuous address space of bottom-layer storage equipment according to a creation sequence. When the metadata is updated,updating operations of all the subdirectories / subfiles in one directory are added to the tail of the directory in sequence in a log form, so that only one IO request is initiated in each metadata operation. When metadata retrieval is carried out, a sub-catalog is analyzed under a parent catalog, all data on a continuous address space corresponding to the parent catalog are firstly read, and thenmultiple acceleration methods are adopted for searching and matching. According to the method, the large directory problem can be effectively solved, directory path analysis has lower delay, the IO request number in the metadata access process can be reduced, and metadata access of a file system can be accelerated.
Owner:SUN YAT SEN UNIV +1

Data block storage method and reading method, apparatuses thereof and block chain

The present invention discloses a data block storage method and reading method, apparatuses thereof, and a block chain. The storage method comprises: before writing a new data block in the block chain, converting transaction record data in each row in a transaction record structure of the new data block to the processing of storing the data in a column structure; creating column structure storageidentification information in the new data block stored in a column structure; and storing the new data block with the column structure storage identification information into the block chain. Since the transaction data in each row is stored in a column structure, and the column structure storage identification information is created, when querying the transaction record data, instead of queryingthe entire transaction record data, only the transaction record data of the corresponding column needs to be queried according to the identification information; a large number of system IO operationsare reduced, large-capacity and high-concurrent query requirements can be supported, and efficient data processing can be facilitated; and in addition, by extending the columnar storage structure, changes in the future transaction record details can be handled.
Owner:CHINA MOBILE COMM LTD RES INST +1

Memory-based high-availability transaction matching system and method

The invention discloses a memory-based high-availability transaction matching system and method, and belongs to the technical field of digital currency transaction. A high tradable matching system comprising a user client, a gateway, an order module, a matching engine cluster module, a matching transaction algorithm module, an asynchronous persistence module, a memory reconstruction module and a zookeeper node registration module is constructed; the matching of the matching engine cluster module and the zookeeper node registration module is used to realize the distribution of transaction pairs and the progress of matching tasks, and the matching transaction algorithm module is used to realize the transaction of orders. The problems that in the prior art, the database load is high, the high availability characteristic is reduced, and the cluster cannot be extended transversely are solved.
Owner:必成汇(成都)科技有限公司

Video file processing method, device and system

The invention discloses a video file processing method, a video file processing device and a video file processing system. The video file processing method comprises the following steps of: receiving a transcoding request message from a client by using a metadata server, wherein the transcoding request message comprises the metadata information of a source file fragment and a target coding format; acquiring the metadata information of a target file fragment corresponding to the metadata information of the source file fragment according to the metadata information of the source file fragment; and if a coding format of the target file fragment is the same as the target coding format in the transcoding request message, transmitting the metadata information of the target file fragment to the client, so that the client can acquire the target file fragment corresponding to the metadata information of the target file fragment according to the metadata information of the target file fragment. Network input and output operation can be reduced, and the application performance of video transcoding can be improved.
Owner:深圳圣马歌科技有限公司

Data platform system based on real-time streaming computing

The invention discloses a data platform system based on real-time streaming computing. The data platform system comprises the following steps: acquiring and storing behavior data information of predetermined data, converting the predetermined data into a series of short and small data streams, and processing real-time streaming data with high throughput and a fault-tolerant mechanism. According tothe data platform system, behavior data information of predetermined data is converted into a series of short batch processing operations, and key information is analyzed and extracted, so that large-batch data are converted into micro-batch data, and the data are rapidly calculated in a distributed mode so as to achieve the effects of large throughput and low delay and satisfy the requirement ofa data release system for high timeliness, and being suitable for services sensitive to time delay.
Owner:HUITONGDA NETWORK CO LTD

Data storage method, device and equipment of full flash memory array and readable storage medium

The invention discloses a data storage method of a full flash memory array. The data storage method comprises the following steps: detecting whether the amount of data cached in a performance layer ofthe full flash memory array is greater than a predetermined threshold value or not; if yes, writing the cache data in the performance layer back to the capacity layer in batches in a data reduction mode, and generating metadata logs of batch operation; and putting the metadata log into the capacity layer. Thus, in order to avoid frequent IO (Input / Output) operation caused by applying the data capacity reduction method to SSD (Solid State Disk) cache, in the scheme, a batch operation mode is adopted to reduce frequent IO operations, and the batch operation not only comprises batch processing of cache data through a data reduction mode and writing back of the cache data to a capacity layer, but also comprises recording of the generated metadata into metadata logs, and batch disk falling tothe capacity layer, so that SSD cache performance is improved. The invention further discloses a data storage device and equipment of the full flash memory array and a computer readable storage medium, and the technical effects can also be achieved.
Owner:SANGFOR TECH INC

Method for semantic query of large-scale knowledge graph based on spark

The invention discloses a method for semantic query of a large-scale knowledge graph based on spark. The method comprises the steps that (1) entities and relations in each triple are replaced by corresponding id; (2) a layered sub-graph index is established based on categories and relations and the index is stored in a hdfs document; (3) operations related to sparql query are translated by an spark operation primitive language; (4) different score functions are distributed according to characteristics of each triple mode, and an execution sequence of each triple mode in the sparql query is determined; and (5) according to the execution sequences of the triple modes and the spark operation primitive language, query and linkage are executed, and then a linkage result is analyzed by a mapping list and then returned. According to the invention, efficient query of massive semantic data is supported; extendibility is very high; and the method has very high practical values for query and application based on large-scale semantic data.
Owner:ZHEJIANG UNIV

Rapid retrieval method and system for mass website basic information

The invention relates to a rapid retrieval method and system for mass website basic information. The method comprises the steps of collecting website basic information, and converting the basic information into a json formatted file of a fixed field to be used as a retrieval object; constructing a reverse index system; rapidly retrieving by use of a full-text retrieval method; and maintaining the reverse index system by use of a B tree structure. Through adoption of the method, working time is saved, and the retrieving efficiency is greatly improved.
Owner:NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT

Massive vehicle-mounted machine track data storage method

The invention discloses a massive vehicle-mounted machine track data storage method which can meet the storage requirement of TB-grade track data generated every day under the condition of access of massive vehicle-mounted machine, and can search track data of a certain vehicle in a short time. The storage method takes a file system as a storage medium, the file system is divided into directory structures with three layers according to years, months and days, each vehicle-mounted machine generates a track file every day, the track file is stored in corresponding directories, and the track files in a previous day are combined and compressed at 0 o'clock every day.
Owner:北京中交兴路车联网科技有限公司

Optimized and accelerated disk monitoring method and system

The present invention relates to an optimized and accelerated disk monitoring method and system. The disk monitoring method comprises the following steps of (1) obtaining the type of disk related information to be queried, and determining whether the disk related information of the type is changed, if so, executing the step (2), and if not, executing the step (3); (2) executing the corresponding database update operation according to the type of the disk related information to be queried, and then executing the step (3); and (3) querying the disk related information in the database, and obtaining the current state of a disk. Compared with the prior art, the problems that the time for refreshing disk data is longer, and the data displayed in pages are different from actual data in a back-end system are solved, IO operation for querying the disk information is decreased, and the influence of monitoring information query on the system is reduced.
Owner:EISOO SOFTWARE

Synchronous log copying control method and system for disaster recovery system

The invention discloses a synchronous log copying control method and system for a disaster recovery system. The method comprises presetting memory data block ID (identity) record sets for memory logs and disk data block ID record sets for disk logs, wherein the memory data block ID record set is identical to the disk data block ID record set, and defining all the disk data block ID record sets to be data block sets with data correction and requiring synchronous copying; in real time acquiring data correcting operation information of a production center and a disaster recovery data to acquire data blocks IDs to be synchronously copied; determining whether the corrected data blocks IDs belong to the memory data block ID record sets; if so, forbidding triggering of update of the disk logs, otherwise, writing the corrected data block IDs into the memory data block ID record sets and the disk data block ID record sets. The synchronous log copying control method and system for the disaster recovery system exploits the time and space limitation of data correction operation of storage systems, reduces write-in delay of the disaster recovery system and improves disaster recovery copying performance.
Owner:SANGFOR TECH INC

Method and device for selecting resources according to system configurations

The invention discloses a method for selecting resources according to system configurations. The method comprises the steps that according to identification information of a resource to be selected, a pre-generated correspondence set of resources and the configuration combinations, the configuration combinations corresponding to resources to be selected are obtained; a target configuration combination matched with the current system configuration is selected from the configuration combinations; according to the identification information of the resource to be selected, a corresponding resource in the resource set corresponding to the target configuration combination is selected. The invention further provides a device for selecting resources according to the system configurations. By adopting the method, due to the fact that the pre-generated correspondence set of the resources and the configuration combinations, every time one resource is selected, the configuration combination information of each host resource set of the resource can be rapidly obtained by inquiring the correspondence, the operation of searching for the resource set is greatly simplified, and therefore the resource selection speed can be increased and the system performance can be improved.
Owner:BANMA ZHIXING NETWORK HONGKONG CO LTD

Low-latency file system address space management method and system and medium

The invention discloses a low-latency file system address space management method and system and a medium. The method comprises the steps that a super block and a block group distribution table are generated from an address space of storage equipment, file system information and the distribution situation of a block group on a linear address space of the storage equipment are stored in the super block, and the block group distribution table is used for marking the distribution situation of data blocks in the corresponding block group; when the file is created, dynamically creating or selectinga corresponding block group according to the size of a specified data block, allocating the data block, writing file data into the allocated data block, and updating the information of a block groupallocation table and a super block. It can be guaranteed that which data blocks are occupied by one file on the storage device can be known only through one IO operation, the read-write delay of the file can be remarkably reduced, the addressing expenditure of file read-write can be remarkably reduced, the file read-write sequence can be remarkably improved, and therefore the read-write performance is improved.
Owner:SUN YAT SEN UNIV

Device and method used for executing Adam gradient descent training algorithm

The invention discloses a device and a method used for executing an Adam gradient descent training algorithm. The device includes a direct memory access unit, an instruction cache unit, a controller unit, a data cache unit and a data processing module. The method includes: firstly, reading a gradient vector and a to-be-updated value vector, and initializing first moment and second moment vectors and corresponding exponential decay rates at the same time; during each iteration, utilizing the gradient vector to update the first moment and second moment vectors, respectively calculating a first biased moment estimation vector and a second biased moment estimation vector, and utilizing the first biased moment estimation vector and the second biased moment estimation vector to update to-be-updated parameters, and continuing training until the to-be-updated parameter vector converges. By utilizing the device and the method, application of the Adam gradient descent algorithm can be realized, and the efficiency of data processing can be greatly improved.
Owner:CAMBRICON TECH CO LTD

Block chain data processing method and device and terminal device

The invention provides a block chain data processing method and device and a terminal device. The block chain data processing method comprises the steps of obtaining the block data to be written intoa block chain, performing column compression processing on the transaction record row data in the block data according to a transaction record column to obtain the block column compressed data, and writing the block column compressed data into the block chain. According to the scheme provided by the invention, by improving the structure of the block data written into the block chain, when the transaction record of the block is queried, the IO operation of the system can be greatly reduced, the resource waste is avoided, the high-concurrency query requirement is met, and the data is efficientlyprocessed.
Owner:CHINA MOBILE COMM LTD RES INST +1

Device and method used for executing RMSprop gradient descent algorithm

The invention discloses a device and a method used for executing an RMSprop gradient descent algorithm. The device includes a direct memory access unit, an instruction cache unit, a controller unit, a data cache unit and a data processing module. The method includes: firstly, reading a gradient vector and a to-be-updated value vector, and initializing a mean square vector at the same time; during each iteration, firstly, utilizing the gradient vector to update the mean square vector, then utilizing the mean square vector to calculate corresponding gradient descent quantity during updating, and updating the to-be-updated parameter vector, and repeating the process until the to-be-updated vector converges. In the whole process, the mean square vector is always stored in the data cache unit. By utilizing the device and the method, application of the RMSprop gradient descent algorithm can be realized, and the efficiency of data processing can be greatly improved.
Owner:CAMBRICON TECH CO LTD

Ordered data recording method and device, ordered data access method and device

The invention provides a recording method and device of ordered data and an access method and device of the ordered data. The recording method comprises the following detailed steps: creating a magnetic disk file for the ordered data; recording the ordered data in the magnetic disk file according to the block mode, and the quantity of blocks is two or more. The recording process comprises the step of recording the field interval of the corresponding ordered data and the recording number in the field interval in each block, wherein the ordered data in a database is stored in order according to the field. The invention has the advantages that data only needs to be read from the blocks and corresponding intervals covered by the records corresponding to the content of the specific pages, and the redundant data can be prevented from being read as much as possible, so that the IO (Input / Output) operation of a magnetic disk is reduced.
Owner:ALIBABA GRP HLDG LTD

Recording method and device of ordered data and access method and device of ordered data

The invention provides a recording method and device of ordered data and an access method and device of the ordered data. The recording method comprises the following detailed steps: creating a magnetic disk file for the ordered data; recording the ordered data in the magnetic disk file according to the block mode, and the quantity of blocks is two or more. The recording process comprises the step of recording the field interval of the corresponding ordered data and the recording number in the field interval in each block, wherein the ordered data in a database is stored in order according to the field. The invention has the advantages that data only needs to be read from the blocks and corresponding intervals covered by the records corresponding to the content of the specific pages, and the redundant data can be prevented from being read as much as possible, so that the IO (Input / Output) operation of a magnetic disk is reduced.
Owner:ALIBABA GRP HLDG LTD

A data block storage method, reading method, device and block chain

The present invention discloses a data block storage method and reading method, apparatuses thereof, and a block chain. The storage method comprises: before writing a new data block in the block chain, converting transaction record data in each row in a transaction record structure of the new data block to the processing of storing the data in a column structure; creating column structure storageidentification information in the new data block stored in a column structure; and storing the new data block with the column structure storage identification information into the block chain. Since the transaction data in each row is stored in a column structure, and the column structure storage identification information is created, when querying the transaction record data, instead of queryingthe entire transaction record data, only the transaction record data of the corresponding column needs to be queried according to the identification information; a large number of system IO operationsare reduced, large-capacity and high-concurrent query requirements can be supported, and efficient data processing can be facilitated; and in addition, by extending the columnar storage structure, changes in the future transaction record details can be handled.
Owner:CHINA MOBILE COMM LTD RES INST +1

A device and method for implementing rmsprop gradient descent algorithm

The present disclosure provides a device and method for executing the RMSprop gradient descent algorithm. The device includes a direct memory access unit, an instruction cache unit, a controller unit, a data cache unit, and a data processing module. The method includes: first reading the gradient vector and the value vector to be updated, and at the same time initializing a mean direction vector; at each iteration, first using the gradient vector to update the mean direction vector, and then using the mean direction vector to calculate the corresponding gradient descent when updating , update the parameter vector to be updated, and repeat this process until the vector to be updated converges. During the whole process, the homogeneous vector is always stored in the data cache unit. Utilizing the disclosure, the application of the RMSprop gradient descent algorithm can be realized, and the efficiency of data processing can be greatly improved.
Owner:CAMBRICON TECH CO LTD

Rapid association classification and hierarchical storage method for network security logs

The invention discloses a rapid association classification and hierarchical storage method for network security logs, which comprises the following steps: defining an XML (Extensible Markup Language) format log template according to different application metadata, the log template comprising alarm log information, association flow log information, application metadata log information, message log information and file attachment log information; associating the alarm logs, the associated flow logs, the application metadata logs, the message logs and the file attachment logs together through flow and transaction information in the logs, wherein each alarm log outputs a complete log; performing hierarchical storage mechanism output on logs, including values in alarm log information, associated flow log information, application metadata log information, message log information and file attachment log information; the hierarchical storage mechanism adopts two levels, the first level adopts a memory file system, and the second level adopts an SSD / HD storage system with RAID. According to the technical scheme, the performance problem of Json log format output is solved.
Owner:深圳市东晟数据有限公司

A data processing method, device and server

The present invention provides a data processing method, device, and server. The data processing method proposed by the present invention includes obtaining an object query request, the query request includes a subset of feature data of the object to be queried; obtaining an object description set, and the object The elements in the description set correspond to objects one by one, and the elements are a collection of feature data of an object; judge whether there is a target object in the object description set, and the feature can be hit in the element corresponding to the target object All feature data in the data subset; if present, output the target object. The present invention can change the original traversal query of a plurality of data tables into a single query of the object description set, thereby significantly reducing IO operations and improving query efficiency.
Owner:TENCENT TECH (SHENZHEN) CO LTD

Data storage management method, device and equipment

The invention provides a data storage management method, device and equipment, and the method comprises the steps: responding to a data writing instruction, writing a data block into zone, taking a unique identifier of the data block as a key, taking the position information of the data block as a value, generating a data block index in a key-value form, and writing the data block index into a storage region corresponding to index information in a memory; when the data volume of the to-be-stored area reaches a set threshold value, packaging the data of the storage area into a file and storing the file in the CMR area; in response to the data reading instruction, retrieving a first target key matched with a target identifier in the instruction in the data block index; the corresponding first target value is positioned according to the first target key, and the target data block is read according to the position information of the data block in the first target value, so that efficient storage and management of the data block and the metadata index information thereof by using the SMR disk are realized.
Owner:ZHEJIANG DAHUA TECH CO LTD

A block chain data processing method, device and terminal equipment

The present invention provides a block chain data processing method, device, and terminal equipment, wherein the block chain data processing method includes: acquiring block data to be written into the block chain, The row data of the transaction record is subjected to column compression processing according to the transaction record column to obtain block column compressed data, and the block column compressed data is written into the block chain. The scheme of the present invention, by improving the structure of the block data written into the block chain, can greatly reduce system IO operations when querying block transaction records, avoid waste of resources, meet high concurrency query requirements and perform efficient data processing. deal with.
Owner:CHINA MOBILE COMM LTD RES INST +1

A data processing method and storage device

The present invention discloses a data processing method and a storage device, in order to at least solve the problem that the current administrator has low efficiency in querying the log data of thesession. The method comprises that: a storage device receives a log storage request, wherein the log storage request carries log data of a session; an index value of a first-level directory of the logdata is determined, wherein the index value of the first-level directory is represented by E0; an index value of a second-level directory of the log data is determined, wherein the index value of thesecond-level directory is represented by B0; an index value of a third-level directory of the log data is determined, wherein the index value of the third-level directory is represented by (X0,Y0), and the (X0,Y0) represents a coordinate which takes the X0 as the abscissa, and the Y0 as the ordinate in a first coordinate system with the (E0, B0) as the origin, the end time of the session as the abscissa system, and the start time of the session as the ordinate system; and according to the E0, B0, and (X0, Y0), the log data is stored. The technical scheme of the present invention is applicableto the technical field of storage.
Owner:HUAWEI TECH CO LTD

A raid5 writing io optimization processing method

ActiveCN103049222BReduce write amplification effectOperate effectivelyInput/output to record carriersWrite amplificationRAID
Disclosed is an RAID5 (redundant array of independent disk 5) write IO optimization processing method. The method includes following steps: 1), storage system creation; 2), data reading and writing operation; and 3), failure data block recovery and has the advantage of capability of effectively lowering write amplification effect of RAID level disk arrays. For conventional magnetic disks, sequential write performance is much higher than random write performance. The address mapping method enables random write operation to be converted into sequential write operation, write performance of the magnetic disk is effectively brought into play, IO operation needed for computation checking is lowered, and response capability of an RAID system is evidently improved, and power consumption of the system is lowered. In addition, an address mapping module can flexibly choose arrangement positions and the number of checking blocks, so that the RAID5 write IO optimization processing method is not limited to be applicable to disk arrays of RAID5 level and can be popularized for realization of RAID4 and RAID6 levels.
Owner:NO 709 RES INST OF CHINA SHIPBUILDING IND CORP
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products