30results about How to "Reduce write volume" patented technology

The invention discloses a data storage method. Data records are taken as basic read-write units for the data storage; and each data record occupies storage space of a virtual small sector and comprises keywords, data and a virtual sector distribution table representing the parameter characteristics of the data records. The data records are written into the storage space, the data of the data record to be written is compared with the data of the written data record of which the keyword is the same and the data is the latest base data; if differential writing is needed, the difference data is assembled, which is served as the data of the data record to be written according to the latest base data; otherwise, the data record to be written is directly written. In the data storage method, the differential data storage is considered in the storage process, the data writing amount is reduced, the occupation of the storage space is reduced and the writing speed is improved.

The invention discloses a distributed flash memory transaction processing method. The method comprises the following steps: S1, carrying out two-phase commit on a transaction, wherein state log records of the two-phase commit protocol are removed and internalized as flash memory metadata operations in interfaces of the transaction, and the end of the distributed transaction is marked with the final commit of a coordinator; S2, during the process of the transaction, storing a data temporary object as a shadow version, wherein a shadow mapping table is used for storing the address of the shadow version or the state of a page in the transaction processing, metadata of the page is used for recording information of the transaction, a transaction metadata page is used for recording the state of the transaction and a transaction state table is used for recording the address of the transaction metadata page; and S3, when the coordinator or a participant has a fault, scanning a part of the memory, reading the flash memory metadata and restoring an FTL mapping table, the shadow mapping table and the transaction state table so as to determine the state of the transaction during the two-phase commit. The distributed flash memory transaction processing method disclosed by the invention has the following advantages that the logs are decreased and the transaction processing efficiency under the distributed flash memory environment is improved.

The invention discloses a storage caching method and a storage caching system perceived by a tile recording disk. The method comprises the following implementation steps of: presetting a data block cache and an identification cache; receiving an access request and segmenting the access request into read block requests, if the read block requests hit in the data block cache, returning and reading data from the solid-state disk, otherwise, reading data from the HA-SMR area; directly writing sequential write requests into HA-SMR region; and segmenting the non-sequential write requests into writeblock requests and detecting whether the write block requests hit, and writing the write block requests into the solid-state disk if the write block requests hit. If missed, a write-in position is decided according to the state of the target HA-SMR area, the SMR area is written in the solid-state disk if the HA-SMR area is in a sequential area state, otherwise, the HA-SMR area is written in the tile recording disk and caches the identifier. The host perception characteristic of the HA-SMR disk accurately identifies sequential write requests and harmless non-sequential write requests of the tile recording disk in the cache admission stage, and the requests are directly written into the tile recording disk, so that the write amount of the solid-state disk is reduced.

The invention provides a stream intelligence prediction differencing and compression algorithm, which comprises the following steps: a, determining a source data stream and a new data stream; b, determining the most suitable dictionary according to the new data stream; c, performing matching on the new data stream based on the most suitable dictionary; d, coding the new data stream according to a matching result; e, determining a differencing data stream according to a coding result. The invention also provides a stream intelligence prediction differencing and compression unit, which comprises a first determining device, a second determining device, a first processing unit, a first coding device and a third determining device, wherein the first determining device is used for determining the source data stream and the new data stream; the second determining device is used for determining the most suitable dictionary according to the new data stream; the first processing unit is used for performing the matching on the new data stream based on the most suitable dictionary; the first coding device is used for coding the new data stream according to the matching result; the third determining device is used for determining the differencing data stream according to the coding result. Different from a traditional dictionary compression algorithm adopting one dictionary, the stream intelligence prediction differencing and compression algorithm provided by the invention is capable of generating multiple dictionaries so as to generate the optimal effect. The stream intelligence prediction differencing and compression algorithm has network adaptivity.

The present invention discloses an erasure code storage-based file modification algorithm optimization method. The method comprises the steps of calculating data stored through an erasure code technology by using an algorithm, then splitting the data in a K:M manner, performing object processing on K data blocks and M verification blocks, and then storing in a unit of object; when the stored data is modified, mapping the modified data to the corresponding data block, so as to locate an object name corresponding to data on the block, that is, data modification refers to change of a specific object; when the data object is changed, recording the changed objects by the algorithm so as to lock unchanged data objects, and marking a correspondence; and writing new object data after the modified data is stored. The method has the advantages that the writing amount of the modified stored large file is small and modified data is restored efficiently.

The invention relates to a rapid persistence method based on CAD (computer-aided design), which comprises the following steps: (1) determining the total number N of nodes in a distributed system cluster, letting F = (N-1) / 2, setting the number of erasure codes as k, and setting the value of a selection parameter x; (2) judging whether the number of response nodes recently received by the Leader node in the cluster is greater than or equal to F, and if so, executing the step (3); (3) judging whether the number of response nodes recently received by the Leader node in the cluster is greater than or equal to F + k; if the number of response nodes recently received by the Leader node is larger than or equal to F + k, copying is carried out according to an F-CAD consistency protocol until all data are persistently written into F + 1 + x nodes, the log entry is submitted, and next round of copying is executed; and if the number of response nodes received by the Leader node recently is smaller than F + k, performing consistent copying on the system according to a CAD protocol until full data is persistently written into at least F + 1 + 0 nodes, submitting the log entry, and executing the next round of copying. The method can reduce the probability of single-point failure.

The invention discloses a disk controllable buffer writing method, a controller, a hybrid IO scheduling method and a scheduler. The invention belongs to the technical field of computer storage. The method comprises: predicting a write delay state of a next write request according to write delay characteristics of a disk, if a disk request queue is empty and a prediction result is high, setting thedisk state to be unavailable, writing filling data until a next prediction result is not high, resetting ADW to be 0, and resetting the state to be available; if the disk request queue is empty and the prediction result is yes, maintaining that the disk is available and writing filling data; and when the heterogeneous storage node receives the write IO request, if l(t) is greater than L and the write delay state of the next write request in the disk is low or medium, or l(t) is less than or equal to L and the write delay state of the next write request in the disk is low, redirecting the request to the disk. The overall performance of the disk can be improved, the service life of the high-speed hard disk in the heterogeneous storage node can be prolonged, and the disk utilization rate canbe increased.

The invention provides a log generation method based on an FTL table. The method comprises the following steps: recording a logical address and a physical address corresponding to an operation requestas log entries; generating log frames by using the log entries; generating FTL frames by using part of FTL blocks, wherein the FTL table includes multiple FTL blocks; and storing the generated FTL frames and the first log frame on an NVM. The invention also provides a method for reconstructing the FTL table by using logs. The method comprises the following steps: acquiring storage locations of the FTL frames and the log frames of the FTL blocks; reading multiple FTL frames and multiple log frames that belong to the FTL blocks; reconstructing the FTL blocks by using the multiple FTL frames; and updating the FTL blocks by using the log frames, wherein the FTL blocks are part of the FTL table.

The invention provides a message distribution method and device. The message distribution method includes: determining receivers of messages to be distributed; according to the receivers, triggering at least one attention user of the receivers to release the to-be-distributed messages, where the attention users include the receivers, users obtained by registration of tags that the receivers pay attention to, and broadcast users that all the users pay attention to within a preset range; sending the to-be-distributed messages released by the attention users of the receivers to the receivers. By the method, data read-in amount can be reduced, message distribution efficiency is improved, and resources are saved.

The invention relates to a data processing method and device, electronic equipment and a storage medium, and relates to the technical field of networks. According to the method and the device, the specified operation is executed on the to-be-stored data according to the to-be-stored data in the memory of the electronic equipment, so that the operation of writing the to-be-stored data into the specified hierarchy of the disk in the electronic equipment is completed. Wherein the specified operation comprises the following steps: on the basis of the to-be-stored data, generating at least two sorting character string table files corresponding to at least two currently defined sorting character string table groups, so as to store the to-be-stored data in the form of the at least two sorting character string table files; wherein the key name range corresponding to any sorting character string table file does not exceed the grouping range of sorting character string table groups corresponding to the sorting character string table file; and dividing the sorted character string table file into the corresponding sorted character string table groups. Therefore, the bandwidth can be saved to a certain extent, the write-in amount to the disk is reduced, and the service life of the disk is prolonged.

The invention discloses a storage cache method and system of tile record disk perception. The implementation steps include setting up data block cache and identification cache in advance; receiving an access request and dividing it into read block requests; , returns the read data from the SSD, otherwise reads the data from the HA‑SMR area; directly writes the sequential write request to the HA‑SMR area; splits the non-sequential write request into a write block request and checks whether it hits, A hit is written to the SSD. If there is a miss, the write position is determined according to the state of the target HA-SMR area. If it is in the sequential area state, it is written to the solid state disk, otherwise it is written to the tile recording disk and its identifier is cached. The present invention utilizes the host perception characteristic of the HA-SMR disk to accurately identify sequential write requests and harmless non-sequential write requests to the tile recording disk in the cache access stage, and directly writes these requests to the tile recording disk, thereby reducing the impact on the solid state disk. write volume.

The purpose of the present invention is to avoid prolonging processing time involved in a power supply being switched on and off and to extend the useful lifespan of non-volatile memory. Counter information and setting information are stored by an EEPROM (115). The counter information and setting information are stored by a first RAM (116) and a second RAM (117). Storage regions for the counter information and setting information in the first RAM (116) are managed as updatable, and storage regions for the counter information and the setting information of the second RAM (117) are managed as non-updatable, by a memory management unit (118). When a change occurs in the content of the counter information and setting information, the counter information and setting information of the first RAM(116) are overwritten by a system control unit (122) according to the content of the change. When the power supply is switched off, the counter information and setting information of the first RAM (116) and the counter information and setting information of the second RAM (117) are read and compared, and only the information differing therebetween is written to the EEPROM (115).

The invention discloses a distributed flash memory transaction processing method, which includes the following steps: S1: perform two-stage commit of the transaction, remove the state log record of the two-stage commit protocol, and internalize it into the flash metadata operation in the transaction interface, so that the coordinator The final submission is used as the end mark of the distributed transaction; S2: During the transaction process, the data temporary object is stored as a shadow version, and the shadow mapping table is used to store the address of the shadow version or the state of the page in the transaction processing, and the page metadata records the transaction information The transaction metadata page records the transaction status and the transaction status table records the transaction metadata page address; S3: When the coordinator or participant fails, scan part of the storage, read the flash metadata, and restore the FTL mapping table, shadow mapping table, transaction The state table, which determines the state of the transaction in two-phase commit. The invention has the advantages of reducing logs and improving transaction processing efficiency in a distributed flash memory environment.

The invention discloses a disk controllable buffer write method, a controller, a hybrid IO scheduling method and a scheduler, which belong to the technical field of computer storage and include: predicting the write delay state of the next write request according to the write delay characteristics of the disk, if the disk If the request queue is empty and the prediction result is high, the disk status is set to be unavailable, and the filling data is written until the next prediction result is not high. At this time, the ADW is reset to 0, and the status is reset to available; if If the disk request queue is empty and the prediction result is medium, keep the disk available and write the filling data; when the heterogeneous storage node receives a write IO request, if l(t)>L, and the next write request in the disk If the write latency status is low or medium, or l(t)≤L, and the write latency status of the next write request in the disk is low, the request is redirected to the disk. The invention can improve the overall performance of the disk, and can improve the service life of the high-speed hard disk and the utilization rate of the disk in heterogeneous storage nodes.

The invention discloses a data storage method. Data records are taken as basic read-write units for the data storage; and each data record occupies storage space of a virtual small sector and comprises keywords, data and a virtual sector distribution table representing the parameter characteristics of the data records. The data records are written into the storage space, the data of the data record to be written is compared with the data of the written data record of which the keyword is the same and the data is the latest base data; if differential writing is needed, the difference data is assembled, which is served as the data of the data record to be written according to the latest base data; otherwise, the data record to be written is directly written. In the data storage method, the differential data storage is considered in the storage process, the data writing amount is reduced, the occupation of the storage space is reduced and the writing speed is improved.

The invention discloses a digital watermark generation method and system for remote sensing images and a control method therefor. The disclosed generation method comprises the steps of obtaining position relation information between at least two adjacent remote sensing images; setting a digital watermark type; setting digital watermark information; generating a first roulette digital watermark of a left-right digital watermark type on the two remote sensing images in the left-right position relation, and / or, generating a second roulette digital watermark of an up-down digital watermark type on the two remote sensing images in the up-down position relation, and / or, generating a third roulette digital watermark of a four-corner digital watermark type on the four remote sensing images of the four-corner position relationship, wherein the first roulette digital watermark, the second roulette digital watermark and the third roulette digital watermark all comprise digital watermark information. According to the scheme, the problems that the information writing amount is large and the position relation between the remote sensing images cannot be distinguished when digital watermarks are applied to a large batch of remote sensing images with position information can be solved.

The present invention relates to the field of software development, in particular to a software low-code development system and a development method combined with the middle platform architecture theory; the software low-code development system includes a research and development material warehouse as a place for depositing middle-end capabilities, and as a consumption middle-end capability The low-code development platform of the site is used as the page design unit of the mid-stage carrier of the front-end capability, the research and development service unit and the mid-stage control unit of the mid-stage capability technology base; The combination of tools not only effectively improves the R&D efficiency of the enterprise, but also provides a clear division of labor for low-code developers; by packaging the business into function packages and superimposing them to realize system development, it effectively solves the contradiction between usability and practicality , and the amount of repeated code writing can be greatly reduced without increasing development costs. At the same time, each functional module in the system can be customized, making the development system more applicable.

The invention provides an intelligent compression storage method for neural network check point data. The neural network check point data comprises weight floating-point number data and optimizer floating-point number data. The method specifically comprises the following steps: compressing and storing weight floating-point number data after each round of training of a neural network by using an incremental compression method; and / or utilizing index value mapping to replace the first n bits of part or all optimizer floating-point number data after each round of training of the neural network and then storing, wherein the number of bits of the index value is smaller than n. The intelligent compression method and system are designed by utilizing the characteristics of deep learning check point data and combining a model training process, and different types of data of the check points are efficiently compressed, so that the storage capacity of a storage system is effectively improved, and the service life of the storage system is effectively prolonged.

The invention discloses a system framework for realizing the selective upgrade of SSD cache data in a storage system. The system framework adopts a data selective upgrade algorithm to store the first hit data block in a queue named a candidate upgrade queue, and uses tracking every When the hit count of hit blocks in a candidate upgrade queue reaches the upgrade limit, the hit data blocks will be cached from the HDD hard disk to the SSD cache, so as to achieve the goal of delaying cache data, reducing write operations, and prolonging the life of SSD; Adopting the system architecture of the present invention can ensure the selective upgrade of SSD cache data, efficiently cache data, and satisfy users' requirements for high-performance storage systems.

The present invention relates to the field of data storage. Disclosed are a data writing method and device for solving the problem of the instable reduction of write delay, the method comprising: sampling data to be written according to a preset sampling algorithm to obtain a target label, the target label being used to represent numerical value distribution of data bits in the data to be written; searching a preset target group label set for a target group label having the highest similarity to the target label, the preset group label set consisting of at least one group label, and each group label corresponding to at least two blank data pages having similar data characteristics; selecting a target blank data page from the at least one blank data page corresponding to the target group label; and writing the data bits in the data to be written that are different from those in the target blank data page into the corresponding positions on the target blank data page. The present invention is mainly used in data read and write operations.

The invention discloses a data pair copying method, device and system and a medium, and the method comprises the steps: reading data from first storage equipment, and obtaining first target data; judging whether second target data exists in the first target data or not, wherein the second target data is data of a first preset size and in a preset format, and the preset format is a data format of return data corresponding to a storage space in which data is not written in the first storage device; if the second target data exists in the first target data, filtering the second target data from the first target data to obtain valid data; and writing the valid data into a second storage device. Thus, useless data can be filtered, only valid data is written, the data writing amount is reduced,the copying time is shortened, meanwhile, the garbage collection triggering frequency is reduced, and therefore the user experience is improved.

The present invention is a method based on CAD fast persistence, and the method includes the following steps: (1) determine the total number of nodes N in the distributed system cluster, make F=(N-1) / 2, and set the erasure code The number is k, and the value of the selection parameter x is set at the same time; (2) judge whether the number of response nodes recently received by the Leader node in the cluster is greater than or equal to F, and if it is greater than or equal to F, perform step (3); (3) determine whether the Leader node in the cluster Whether the number of response nodes recently received by the node is greater than or equal to F+k; if the number of response nodes recently received by the Leader node is greater than or equal to F+k, it will be replicated according to the F-CAD consistency protocol until F+1+x nodes Write the full amount of data persistently, submit the log entry, and execute the next round of replication; if the number of response nodes recently received by the Leader node is less than F+k, the system will be replicated consistently according to the CAD protocol until the full amount of data is persistently written Enter at least F+1+0 nodes, submit the log entry, and execute the next round of replication. This method can reduce the probability of single point of failure.

The invention discloses a method and device for monitoring eMMC, wherein the method includes the following steps: monitoring the file pre-written into the eMMC hotspot address; writing the file to a non-eMMC storage location; when certain conditions are met; Files in non-eMMC storage locations are written to eMMC. This technical solution effectively reduces the amount of eMMC writing by writing files to non-eMMC storage locations, especially the frequent writing of some hotspot addresses, which improves the service life of eMMC particles. At the same time, when writing to the memory, it can Make the system run more smoothly.

The present application discloses a method and device for optimizing garbage collection based on storage array data access. The method includes: classifying and discriminating a received data access request according to a request source and a preset data hot and cold classification algorithm, and obtaining corresponding classification information ; Call the cache queue algorithm to cache and judge the data access request according to the classification information, so as to be added to the target cache queue of the corresponding classification; if the data to be accessed in the target cache queue reaches the preset standard threshold, the data to be accessed will be accessed according to the address mapping table. Data is written to the corresponding target node in stripe units; if there is no writable storage space in the storage array or the blank nodes in the work queue are less than the preset number, a garbage collection that delays the deletion of invalid data is used. mechanism for data recovery. By implementing the present application, the problem of write amplification of the storage array and the serious IO problem caused in the prior art can be solved.