A data optimization storage method and device, computer equipment and storage medium

Abstract

The application provides a data optimization storage method and device, computer equipment and a storage medium, which can be applied to the field of cloud computing or the field of block chains, and is used for solving the problem of a large number of unnecessary storage resource occupation in the process of storing data. The method comprises the following steps: traversing to-be-stored data, determining a data proportion of repeated data in the to-be-stored data; when the data proportion meets a preset optimization condition, taking the to-be-stored data as to-be-optimized storage data; determining a non-repeated data sequence of the to-be-optimized storage data based on each non-repeated data contained in the to-be-optimized storage data; determining a data identifier sequence of the to-be-optimized storage data based on a data identifier of each non-repeated data and a data identifier of each repeated data contained in the to-be-optimized storage data; and storing the to-be-optimized storage data based on the data identifier sequence and the non-repeated data sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, and particularly relates to a data optimization storage method and device, computer equipment and a storage medium. BACKGROUND

[0002] With the continuous development of technology, more and more data needs to be stored by the equipment. However, in some cases, the repeated data accounts for a large proportion in the data to be stored, so that the storage of the repeated data occupies a large amount of storage resources.

[0003] For example, for the usage rate of a central processing unit (CPU) at each time point, if the CPU is mostly idle, a large amount of 0 values are included in the CPU usage rate information to be stored.

[0004] It can be seen that a large amount of unnecessary storage resources are occupied in the process of storing data. SUMMARY

[0005] Embodiments of the present application provide a data optimization storage method and device, computer equipment and a storage medium, which are used to solve the problem of a large amount of unnecessary storage resource occupation in the process of storing data.

[0006] In a first aspect, a data optimization storage method is provided, comprising:

[0007] traversing to-be-stored data to determine a data proportion of repeated data in the to-be-stored data;

[0008] when the data proportion meets a preset optimization condition, taking the to-be-stored data as to-be-optimized storage data;

[0009] determining a non-repeated data sequence of the to-be-optimized storage data based on each non-repeated data included in the to-be-optimized storage data;

[0010] determining a data identifier sequence of the to-be-optimized storage data based on a data identifier of each non-repeated data and a data identifier of each repeated data included in the to-be-optimized storage data;

[0011] storing the to-be-optimized storage data based on the data identifier sequence and the non-repeated data sequence.

[0012] In a second aspect, a data optimization storage device is provided, comprising:

[0013] a transceiving module configured to traverse to-be-stored data to determine a data proportion of repeated data in the to-be-stored data;

[0014] The transceiving module is further configured to: when the data proportion satisfies a preset optimization condition, taking the to-be-stored data as to-be-optimized storage data.

[0015] The processing module is configured to: determine a non-repeated data sequence of the to-be-optimized storage data based on each non-repeated data contained in the to-be-optimized storage data.

[0016] The processing module is further configured to: determine a data identifier sequence of the to-be-optimized storage data based on a respective data identifier of each non-repeated data and a respective data identifier of each repeated data contained in the to-be-optimized storage data.

[0017] The processing module is further configured to: store the to-be-optimized storage data based on the data identifier sequence and the non-repeated data sequence.

[0018] Optionally, the transceiving module is specifically configured to:

[0019] obtain to-be-stored data;

[0020] traverse the to-be-stored data, and respectively determine whether each data included in the to-be-stored data is a specified data;

[0021] count a repetition number of repeated data that is the specified data among the each data;

[0022] determine a data proportion of the repeated data in the to-be-stored data based on the repetition number.

[0023] Optionally, the processing module is specifically configured to:

[0024] determine the data proportion of the repeated data in the to-be-stored data based on a ratio between the repetition number and a total data amount, wherein the total data amount is a quantity of data contained in the to-be-stored data; or

[0025] determine the data proportion of the repeated data in the to-be-stored data based on a ratio between the repetition number and a non-repetition number, wherein the non-repetition number is a quantity of non-repeated data contained in the to-be-stored data.

[0026] Optionally, the processing module is specifically configured to:

[0027] determine a minimum non-repeated data among the each non-repeated data;

[0028] determine a data difference value between each non-repeated data and the minimum non-repeated data, respectively;

[0029] adopt each data difference value obtained to represent a corresponding non-repeated data, respectively, and arrange the each data difference value in sequence to obtain the non-repeated data sequence.

[0030] Optionally, the processing module is specifically used for:

[0031] Based on the data identifiers of each of the non-duplicate data, determine the sequence of non-duplicate data identifiers of the data to be optimized for storage;

[0032] If the data type of the data to be optimized is a data-intensive type, then the non-repeating data identifier sequence is used as the data identifier sequence, wherein the data-intensive type indicates that the data to be optimized does not contain null values.

[0033] Optionally, the processing module is further configured to:

[0034] After determining the sequence of non-repeating data identifiers of the storage data to be optimized based on the data identifiers of each non-repeating data, if the data type of the storage data to be optimized is a data sparse type, then the sequence of duplicate data identifiers of the storage data to be optimized is determined based on the data identifiers of each duplicate data contained in the storage data to be optimized, wherein the data sparse type indicates that the storage data to be optimized contains null data.

[0035] The non-repeating data identifier sequence and the repeating data identifier sequence are used as the data identifier sequence.

[0036] Optionally, the processing module is specifically used for:

[0037] Iterate through the data to be optimized and determine whether each piece of data included in the data to be stored is the specified data.

[0038] Based on the non-duplicate data that is not the specified data in each of the data, obtain the data identifier of the non-duplicate data;

[0039] According to the order of the non-repeating data in the non-repeating data sequence, the data identifiers of each non-repeating data are arranged sequentially to obtain the non-repeating data identifier sequence.

[0040] Optionally, the processing module is specifically used for:

[0041] Iterate through the data to be optimized and determine whether each piece of data included in the data to be stored is the specified data.

[0042] Based on the duplicate data that is the specified data in each of the data, obtain the data identifier of the duplicate data;

[0043] The data identifiers of each of the repeated data are arranged sequentially to obtain the repeated data identifier sequence.

[0044] Optionally, the processing module is further configured to:

[0045] After storing the to-be-optimized stored data based on the data identifier sequence and the non-repeated data sequence, receive a data reading instruction, wherein the data reading instruction comprises a specified data identifier;

[0046] In the preset plurality of data identifier intervals, determine a specified data identifier interval containing the specified data identifier;

[0047] Based on a mapping relationship between the pre-stored data identifier interval and the optimized stored data identifier sequence, load a target data identifier sequence corresponding to the specified data identifier interval;

[0048] Based on the target data identifier sequence, determine whether target data corresponding to the specified data identifier is non-repeated data;

[0049] If the target data corresponding to the specified data identifier is non-repeated data, then based on the target data identifier sequence, obtain a target non-repeated data sequence;

[0050] Based on the specified data identifier, read the target data in the target non-repeated data sequence.

[0051] Optionally, the processing module is further configured to:

[0052] Before determining whether the target data corresponding to the specified data identifier is non-repeated data based on the target data identifier sequence, determine a data type of original data corresponding to the target data identifier sequence, wherein the original data is to-be-optimized stored data corresponding to the target data identifier sequence before the optimization storage, and the data type of the original data is used to represent whether the original data contains null data;

[0053] The processing module is specifically configured to:

[0054] If the data type of the original data is a data sparse type, then based on a target non-repeated data identifier sequence in the target data identifier sequence, determine whether the target data corresponding to the specified data identifier is non-repeated data.

[0055] Optionally, the processing module is further configured to:

[0056] If the data type of the original data is the data sparse type, after determining whether the target data corresponding to the specified data identifier is non-redundant data based on the target non-redundant data identifier sequence in the target data identifier sequence, if the target data corresponding to the specified data identifier is not non-redundant data, then determining whether the target data corresponding to the specified data identifier is redundant data based on the target redundant data identifier sequence in the target data identifier sequence;

[0057] If the target data corresponding to the specified data identifier is redundant data, the repeated specified data is taken as the target data.

[0058] In a third aspect, a computer device is provided, comprising:

[0059] a memory configured to store program instructions;

[0060] a processor configured to invoke the program instructions stored in the memory and execute the method according to the first aspect according to the obtained program instructions.

[0061] In a fourth aspect, a computer readable storage medium is provided, the storage medium stores computer executable instructions, and the computer executable instructions are used to make a computer execute the method according to the first aspect.

[0062] In the embodiments of the present application, the non-redundant data sequence is determined based on each non-redundant data in the to-be-optimized storage data, and the data identifier sequence is determined based on the data identifier of each non-redundant data and the data identifier of each redundant data in the to-be-optimized storage data. The to-be-optimized storage data is stored based on the non-redundant data sequence and the data identifier sequence. Compared with directly storing the to-be-optimized storage data, the unnecessary occupation of storage resources for redundant data can be reduced, the consumption of storage resources is reduced, and the rationality of the distribution of storage resources is improved. In the embodiments of the present application, only the to-be-stored data whose data proportion of redundant data meets the preset optimization condition is stored by optimization, so that the to-be-stored data whose data proportion of redundant data is high can be stored by optimization, and the to-be-stored data whose data proportion of redundant data is low is not stored by optimization. The consumption of processing resources in the process of storing the to-be-stored data whose data proportion of redundant data is low by optimization is reduced, the occupation of processing resources and storage resources is effectively balanced, and the rationality of the distribution of processing resources and storage resources is improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 An application scenario of the data optimization storage method provided in the embodiments of the present application;

[0064] Figure 2 A flowchart of the data optimization storage method provided in the embodiments of the present application;

[0065] Figure 3 A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application Figure 1 ;

[0066] Figure 4 A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application Figure 2 ;

[0067] Figure 5a A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application Figure 3 ;

[0068] Figure 5b A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application Figure 4 ;

[0069] Figure 6a A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application

[0070] Figure 6b A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application

[0071] Figure 7 A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application Figure 1 ;

[0072] Figure 8 A schematic diagram of one principle of the data optimization storage method provided by the embodiment of the present application Figure 2 .DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application.

[0074] Some terms in the embodiments of the present application will be explained below to facilitate understanding by those skilled in the art.

[0075] (1) 0-value data and null-value data:

[0076] 0-value data means data with a value of "0", and null-value data means data with a value that does not exist or is empty. Data with a value that does not exist or is empty does not mean that the value of the data is "0", and data with a value of "0" does not mean that the value of the data does not exist or is empty.

[0077] Embodiments of the present application relate to cloud technology and blockchain. It is designed based on cloud computing and cloud storage in cloud technology.

[0078] Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, network, etc. in a wide area network or a local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool, and be used on demand, flexibly and conveniently. Cloud computing technology will become an important support. The background service of a technical network system requires a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, every item may have its own identification mark and needs to be transmitted to the background system for logical processing. Different levels of data will be processed separately, and various industry data will need strong system support, which can only be realized through cloud computing.

[0079] Cloud computing is a computing mode that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services according to needs. The network that provides resources is called "cloud". The resources in the "cloud" can be infinitely expanded in the eyes of the user, and can be obtained at any time, used on demand, expanded at any time, and paid according to use.

[0080] As a basic capability provider of cloud computing, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an Infrastructure as a Service (IaaS) platform) will be established, and a plurality of types of virtual resources will be deployed in the resource pool for external customers to select and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines containing operating systems), storage devices, network devices.

[0081] According to logical function division, a Platform as a Service (PaaS) layer can be deployed on the IaaS layer, and a Software as a Service (SaaS) layer can be deployed above the PaaS layer, or the SaaS can be directly deployed on the IaaS. PaaS is a platform for software running, such as databases, web containers, etc. SaaS is various business software, such as web portal websites, SMS mass senders, etc. Generally, SaaS and PaaS are upper layers relative to IaaS.

[0082] Cloud storage is a new concept extended and developed on the basis of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that collects a large number of various types of storage devices (storage devices are also referred to as storage nodes) in a network through cluster application, grid technology, and distributed storage file system functions, and cooperatively provides data storage and business access functions.

[0083] At present, the storage method of the storage system is to create a logical volume, and assign a physical storage space to each logical volume when the logical volume is created. The physical storage space can be a disk of a certain storage device or a disk group of several storage devices. The client stores data on a certain logical volume, that is, stores data on a file system. The file system divides the data into many parts, each part being an object. The object contains not only data but also additional information such as data ID entity (ID). The file system writes each object to the physical storage space of the logical volume, and records the storage location information of each object. Thus, when the client requests to access data, the file system can enable the client to access the data according to the storage location information of each object.

[0084] The process of assigning a physical storage space to a logical volume by the storage system is as follows: according to the capacity estimation of the object stored in the logical volume (the estimation often has a large margin relative to the actual capacity of the object to be stored) and the group of RAID (Redundant Array of Independent Disk), the physical storage space is divided into sections in advance. A logical volume can be understood as a section, so that the logical volume is assigned a physical storage space.

[0085] Blockchain is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm. Blockchain is essentially a decentralized database, which is a series of data blocks associated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity (anti-fake) of the information and generate the next block. Blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer.

[0086] The blockchain underlying platform can include user management, basic service, smart contract, and operation monitoring processing modules. Among them, the user management module is responsible for the identity information management of all blockchain participants, including maintaining public and private key generation (account management), key management, and user real identity and blockchain address correspondence maintenance (permission management), etc., and under authorization, supervises and audits the transaction of certain real identities, provides risk control rule configuration (risk audit); the basic service module is deployed on all blockchain node devices to verify the validity of business requests, and record to the storage after consensus for valid requests, for a new business request, the basic service first interface adaptation analysis and authentication processing (interface adaptation), then encrypt the business information through the consensus algorithm (consensus management), after encryption, the complete and consistent transmission to the shared ledger (network communication), and record storage; the smart contract module is responsible for contract registration and issuance, contract triggering and contract execution, developers can define contract logic through a certain programming language, publish to the blockchain (contract registration), according to the logic of the contract terms, call the key or other event triggers to execute, complete the contract logic, and also provide contract upgrade and cancellation functions; the operation monitoring module is mainly responsible for the deployment, configuration modification, contract setting, cloud adaptation in the product release process, and the real-time state visualization output in the product running, such as: alarm, monitoring network situation, monitoring node device health status, etc.

[0087] The platform product service layer provides basic capabilities and implementation framework of typical applications, and developers can stack business characteristics based on these basic capabilities to complete the blockchain implementation of business logic. The application service layer provides application services based on the blockchain scheme for business participants to use.

[0088] The application field of the data optimization storage method provided by the embodiments of the present application is briefly introduced below.

[0089] With the continuous development of science and technology, more and more data needs to be stored by devices. However, in some cases, the repeated data occupies a large proportion in the data to be stored, so that the storage of repeated data occupies a large amount of storage resources.

[0090] For example, for the CPU usage rate at each time point or time period, if the CPU is mostly idle, then the CPU usage rate information to be stored includes a large number of 0 values. Please refer to Table 1, taking 10000 CPU usage rate data as an example, all the 10000 CPU usage rate data are not empty.

[0091] Table 1

[0092]

[0093]

[0094] Please refer to Table 2, taking 10000 CPU usage rate data as an example, of which 6000 CPU usage rate data is not empty, and the remaining 4000 CPU usage rate data is null data.

[0095] Table 2

[0096]

[0097] There are a lot of repeated data "0" in the data shown in Table 1 and Table 2, so when saving the data, the repeated data "0" will occupy a lot of storage resources. It can be seen that there is a lot of unnecessary storage resource occupation in the process of storing data.

[0098] In order to solve the problem of a lot of unnecessary storage resource occupation in the process of storing data, the application proposes a data optimization storage method. The method determines the data proportion of repeated data in the to-be-stored data by traversing the to-be-stored data. When the data proportion meets the preset optimization condition, the to-be-stored data is used as the to-be-optimized storage data. After obtaining the to-be-optimized storage data, based on each non-repeated data contained in the to-be-optimized storage data, the non-repeated data sequence of the to-be-optimized storage data is determined. Based on the data identifier of each non-repeated data and the data identifier of each repeated data contained in the to-be-optimized storage data, the data identifier sequence of the to-be-optimized storage data is determined. Based on the data identifier sequence and the non-repeated data sequence, the to-be-optimized storage data is stored.

[0099] In the embodiment of the application, the non-repeated data sequence is determined based on each non-repeated data in the to-be-optimized storage data, and the data identifier sequence is determined based on the data identifier of each non-repeated data and the data identifier of each repeated data in the to-be-optimized storage data. Compared with directly storing the to-be-optimized storage data, the unnecessary occupation of storage resources for repeated data can be reduced, the consumption of storage resources is reduced, and the rationality of the allocation of storage resources is improved.

[0100] In addition, in the embodiment of the application, only the to-be-stored data whose data proportion of repeated data meets the preset optimization condition is optimized and stored, so that the to-be-stored data with a high data proportion of repeated data can be optimized and stored, and the to-be-stored data with a low data proportion of repeated data is not optimized and stored, thereby reducing the consumption of processing resources in the process of optimizing and storing the to-be-stored data with a low data proportion of repeated data, effectively balancing the occupation of processing resources and storage resources, and improving the rationality of the allocation of processing resources and storage resources.

[0101] The application scenario of the data optimization storage method provided in the present application is described below.

[0102] Referring to Figure 1 , an application scenario of the data optimization storage method provided in the present application is described. The application scenario includes a client 101, a server 102, and a storage 103. The client 101 and the server 102 can communicate with each other, and the storage 103 and the server 102 can communicate with each other. The communication mode can be wired communication technology, such as communication through a network cable or a serial cable; or wireless communication technology, such as Bluetooth or wireless fidelity (WIFI) technology, without limitation.

[0103] The client 101 generally refers to a device that can provide the server 102 with data records to be optimized for storage, such as a terminal device, a third-party application accessible by the terminal device, or a webpage accessible by the terminal device. The terminal device can be, for example, a mobile phone, a tablet computer, or a personal computer. The server 102 generally refers to a device that can save data to be optimized for storage, such as a terminal device or a server. The server can be, for example, a cloud server or a local server. The storage 103 generally refers to a device that can be used to save data records to be optimized for storage, such as a database. The database can be, for example, a cloud database or a local database of the server 102. The client 101, the server 102, and the storage 103 can all use cloud computing to reduce the occupation of local computing resources; similarly, they can also use cloud storage to reduce the occupation of local storage resources.

[0104] As an embodiment, the client 101 and the server 102 can be the same device, the client 101 and the storage 103 can be the same device, the server 102 and the storage 103 can be the same device, the client 101, the server 102, and the storage 103 can be the same device, and the like, without limitation. In the present application, the client 101, the server 102, and the storage 103 are taken as different devices for example.

[0105] Based on Figure 1 , the data optimization storage method provided in the present application is described in detail, taking the target client as the client 101, the server as the server 102, and the storage cluster as the storage 103 for example.

[0106] Referring to Figure 2 , a flowchart of the data optimization storage method provided in the present application is described.

[0107] S201, traversing the data to be stored, determining the data proportion of the duplicate data in the data to be stored.

[0108] The target client can send the to-be-stored data to the server, which can be data generated by the target client, data received by the target client from other devices, or the like, without limitation. The target client can send the to-be-stored data to the server at various times, two of which are described below without limitation. The target client can send the to-be-stored data to the server at a preset time interval, for example, the target client can generate log data between a start time and an end time, and send the log data to the server at the end time. The target client can send the to-be-stored data to the server when the data volume reaches a preset data volume, for example, the target client can send the log data to the server when the data volume of the obtained log data reaches a threshold.

[0109] After receiving the to-be-stored data sent by the target client, the server determines whether the to-be-stored data is to-be-optimized storage data that needs to be optimized for storage by traversing the to-be-stored data, or determines whether the to-be-stored data includes to-be-optimized storage data that needs to be optimized for storage. Taking the case where the server determines whether the to-be-stored data is to-be-optimized storage data that needs to be optimized for storage, the server can determine whether the to-be-stored data is to-be-optimized storage data in various ways, two of which are described below.

[0110] Method one:

[0111] Based on the number of repetitions of the repeated data in the to-be-stored data, to-be-optimized storage data is determined.

[0112] After obtaining the to-be-stored data, the server can traverse the to-be-stored data and count the number of repetitions of the repeated data included in the to-be-stored data. Taking one of the data included in the to-be-stored data as an example during the traversal process, the processing of other data is similar to that of the one data, which is not described again. The server determines whether the one data is a specified data, which can be a data that can have repetitions that are preset, or a data that has a number of occurrences greater than a preset number of occurrences determined by the server during the traversal of the to-be-stored data, without limitation. For example, the specified data is "0". If the server determines that the one data is the specified data, the server determines that the one data is repeated data and updates the number of repetitions of the repeated data included in the to-be-stored data. For example, the number of repetitions of the repeated data is initially 0, and after it is determined that the one data is repeated data, the number of repetitions is updated to 1.

[0113] After obtaining the number of repetitions of the repeated data, the server can determine the data proportion of the repeated data in the to-be-stored data. There are various methods for determining the data proportion based on the number of repetitions, for example, the server can determine the total amount of data after traversing the to-be-stored data, and obtain the total amount of data. The server determines the data proportion based on the ratio between the number of repetitions and the total amount of data; for another example, the server can determine the number of non-repetitive data after traversing the to-be-stored data. The server determines the data proportion based on the ratio between the number of repetitions and the number of non-repetitive data, and the like, which is not limited.

[0114] Method two:

[0115] Based on the marking information of the to-be-stored data, determine the to-be-optimized storage data.

[0116] The target client can mark the to-be-stored data before sending the to-be-stored data to the server, so that the target client can determine whether to optimize the storage of the to-be-stored data when obtaining the to-be-stored data. The target client can mark the data proportion of the repeated data in the to-be-stored data, or mark the to-be-stored data whose data proportion of the repeated data meets the preset optimization condition, or directly mark the to-be-stored data that needs to be optimized and stored, so that the server can determine the marking information of the to-be-stored data when receiving the to-be-stored data sent by the target client. If there is a mark in the marking information of the to-be-stored data, the server can determine whether the data proportion of the repeated data meets the preset optimization condition based on the mark, or determine the to-be-optimized storage data based on the mark, and optimize the storage of the to-be-optimized storage data, and the like, which is not limited. If there is no mark in the marking information of the to-be-stored data, the server directly stores the to-be-stored data.

[0117] S202, when the data proportion meets the preset optimization condition, the to-be-stored data is used as to-be-optimized storage data.

[0118] The server can determine whether the data proportion meets a preset optimization condition after obtaining the data proportion of the repeated data. The preset optimization condition can be various, for example, the preset optimization condition can be a proportion threshold between the repeated quantity and the total quantity of data. After obtaining the ratio between the repeated quantity and the total quantity of data, the server determines whether the ratio reaches the proportion threshold. If the proportion reaches the proportion threshold, it is determined that the repeated quantity meets the preset optimization condition. If the proportion does not reach the proportion threshold, it is determined that the repeated quantity does not meet the preset optimization condition. For example, when the preset optimization condition is the proportion threshold between the repeated quantity and the total quantity of data, the preset optimization condition can be "index.skip_zero_threshold". When the value of "index.skip_zero_threshold" is 0.25, if the server determines that the proportion between the repeated quantity and the total quantity of data is greater than 0.25, it is determined that the ratio reaches the proportion threshold, that is, the preset optimization condition is met.

[0119] For another example, the preset optimization condition can be a proportion threshold between the repeated quantity and the non-repeated quantity. After obtaining the ratio between the repeated quantity and the non-repeated quantity, the server determines whether the ratio reaches the proportion threshold. If the ratio reaches the proportion threshold, it is determined that the repeated quantity meets the preset optimization condition. If the ratio does not reach the proportion threshold, it is determined that the repeated quantity does not meet the preset optimization condition.

[0120] If it is determined that the data proportion meets the preset optimization condition, it can be determined that the to-be-stored data needs to be stored by optimization, and the to-be-stored data is determined as to-be-optimized storage data. If it is determined that the data proportion does not meet the preset optimization condition, it can be determined that the to-be-stored data does not need to be stored by optimization, and can be directly stored.

[0121] As an embodiment, the target client can directly send the to-be-optimized storage data to the server, and the server receives the to-be-optimized storage data sent by the target client, thereby obtaining the to-be-optimized storage data that needs to be stored by optimization.

[0122] S203, based on each non-repeated data contained in the to-be-optimized storage data, determining a non-repeated data sequence of the to-be-optimized storage data.

[0123] After obtaining the to-be-optimized storage data, the server can traverse the obtained to-be-optimized storage data to determine each non-repeated data contained in the to-be-optimized storage data. The server traverses the to-be-optimized storage data to determine each non-repeated data, which is similar to the process of traversing the to-be-stored data to determine the repeated data as described above. During the traversal process, after the server determines whether a data is the specified data, if the data is not the specified data, the server can determine that the data is the non-repeated data. After the server determines each non-repeated data, each non-repeated data can be arranged in sequence according to the arrangement order of each non-repeated data in the to-be-optimized storage data to obtain a non-repeated data sequence of the to-be-optimized storage data.

[0124] As an embodiment, when traversing the to-be-stored data, the server can obtain each non-repeated data contained in the to-be-stored data. Thus, when determining to optimize the storage of the to-be-stored data, the server can directly obtain each non-repeated data contained in the to-be-optimized storage data, without traversing the to-be-optimized storage data.

[0125] As an embodiment, after obtaining each non-repeated data of the to-be-optimized storage data, the server can determine the minimum non-repeated data among the non-repeated data. After obtaining the minimum non-repeated data contained in the to-be-optimized storage data, the server can perform difference operation on each non-repeated data and the minimum non-repeated data to determine the data difference between each non-repeated data and the minimum non-repeated data, respectively. Each data difference obtained is used to represent the corresponding non-repeated data, and each data difference is arranged in sequence to obtain a non-repeated data sequence. By calculating the data difference between each non-repeated data and the minimum non-repeated data, and using the data difference to represent the corresponding non-repeated data, the data length of each non-repeated data can be reduced, the amount of data to be stored can be further reduced, and unnecessary occupation of storage resources can be avoided.

[0126] For example, each non-repeated data includes 65582, 65590, 65585, 65536, and 65589, wherein the minimum non-repeated data is 65536. Then, the data difference between 65582 and 65536 is 46, the data difference between 65590 and 65536 is 54, the data difference between 65585 and 65536 is 49, the data difference between 65536 and 65536 is 0, and the data difference between 65589 and 65536 is 53. Using the calculated data difference to represent the corresponding non-repeated data, each non-repeated data can be represented as 46, 54, 49, and 0. It can be seen that storing 46, 54, 49, and 0 occupies less storage resources than storing 65582, 65590, 65585, 65536, and 65589, and further reduces the consumption of storage resources.

[0127] As an embodiment, the to-be-optimized storage data can include null data, and the to-be-optimized storage data can be divided into two data types according to whether the to-be-optimized storage data includes null data, that is, a data-intensive type and a data-sparse type. The to-be-optimized storage data of the data-intensive type does not include null data, and the to-be-optimized storage data of the data-sparse type includes null data.

[0128] After obtaining the to-be-optimized storage data, the server can traverse the obtained to-be-optimized storage data. Taking one of the data included in the to-be-optimized storage data as an example, the processing of other data is similar to the processing of the one data, and details are not repeated here. The server can determine whether the one data is null data. If the server determines that the one data is null data, it is determined that the data type of the to-be-optimized storage data is the data-sparse type, and the determination of whether other data included in the to-be-optimized storage data is null data is not performed. The next data can also be determined to obtain all null data included in the to-be-optimized storage data. If the server determines that the one data is not null data, the next data is determined until all data included in the to-be-optimized storage data is determined to be not null data, and it is determined that the data type of the to-be-optimized storage data is the data-intensive type.

[0129] As an embodiment, if the server obtains the to-be-stored data, the server can obtain the data type of the to-be-stored data and the minimum non-redundant data in the to-be-stored data at the same time in the process of traversing the to-be-stored data. Therefore, when the server determines the to-be-stored data as the to-be-optimized storage data, the data type of the to-be-optimized storage data can be directly obtained, and the minimum non-redundant data of the to-be-optimized storage data can also be directly obtained.

[0130] S204, determining the data identifier sequence of the to-be-optimized storage data based on the data identifiers of the respective non-redundant data and the data identifiers of the respective redundant data included in the to-be-optimized storage data.

[0131] After obtaining the to-be-optimized storage data, the server can determine the data identifier sequence of the to-be-optimized storage data. The server can arrange the data identifiers of the respective non-redundant data included in the to-be-optimized storage data in sequence, and arrange the data identifiers of the respective redundant data included in the to-be-optimized storage data in sequence, to determine the data identifier sequence of the to-be-optimized storage data.

[0132] As an embodiment, the server can determine the data identifier sequence of the to-be-optimized storage data according to the data type of the to-be-optimized storage data. When the data type of the to-be-optimized storage data is the data-intensive type, the server can determine the non-repeated data identifier sequence as the data identifier sequence of the to-be-optimized storage data. When the data type of the to-be-optimized storage data is the data-sparse type, the server can arrange the data identifiers of the respective non-repeated data included in the to-be-optimized storage data in sequence to obtain the non-repeated data identifier sequence of the to-be-optimized storage data, arrange the data identifiers of the respective repeated data included in the to-be-optimized storage data in sequence to obtain the repeated data identifier sequence of the to-be-optimized storage data, and determine the data identifier sequence of the to-be-optimized storage data according to the obtained non-repeated data identifier sequence and repeated data identifier sequence.

[0133] As an embodiment, after obtaining the to-be-optimized storage data, the server can traverse the obtained to-be-optimized storage data. For example, one data included in the to-be-optimized storage data, and the processing of other data is similar to that of the one data, which will not be described herein again. The server can determine whether the one data is the specified data. If the server determines that the one data is the specified data, it is determined that the one data is the repeated data, and the data identifier of the repeated data is obtained. If the server determines that the one data is not the specified data and is not the null data, it is determined that the one data is the non-repeated data, and the data identifier of the non-repeated data is obtained. Until the traversal of all data included in the to-be-optimized storage data is completed, the non-repeated data identifier sequence is determined based on the obtained data identifiers of the respective non-repeated data, and the repeated data identifier sequence is determined based on the obtained data identifiers of the respective repeated data.

[0134] S205, based on the data identifier sequence and the non-repeated data sequence, storing the to-be-optimized storage data.

[0135] After obtaining the data identifier sequence and the non-repeated data sequence, the server can save the data identifier sequence and the non-repeated data sequence in the storage cluster to achieve the purpose of storing the to-be-optimized storage data. Compared with directly saving the to-be-optimized storage data, the method of saving the data identifier sequence and the non-repeated data sequence does not need to save the repeated data in the to-be-optimized storage data, thereby reducing the consumption of storage resources and reducing the unnecessary occupation of storage resources.

[0136] As an embodiment, the data type of the to-be-optimized storage data can be different, and the data identification sequence can be different. Therefore, the server can save the data type of the to-be-optimized storage data when saving the data identification sequence and the non-repeated data sequence in the storage cluster. Thus, the server can determine, based on the saved data type of the to-be-optimized storage data, whether the saved data identification sequence includes only the data identification of each non-repeated data or includes the data identification of each non-repeated data and the data identification of each repeated data, so as to facilitate the server to read data based on the saved data identification sequence and the non-repeated data sequence.

[0137] The server can save the data type of the to-be-optimized storage data in various ways. The server can add a parameter name for saving the data type of the to-be-optimized storage data, and different parameter values corresponding to the parameter name represent different data types. The server can also use a parameter name representing whether the data is optimized for storage to represent that the to-be-optimized storage data is optimized for storage and the data type of the to-be-optimized storage data. For example, for a piece of data saved in the storage cluster, if the attribute parameter of the data is 00, it means that the data is not optimized for storage; if the attribute parameter of the data is 10, it means that the data is optimized for storage, and the data type of the data is the data sparse type; if the attribute parameter of the data is 11, it means that the data is optimized for storage, and the data type of the data is the data dense type, and so on.

[0138] As an embodiment, if the non-repeated data sequence contains the data difference between each non-repeated data and the minimum non-repeated data, the server can save the minimum non-repeated data when saving the data identification sequence and the non-repeated data sequence. Thus, the server can determine each non-repeated data based on the minimum non-repeated data and each data difference, so as to facilitate the server to read data based on the saved data identification sequence and the non-repeated data sequence.

[0139] The server can save the minimum non-repeated data in various ways. The server can add a parameter name for saving the minimum non-repeated data, and the parameter value corresponding to the parameter name is the minimum non-repeated data. The server can add a pointer at the position of the minimum non-repeated data in the non-repeated data sequence, and the position indicated by the pointer saves the minimum non-repeated data. Other positions in the non-repeated data sequence save each data difference, and so on.

[0140] After the server optimally stores each data requiring optimization storage, the server can receive a data read instruction in response to a read operation for a target client. Thus, the server can read corresponding data in the storage cluster based on the data read instruction. The data read by the server based on the data read instruction can be one piece of data, multiple pieces of data, or all data.

[0141] The process of data reading by the server based on the data read instruction is introduced below. The process of data reading is introduced in two cases. One case is reading one piece of target data. The case of reading multiple pieces of data is similar to the case of reading one piece of data, which can be reading each piece of data in multiple pieces of data in sequence, or reading each piece of data at the same time, etc., and the specific implementation is not limited. The other case is reading all data.

[0142] Case 1: Read one piece of target data.

[0143] The data read instruction received by the server can include a specified data identifier, which is used to indicate reading the one piece of target data. The server can pre-store a plurality of preset data identifier intervals. The server can determine a specified data identifier interval containing the specified data identifier based on the plurality of pre-stored data identifier intervals. For example, when the data identifier is a numerical number, the data identifier interval can be [0, 10], [11, 20], [21, 30], etc. When the data identifier is a character number, the data identifier interval can be [a1, b1, c1, d1], [a2, b2, c2, d2], [a3, b3, c3, d3], etc.

[0144] The server can pre-store a mapping relationship between each data identifier interval and the optimized stored data identifier sequence. After the server determines the specified data identifier interval, the server can determine the target data identifier sequence corresponding to the specified data identifier interval based on the mapping relationship. Thus, when reading one piece of target data, the server does not need to obtain all the optimized stored data identifier sequence, but only needs to obtain the target data identifier sequence corresponding to the specified data identifier interval, thereby reducing the amount of data that needs to be processed by the server when reading data.

[0145] After the server determines the target data identifier sequence corresponding to the specified data identifier interval, the server can load the target data identifier sequence and determine whether the target data corresponding to the specified data identifier is non-redundant data based on the target data identifier sequence.

[0146] If the server determines that the target data corresponding to the specified data identifier is non-repetitive data, the server obtains a target non-repetitive data sequence based on the target data identifier sequence. After obtaining the target non-repetitive data sequence, the server reads the target data in the target non-repetitive data sequence based on the specified data identifier. If the server determines that the target data corresponding to the specified data identifier is not non-repetitive data, the server further determines whether the target data is repetitive data, and if the target data is repetitive data, the specified data is determined as the target data.

[0147] As an embodiment, if the data type of the original data corresponding to the target data identifier sequence is also saved in the storage cluster, the server can first determine the data type of the original data corresponding to the target data identifier sequence. If the data type of the original data is data-intensive type, indicating that the original data does not contain null data, it means that the target data identifier sequence only includes a non-repetitive data identifier sequence. If the target data identifier sequence only includes a non-repetitive data identifier sequence, the server determines whether the target data identifier sequence includes the specified data identifier. If the target data identifier sequence includes the specified data identifier, the server determines that the target data is non-repetitive data. If the target data identifier sequence does not include the specified data identifier, the server determines that the target data is repetitive data.

[0148] If the data type of the original data is data-sparse type, indicating that the original data contains null data, it means that the target data identifier sequence includes both a non-repetitive data identifier sequence and a repetitive data identifier sequence. If the target data identifier sequence includes both a non-repetitive data identifier sequence and a repetitive data identifier sequence, the server can first determine whether the non-repetitive data identifier sequence includes the specified data identifier, and then determine whether the repetitive data identifier sequence includes the specified data identifier if the non-repetitive data identifier sequence does not include the specified data identifier. The server can also first determine whether the repetitive data identifier sequence includes the specified data identifier, and then determine whether the non-repetitive data identifier sequence includes the specified data identifier if the repetitive data identifier sequence does not include the specified data identifier.

[0149] If the server determines that the non-repetitive data identifier sequence includes the specified data identifier, the server determines that the target data is non-repetitive data. If the server determines that the non-repetitive data identifier sequence does not include the specified data identifier, and the repetitive data identifier sequence includes the specified data identifier, the server determines that the target data is repetitive data. If the server determines that the non-repetitive data identifier sequence does not include the specified data identifier, and the repetitive data identifier sequence does not include the specified data identifier, the server determines that the target data is null data.

[0150] When the server determines that the specified data identifier is included in the non-repeated data identifier sequence, the server can determine the corresponding target non-repeated data sequence based on the non-repeated data identifier sequence. After obtaining the target non-repeated data sequence, the server can determine the arrangement order of the specified data identifier in the non-repeated data identifier sequence, and based on the arrangement order, determine the non-repeated data in the target non-repeated data sequence in the corresponding arrangement order, and determine the non-repeated data as the target data. When the server determines that the specified data identifier is included in the repeated data identifier sequence, the server can determine the repeated specified data as the target data.

[0151] In the embodiments of the present application, when the server reads a piece of data, it does not need to obtain all the data that has been optimized for storage, nor does it need to decompress the data, thereby improving the efficiency of reading data and reducing the amount of data that the server needs to process.

[0152] Case two: reading all the target data.

[0153] The data reading instruction received by the server can include the data identifiers of all the target data respectively, or can include a specified data identifier, which is used to indicate reading all the target data that has been optimized for storage. For example, the data reading instruction includes the specified data identifier "ALL", so it can be determined that the data reading instruction is used to read all the target data.

[0154] The server loads the target data identifier sequence and the target non-repeated data sequence, and based on the obtained target data identifier sequence and the target non-repeated data sequence, determines the repeated data or non-repeated data corresponding to each data identifier respectively, and obtains all the target data.

[0155] As an embodiment, if the data type of the original data includes the data dense type and the data sparse type, when the data type of the original data is the data dense type, the target data identifier sequence obtained by the server only includes the non-repeated data identifier sequence, so that the server can correspond each data identifier and non-repeated data in turn based on the arrangement order of each data identifier in the non-repeated data identifier sequence and the arrangement order of each non-repeated data in the non-repeated data sequence. For data identifiers that are not included in the non-repeated data identifier sequence, they are all regarded as data identifiers of repeated data, and the data identifiers that are not included in the non-repeated data identifier sequence are all corresponding to specified data, and all the target data is obtained.

[0156] When the data type of the original data is the data sparse type, the target data identifier sequence obtained by the server includes both the non-repeated data identifier sequence and the repeated data identifier sequence, so that the server can sequentially correspond to each data identifier and non-repeated data based on the arrangement order of each data identifier in the non-repeated data identifier sequence and the arrangement order of each non-repeated data in the non-repeated data sequence, and for the data identifier in the repeated data identifier sequence, a specified data is obtained.

[0157] As an embodiment, the storage cluster can also save the data length of the original data, so that the server can determine each target data by using a temporary variable and a data pointer. Taking the data identifier as a numerical number as an example, the server can establish a data sequence based on the data length. When the data type of the original data is the data dense type, the data pointer sequentially points to each bit in the non-repeated data identifier sequence. When the pointer points to the first bit in the non-repeated data identifier sequence, it is determined whether the first numerical number in the target non-repeated data identifier sequence is 1. If the first numerical number is 1, the value of the temporary variable is updated to "true". When the value of the temporary variable is "true", the server reads the first non-repeated data from the target non-repeated data sequence and updates the first bit in the data sequence to obtain the first target data. The server updates the data pointer to point to the second bit in the non-repeated data identifier sequence, and determines whether the second numerical number in the target non-repeated data identifier sequence is 2. If the second numerical number is not 2, the value of the temporary variable is updated to "flase", the second bit in the data sequence is updated to a specified data, and the second target data is obtained. At this time, the data pointer is not updated, and it is determined whether the second numerical number in the target non-repeated data identifier sequence is 3. If the second numerical number is 3, the value of the temporary variable is updated to "true". When the value of the temporary variable is "true", the server reads the second non-repeated data from the target non-repeated data sequence and updates the third bit in the data sequence to obtain the third target data. In this way, each bit in the data sequence is updated until all target data is obtained.

[0158] When the data type of the original data is the data-intensive type, the non-repeated data pointer points to each bit in the non-repeated data identifier sequence in turn, and the repeated data pointer points to each bit in the repeated data identifier sequence in turn. When the non-repeated data pointer points to the first bit in the non-repeated data identifier sequence and the repeated data pointer points to the first bit in the repeated data identifier sequence, if the value number of the first bit in the non-repeated data identifier sequence is greater than the value number of the first bit in the repeated data identifier sequence, the server determines whether the first value number in the repeated data identifier sequence is 1. If the first value number is 1, the server updates the specified data in the first bit of the data sequence to obtain the first bit target data. If the first value number is not 1, the server updates the null value data in the first bit of the data sequence to obtain the first bit target data. The server updates the repeated data pointer to point to the second bit in the repeated data identifier sequence. If the value number of the first bit in the non-repeated data identifier sequence is less than the value number of the second bit in the repeated data identifier sequence, the server determines whether the first value number in the non-repeated data identifier sequence is 2. If the first value number is 2, the server reads the first non-repeated data in the non-repeated data sequence and updates the second bit of the data sequence to obtain the second bit target data. If the first value number is not 2, the server updates the null value data in the second bit of the data sequence to obtain the second bit target data. In this way, each bit in the data sequence is updated to obtain all target data.

[0159] As an embodiment, in the process of storing data, the server can use the data optimization storage method provided in the embodiments of the present application for optimization storage when the feature switch is in the open state, and use other methods, such as a direct storage method, for data storage when the feature switch is in the closed state. For example, the feature switch is “index.skip_zero_enabled”. When the value of “index.skip_zero_enabled” is “true”, it indicates that the feature switch is in the open state. When the value of “index.skip_zero_enabled” is “flase”, it indicates that the feature switch is in the closed state.

[0160] The data optimization storage method provided in the embodiments of the present application is exemplarily introduced below by taking the optimization storage CPU usage rate as an example.

[0161] For example, refer to Figure 3 The server obtains the data to be optimized for storage, which includes 10,000 pieces of data. The data type of the data to be optimized is the data-intensive type. Refer to Table 3.

[0162] Table 3

[0163]

[0164] The data identifiers in the to-be-optimized storage data are numbered by the respective CPU usage rates, and the repeated specified data is "0". The server can determine, by traversing the to-be-optimized storage data, the non-repeated data whose CPU usage rate is not 0, and the numerical number corresponding to the non-repeated data.

[0165] Taking Table 3 as an example, the server can determine that the respective non-repeated data are 0.96, 0.60, 0.52,..., 0.90 in turn, so that the server can obtain the non-repeated data sequence {0.96, 0.60, 0.52,..., 0.90}. When determining the respective non-repeated data, the server can obtain the respective numerical numbers of the respective non-repeated data, and the respective numerical numbers are 1, 2, 6,..., 10000 in turn, so that the server can obtain the data identifier sequence {1, 2, 6,..., 10000}.

[0166] The server can save the non-repeated data sequence {0.96, 0.60, 0.52,..., 0.90} and the data identifier sequence {1, 2, 6,..., 10000} to achieve the purpose of saving the to-be-optimized storage data. Compared with the manner of directly saving the data in Table 3, it is not necessary to save the repeated "0" and the numerical numbers corresponding to the respective "0".

[0167] For another example, please refer to Figure 4 The server obtains to-be-optimized storage data, the to-be-optimized storage data includes 10000 pieces of data, the data type of the to-be-optimized data is a data sparse type, and actually, only 6000 pieces of data are not empty. Please refer to Table 4.

[0168] Table 4

[0169]

[0170] The data identifiers in the to-be-optimized storage data are numbered by the respective CPU usage rates, and the repeated specified data is "0". The server can determine, by traversing the to-be-optimized storage data, the non-repeated data whose CPU usage rate is not 0, and the numerical number corresponding to the non-repeated data, and the repeated data whose CPU usage rate is 0, and the numerical number corresponding to the repeated data.

[0171] For example, referring to Table 4, the server can determine that the respective non-repeated data are 0.96, 0.60, 0.52, …, 0.90 in sequence, and thus the server can obtain a non-repeated data sequence {0.96, 0.60, 0.52, …, 0.90}. When determining the respective non-repeated data, the server can obtain respective numerical identifiers of the respective non-repeated data, which are 3, 6, 7, …, 10000 in sequence, and thus the server can obtain a non-repeated data identifier sequence {3, 6, 7, …, 10000}. The server can determine that respective numerical identifiers of the respective repeated data are 1, 5, …, 9999 in sequence, and thus the server can obtain a repeated data identifier sequence {1, 5, …, 9999}.

[0172] The server can take the non-repeated data identifier sequence {3, 6, 7, …, 10000} and the repeated data identifier sequence {1, 5, …, 9999} as a data identifier sequence. By saving the non-repeated data sequence {0.96, 0.60, 0.52, …, 0.90}, the repeated data identifier sequence {1, 5, …, 9999}, and the data identifier sequence {3, 6, 7, …, 10000}, the server achieves the purpose of saving the to-be-optimized stored data, and compared with directly saving the data in Table 4, the server does not need to save the repeated “0” and does not need to save the numerical identifiers corresponding to the respective null values.

[0173] In the optimized stored data, the server can read one or more pieces of data. After receiving a data reading instruction, the server can determine the corresponding target data according to a specified data identifier included in the data reading instruction.

[0174] For example, the server determines that the original data corresponding to the optimized stored data is of the data-intensive type, based on the data shown in Table 3. The server can pre-store a plurality of data identifier intervals, which are [0, 100], [101, 200], [201, 300], …, [9001, 10000] in sequence, and mapping relationships between the respective data identifier intervals and the optimized stored data identifier sequence. Please refer to Figure 5a If the specified data identifier is the numerical identifier “5”, the server determines that the data identifier interval corresponding to the specified data identifier is [0, 100], and thus the server can determine the target data identifier sequence corresponding to [0, 100], i.e., the target non-repeated data identifier sequence, which is {1, 2, 6, …, 100}, and thus the server can obtain the target non-repeated data sequence corresponding to the target non-repeated data identifier sequence, which is {0.96, 0.60, 0.52, …, 0.25}.

[0175] Since the specified data identifier is the numerical number "5", based on the target non-repeating data identifier sequence, the server can determine that the target data corresponding to the specified data identifier is not non-repeating data, but the repeated specified data "0", so that the server reads the target data "0".

[0176] Please refer to Figure 5b If the specified data identifier is the numerical number "6", the target non-repeating data identifier sequence, the server can determine that the target data corresponding to the specified data identifier is non-repeating data. The server can determine the third non-repeating data, i.e. 0.52, according to the arrangement order of "6" in {1, 2, 6,..., 100}, i.e. the third position, in the target non-repeating data sequence {0.96, 0.60, 0.52,..., 0.25}, so that the server reads the target data "0.52".

[0177] For another example, the server determines that the original data corresponding to the data stored after optimization is of the data sparse type, and continues to take the data shown in Table 4 as an example. The server can pre-store a plurality of data identifier intervals, which are [0, 100], [101, 200], [201, 300],..., [9001, 10000] in turn, and the mapping relationship between each data identifier interval and the data identifier sequence stored after optimization. Please refer to Figure 6a If the specified data identifier is the numerical number "5", the server determines that the data identifier interval corresponding to the specified data identifier is [0, 100], so that the server can determine the target data identifier sequence corresponding to [0, 100], including the target non-repeating data identifier sequence, which is {3, 6, 7,..., 100}, and the target repeating data identifier sequence, which is {1, 5,..., 99}. Therefore, the server can obtain the target non-repeating data sequence {0.96, 0.60, 0.52,..., 0.25} corresponding to the target non-repeating data identifier sequence.

[0178] Since the specified data identifier is the numerical number "5", based on the target non-repeating data identifier sequence, the server can determine that the target data corresponding to the specified data identifier is not non-repeating data. The server can determine that the target data corresponding to the specified data identifier is the repeated specified data "0" based on the target repeating data identifier sequence, so that the server reads the target data "0".

[0179] Please refer to Figure 6bIf the specified data identifier is the numerical value "6", the server can determine that the target data corresponding to the specified data identifier is non-repetitive data. The server can determine the second non-repetitive data, i.e. 0.60, according to the arrangement order of "6" in {3, 6, 7,..., 100}, i.e. the second position, in the target non-repetitive data sequence {0.96, 0.60, 0.52,..., 0.25}, so that the server reads the target data "0.60".

[0180] In the optimized stored data, the server can read all the data. After receiving the data read instruction, the server can determine to read all the data according to the specified data identifier contained in the data read instruction.

[0181] For example, the server determines that the original data corresponding to the optimized stored data is of the data-intensive type, and based on the data shown in Table 3, please refer to Table 5.

[0182] Table 5

[0183] Non-repeating data identification sequence 1 2 6 …… 10000 Data sequence 1 2 3 …… 10000 Data pointer position *

[0184] The server can establish a data sequence based on the length of the stored data. The data pointer is first pointed to the first position of the non-repetitive data identifier sequence, and the numerical value of the first position of the non-repetitive data identifier sequence is "1", so that the first position of the data sequence with the numerical value "1" is non-repetitive data. The server determines the first target data, i.e. 0.95, based on the first position of the non-repetitive data sequence.

[0185] The position of the data pointer is moved from the first position of the non-repetitive data identifier sequence to the second position, and the first position of the non-repetitive data identifier sequence can be marked as "read", please refer to Table 6.

[0186] Table 6

[0187] Non-repeating data identification sequence 1 2 6 …… 10000 Data sequence 1 2 3 …… 10000 Data pointer position Read *

[0188] The data pointer is now pointed to the second position of the non-repetitive data identifier sequence, and the numerical value of the second position of the non-repetitive data identifier sequence is "2", so that the second position of the data sequence with the numerical value "2" is non-repetitive data. The server determines the second target data, i.e. 0.60, based on the second position of the non-repetitive data sequence.

[0189] The position of the data pointer is moved from the second position of the non-repetitive data identifier sequence to the third position, and the second position of the non-repetitive data identifier sequence can be marked as "read", please refer to Table 7.

[0190] Table 7

[0191] Non-repeating data identification sequence 1 2 6 …… 10000 Data sequence 1 2 3 …… 10000 Data pointer position Read Read *

[0192] The data pointer points to the third bit of the non-repeated data identifier sequence at this time, the numerical value of the third bit of the non-repeated data identifier sequence is "6", the numerical value of the third bit of the non-repeated data identifier sequence is greater than the numerical value "3" of the third bit of the data sequence, so it is indicated that the third bit of the data sequence with the numerical value "3" is the specified repeated data "0". The server determines the third target data, which is 0.

[0193] The position of the data pointer does not change, and the server continues to determine the size relationship between the numerical value of the third bit of the non-repeated data identifier sequence and the numerical value "4" of the fourth bit of the data sequence. The numerical value of the fourth bit of the non-repeated data identifier sequence is greater than the numerical value of the fourth bit of the data sequence, so the server determines the fourth target data, which is 0.

[0194] Until the server determines that the numerical value of the third bit of the non-repeated data identifier sequence is equal to the numerical value "6" of the sixth bit of the data sequence, it is indicated that the sixth bit of the data sequence is non-repeated data. The server determines the sixth target data based on the third bit of the non-repeated data sequence, which is 0.52.

[0195] After obtaining 10,000 pieces of data, the server reads the target data, and can read out the data as shown in Table 3.

[0196] For another example, the server determines that the original data corresponding to the optimized stored data is of the data sparse type, and based on the data shown in Table 4, please refer to Table 8 and Table 9.

[0197] Table 8

[0198]

[0199] Table 9

[0200]

[0201] The server can establish a data sequence based on the length of the stored data. The non-repeated data pointer first points to the first bit of the non-repeated data identifier sequence, and the repeated data pointer first points to the first bit of the repeated data identifier sequence. The numerical value of the first bit of the non-repeated data identifier sequence is "3", so it is indicated that the first bit of the data sequence with the numerical value "1" is not non-repeated data. The numerical value of the first bit of the repeated data identifier sequence is "1", so it is indicated that the first bit of the data sequence with the numerical value "1" is repeated data "0". The server determines the first target data, which is 0.

[0202] The position of the repeated data pointer is moved from the first bit of the repeated data identifier sequence to the second bit, and the first bit of the repeated data identifier sequence can be marked as "read", please refer to Table 10.

[0203] Table 10

[0204]

[0205] The non-repeated data pointer points to the first bit of the non-repeated data identification sequence, and the repeated data pointer points to the second bit of the repeated data identification sequence. The numerical value of the first bit of the non-repeated data identification sequence is 3, which is greater than the numerical value of the second bit of the data sequence, which is 2. Therefore, the second bit of the data sequence with the numerical value of 2 is not non-repeated data. The numerical value of the second bit of the repeated data identification sequence is 5, which is greater than the numerical value of the second bit of the data sequence, which is 2. Therefore, the second bit of the data sequence with the numerical value of 2 is not repeated data. Therefore, the server determines that the second bit of the data sequence is a null value.

[0206] The position of the non-repeated data pointer and the position of the repeated data pointer are not moved. The numerical value of the first bit of the non-repeated data identification sequence is 3, which is equal to the numerical value of the third bit of the data sequence, which is 3. Therefore, the third bit of the data sequence with the numerical value of 3 is non-repeated data. According to the non-repeated data sequence {0.96, 0.60, 0.52, …, 0.90}, the server determines that the first bit is 0.96, and determines 0.96 as the target data of the third bit of the data sequence.

[0207] The position of the non-repeated data pointer is moved from the first bit to the second bit of the non-repeated data identification sequence, and the first bit of the non-repeated data identification sequence can be marked as “read”. Please refer to Table 11.

[0208] Table 11

[0209]

[0210] Until 10,000 pieces of data are obtained, the server reads the target data, and can read out the data as shown in Table 4.

[0211] In actual application, for a business with a high proportion of repeated data, such as “0” accounting for more than 96%, if the data type is a data-intensive type, then the size of the storage resource of the storage cluster before data optimization storage is 122.4 MB, and the size of the storage resource after data optimization storage is 11.8 MB. If the data type is a data-sparse type, then the size of the storage resource of the storage cluster before data optimization storage is 133.3 MB, and the size of the storage resource after data optimization storage is 10.4 MB.

[0212] Based on the same inventive concept, the embodiment of the present application provides a data optimization storage device, which is equivalent to the server discussed above, and can realize the functions corresponding to the data optimization storage method. Please refer to Figure 7 The device comprises a transceiver module 701 and a processing module 702, wherein:

[0213] The transceiver module 701 is configured to traverse the to-be-stored data, and determine the data proportion of the repeated data in the to-be-stored data.

[0214] The transceiver module 701 is further configured to, when the data proportion meets the preset optimization condition, take the to-be-stored data as the to-be-optimized storage data.

[0215] The processing module 702 is configured to determine the non-repeated data sequence of the to-be-optimized storage data based on each non-repeated data contained in the to-be-optimized storage data.

[0216] The processing module 702 is further configured to determine the data identifier sequence of the to-be-optimized storage data based on the data identifier of each non-repeated data and the data identifier of each repeated data contained in the to-be-optimized storage data.

[0217] The processing module 702 is further configured to store the to-be-optimized storage data based on the data identifier sequence and the non-repeated data sequence.

[0218] In a possible embodiment, the transceiver module 701 is specifically configured to:

[0219] Obtain the to-be-stored data.

[0220] Traverse the to-be-stored data, and respectively determine whether each data included in the to-be-stored data is the specified data.

[0221] Count the repetition number of the repeated data that is the specified data in each data.

[0222] Determine the data proportion of the repeated data in the to-be-stored data based on the repetition number.

[0223] In a possible embodiment, the processing module 702 is specifically configured to:

[0224] Determine the data proportion of the repeated data in the to-be-stored data based on the ratio between the repetition number and the total data amount, wherein the total data amount is the number of data contained in the to-be-stored data; or

[0225] Determine the data proportion of the repeated data in the to-be-stored data based on the ratio between the repetition number and the non-repetition number, wherein the non-repetition number is the number of non-repeated data contained in the to-be-stored data.

[0226] In a possible embodiment, the processing module 702 is specifically configured to:

[0227] determining minimum non-repetitive data in each non-repetitive data;

[0228] determining data difference between each non-repetitive data and the minimum non-repetitive data respectively;

[0229] adopting each data difference to represent the corresponding non-repetitive data respectively, and arranging each data difference in sequence to obtain a non-repetitive data sequence.

[0230] In a possible implementation, the processing module 702 is specifically configured to:

[0231] determining a non-repetitive data identifier sequence of the to-be-optimized storage data based on the data identifiers of the non-repetitive data respectively;

[0232] if the data type of the to-be-optimized storage data is a data-intensive type, adopting the non-repetitive data identifier sequence as the data identifier sequence, wherein the data-intensive type represents that the to-be-optimized storage data does not contain null data.

[0233] In a possible implementation, the processing module 702 is further configured to:

[0234] after determining the non-repetitive data identifier sequence of the to-be-optimized storage data based on the data identifiers of the non-repetitive data respectively, if the data type of the to-be-optimized storage data is a data-sparse type, determining a repetitive data identifier sequence of the to-be-optimized storage data based on the data identifiers of the repetitive data contained in the to-be-optimized storage data respectively, wherein the data-sparse type represents that the to-be-optimized storage data contains null data.

[0235] adopting the non-repetitive data identifier sequence and the repetitive data identifier sequence as the data identifier sequence.

[0236] In a possible implementation, the processing module 702 is specifically configured to:

[0237] traversing the to-be-optimized storage data to determine whether each data included in the to-be-optimized storage data is the specified data respectively;

[0238] adopting the data identifiers of the non-repetitive data in each data that is not the specified data to obtain a non-repetitive data identifier sequence.

[0239] arranging the data identifiers of each non-repetitive data in sequence according to the arrangement order of the non-repetitive data in the non-repetitive data sequence to obtain the non-repetitive data identifier sequence.

[0240] In a possible implementation, the processing module 702 is specifically configured to:

[0241] traversing the to-be-optimized storage data to determine whether each data included in the to-be-optimized storage data is the specified data respectively;

[0242] obtain data identifiers of the duplicate data based on the duplicate data being specified data in each of the data;

[0243] arrange the data identifiers of each of the duplicate data in sequence to obtain a duplicate data identifier sequence.

[0244] In a possible implementation, the processing module 702 is further configured to:

[0245] after storing the to-be-optimized storage data based on the data identifier sequence and the non-duplicate data sequence, receive a data reading instruction, wherein the data reading instruction comprises a specified data identifier;

[0246] determine a specified data identifier interval containing the specified data identifier in a plurality of preset data identifier intervals;

[0247] load a target data identifier sequence corresponding to the specified data identifier interval based on a mapping relationship between the pre-stored data identifier interval and the optimized storage data identifier sequence;

[0248] determine whether target data corresponding to the specified data identifier is non-duplicate data based on the target data identifier sequence;

[0249] if the target data corresponding to the specified data identifier is non-duplicate data, obtain a target non-duplicate data sequence based on the target data identifier sequence;

[0250] read the target data in the target non-duplicate data sequence based on the specified data identifier.

[0251] In a possible implementation, the processing module 702 is further configured to:

[0252] before determining whether the target data corresponding to the specified data identifier is non-duplicate data based on the target data identifier sequence, determine a data type of original data corresponding to the target data identifier sequence, wherein the original data is to-be-optimized storage data corresponding to the target data identifier sequence before the optimization storage is performed, and the data type of the original data is used to represent whether the original data contains null data;

[0253] The processing module 702 is specifically configured to:

[0254] if the data type of the original data is a data sparse type, determine whether the target data corresponding to the specified data identifier is non-duplicate data based on a target non-duplicate data identifier sequence in the target data identifier sequence.

[0255] In a possible implementation, the processing module 702 is further configured to:

[0256] If the data type of the original data is the data sparse type, the target non-repeated data identifier sequence in the target data identifier sequence is determined, and whether the target data corresponding to the specified data identifier is non-repeated data is determined. If the target data corresponding to the specified data identifier is not non-repeated data, the target repeated data identifier sequence in the target data identifier sequence is determined, and whether the target data corresponding to the specified data identifier is repeated data is determined.

[0257] If the target data corresponding to the specified data identifier is repeated data, the repeated specified data is taken as the target data.

[0258] Based on the same inventive concept, an embodiment of the present application provides a computer device, which is described below.

[0259] Please refer to Figure 8 The data optimization storage device described above can run on the computer device 800, and the current version and the historical version of the data storage program and the application software corresponding to the data storage program can be installed on the computer device 800. The computer device 800 includes a display unit 840, a processor 880, and a memory 820, wherein the display unit 840 includes a display panel 841, which is used to display the user interaction interface and the like.

[0260] In a possible embodiment, the display panel 841 can be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0261] The processor 880 is used to read the computer program and then execute the method defined by the computer program. For example, the processor 880 reads the data storage program or a file, so as to run the data storage program on the computer device 800 and display the corresponding interface on the display unit 840. The processor 880 can include one or more general-purpose processors, and can also include one or more digital signal processors (DSPs) for performing related operations to implement the technical solutions provided in the embodiments of the present application.

[0262] The memory 820 generally includes internal memory and external memory. The internal memory can be random access memory (RAM), read only memory (ROM), and cache memory (CACHE), etc. The external memory can be a hard disk, an optical disk, a USB disk, a floppy disk, or a tape drive, etc. The memory 820 is used to store computer programs and other data. The computer programs include application programs corresponding to each client, etc. The other data can include data generated after the operating system or application programs are run, including system data (such as configuration parameters of the operating system) and user data. In the embodiments of the present application, program instructions are stored in the memory 820, and the processor 880 executes the program instructions stored in the memory 820 to implement any one of the data optimization storage methods discussed in the foregoing figures.

[0263] The display unit 840 is used to receive inputted digital information, character information, or contact touch operation / non-contact gesture, and generate signal input related to user settings and function control of the computer device 800, etc. Specifically, in the embodiments of the present application, the display unit 840 can include a display panel 841. The display panel 841, for example, a touch screen, can collect touch operations (such as user operations on or near the display panel 841 using fingers, styluses, or any suitable objects or accessories) of a user on or near the display panel 841, and drive corresponding connection devices according to pre-set programs.

[0264] In a possible embodiment, the display panel 841 can include two parts of a touch detection device and a touch controller. The touch detection device detects the touch position of the player and detects the signal brought by the touch operation, and transmits the signal to the touch controller. The touch controller receives the touch information from the touch detection device, converts it into touch coordinates, and sends it to the processor 880, and can receive commands from the processor 880 and execute them.

[0265] The display panel 841 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave, etc. In addition to the display unit 840, the computer device 800 can further include an input unit 830. The input unit 830 can include a graphical input device 831 and other input devices 832. The other input devices can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, on / off keys, etc.), trackballs, mice, joysticks, etc.

[0266] In addition to the above, the computer device 800 can further include a power supply 890 for supplying power to other modules, an audio circuit 860, a near field communication module 870, and an RF circuit 810. The computer device 800 can further include one or more sensors 850, such as an acceleration sensor, a light sensor, a pressure sensor, etc. The audio circuit 860 specifically includes a speaker 861 and a microphone 862, etc., for example, the computer device 800 can collect the user's voice through the microphone 862, perform corresponding operations, etc.

[0267] As an embodiment, the number of processors 880 can be one or more, and the processor 880 and the memory 820 can be coupled or relatively independent.

[0268] As an embodiment, Figure 8 The processor 880 in the computer device 800 can be used to implement the functions of the transceiver module 701 and the processing module 702 in the computer device 800. Figure 7

[0269] As an embodiment, Figure 8 The processor 880 in the computer device 800 can be used to implement the functions of the server 102 corresponding to the functions discussed above.

[0270] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the foregoing program can be stored in a computer readable storage medium, and the program is executed to execute the steps of the above-mentioned method embodiments; and the foregoing storage medium includes: mobile storage equipment, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disc or optical disc and various storage program codes.

[0271] Alternatively, the integrated unit of the present application can be stored in a computer readable storage medium if it is realized in the form of a software function module and sold or used as an independent product. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present application. The foregoing storage medium includes: mobile storage equipment, ROM, RAM, magnetic disc or optical disc, and various storage program codes.

[0272] ​Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A data optimization storage method, characterized by, The method comprises the following steps: traversing the to-be-stored data to determine a data proportion of repeated data in the to-be-stored data; when the data proportion meets a preset optimization condition, taking the to-be-stored data as to-be-optimized storage data; using a data difference between each non-repeated data included in the to-be-optimized storage data and the smallest non-repeated data in the to-be-optimized storage data to represent the corresponding non-repeated data, and sequentially arranging each data difference to obtain a non-repeated data sequence of the to-be-optimized storage data; based on the data identifier of each non-repeated data and the data identifier of each repeated data included in the to-be-optimized storage data, determining a data identifier sequence of the to-be-optimized storage data; based on the data identifier sequence and the non-repeated data sequence, storing the to-be-optimized storage data; wherein, based on the data identifier of each non-repeated data and the data identifier of each repeated data included in the to-be-optimized storage data, determining the data identifier sequence of the to-be-optimized storage data comprises: traversing the to-be-optimized storage data, and obtaining a data identifier of non-repeated data for each non-repeated data in the to-be-stored data that is not a specified data; arranging the data identifier of each non-repeated data in the order of the non-repeated data sequence to obtain the non-repeated data identifier sequence; if the data type of the to-be-optimized storage data is a data-intensive type, taking the non-repeated data identifier sequence as the data identifier sequence, wherein the data-intensive type represents that the to-be-optimized storage data does not include null data.

2. The method of claim 1, wherein, traversing the to-be-stored data to determine a data proportion of repeated data in the to-be-stored data comprises: obtaining to-be-stored data; traversing the to-be-stored data to determine whether each data included in the to-be-stored data is a specified data; counting the number of repeated data that is the specified data among the data; based on the number of repeated data, determining the data proportion of repeated data in the to-be-stored data.

3. The method of claim 2, wherein, based on the number of repeated data, determining the data proportion of repeated data in the to-be-stored data comprises: based on the ratio between the number of repeated data and the total amount of data, determining the data proportion of repeated data in the to-be-stored data, wherein the total amount of data is the number of data included in the to-be-stored data; or based on the ratio between the number of repeated data and the number of non-repeated data, determining the data proportion of repeated data in the to-be-stored data, wherein the number of non-repeated data is the number of non-repeated data included in the to-be-stored data.

4. The method of claim 1, wherein, after determining the non-repeated data identifier sequence of the to-be-optimized storage data based on the data identifier of each non-repeated data, the method further comprises: if the data type of the to-be-optimized storage data is a data sparse type, determining a repeated data identifier sequence of the to-be-optimized storage data based on the data identifier of each repeated data included in the to-be-optimized storage data, wherein the data sparse type represents that the to-be-optimized storage data includes null data; The non-repeated data identification sequence and the repeated data identification sequence are taken as the data identification sequence.

5. The method of claim 4, wherein, If the data type of the data to be optimized is a data sparse type, a repeated data identification sequence of the data to be optimized is determined based on respective data identifications of respective repeated data included in the data to be optimized. traversing the data to be optimized, respectively determining whether each data included in the data to be stored is specified data; based on the repeated data among the respective data being the specified data, obtaining a data identification of the repeated data; sequentially arranging the respective data identifications of the respective repeated data to obtain the repeated data identification sequence.

6. The method according to any one of claims 1 to 5, characterized in that, After storing the data to be optimized based on the data identification sequence and the non-repeated data sequence, the method further includes: receiving a data reading instruction, wherein the data reading instruction includes a specified data identification; determining a specified data identification interval containing the specified data identification among a plurality of preset data identification intervals; loading a target data identification sequence corresponding to the specified data identification interval based on a mapping relationship between the preset data identification interval and the optimized stored data identification sequence; determining whether target data corresponding to the specified data identification is non-repeated data based on the target data identification sequence; if the target data corresponding to the specified data identification is non-repeated data, obtaining a target non-repeated data sequence based on the target data identification sequence; reading the target data in the target non-repeated data sequence based on the specified data identification.

7. The method of claim 6, wherein, Before determining whether the target data corresponding to the specified data identification is non-repeated data based on the target data identification sequence, the method further includes: determining a data type of original data corresponding to the target data identification sequence, wherein the original data is the data to be optimized corresponding to the target data identification sequence before optimization storage, and the data type of the original data is used to represent whether the original data contains null data; determining whether the target data corresponding to the specified data identification is non-repeated data based on the target data identification sequence, including: if the data type of the original data is a data sparse type, determining whether the target data corresponding to the specified data identification is non-repeated data based on a target non-repeated data identification sequence in the target data identification sequence.

8. The method of claim 7, wherein, After determining whether the target data corresponding to the specified data identification is non-repeated data based on the target non-repeated data identification sequence in the target data identification sequence if the data type of the original data is a data sparse type, the method further includes: if the target data corresponding to the specified data identification is not non-repeated data, determining whether the target data corresponding to the specified data identification is repeated data based on a target repeated data identification sequence in the target data identification sequence; if the target data corresponding to the specified data identification is repeated data, taking the repeated specified data as the target data.

9. A data optimization storage device, characterized by, The method includes: a transceiver module configured to traverse the data to be stored and determine a data proportion of repeated data in the data to be stored; The transceiving module is further configured to: when the data proportion satisfies a preset optimization condition, taking the to-be-stored data as to-be-optimized storage data. The processing module is configured to: represent each non-redundant data in the to-be-optimized storage data by a data difference value between each non-redundant data and the smallest non-redundant data in the to-be-optimized storage data, and sequentially arrange each data difference value to obtain a non-redundant data sequence of the to-be-optimized storage data. The processing module is further configured to: determine a data identifier sequence of the to-be-optimized storage data based on the data identifier of each non-redundant data and the data identifier of each redundant data included in the to-be-optimized storage data. The processing module is further configured to: store the to-be-optimized storage data based on the data identifier sequence and the non-redundant data sequence. The processing module is specifically configured to: traverse the to-be-optimized storage data, and obtain a data identifier of a non-redundant data that is not a specified data in each data included in the to-be-stored data; arrange the data identifier of each non-redundant data in a sequence according to the arrangement order of the non-redundant data represented by the non-redundant data sequence to obtain the non-redundant data identifier sequence; if a data type of the to-be-optimized storage data is a data-intensive type, taking the non-redundant data identifier sequence as the data identifier sequence, wherein the data-intensive type represents that the to-be-optimized storage data does not include null data.

10. A computer device, comprising: comprise: a memory configured to store program instructions; a processor configured to invoke the program instructions stored in the memory, and execute the method in any one of claims 1-8 according to the obtained program instructions.

11. A computer readable storage medium, characterized in that, The storage medium stores computer executable instructions, and the computer executable instructions are used to make a computer execute the method in any one of claims 1-8.

Images