Data storage method, device, storage medium and electronic device

Abstract

The present application discloses a data storage method, device, storage medium and electronic device, and relates to the field of data storage technology. First, the data to be stored is obtained, and the target coding model corresponding to the data to be stored is determined from the coding model set; then the data to be stored is stored based on the target coding model. Since the corresponding target coding model can be determined according to the data characteristics of the data to be stored or the user's needs before the data to be stored is stored, and then the data to be stored is stored according to the target coding model, the coding mode during data storage can be enriched and the writing efficiency during data storage can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a data storage method, device, storage medium and electronic device. Background Art

[0002] With the development of science and technology, people's demand for the use of terminals is increasing. Various smart terminals generate massive amounts of data every moment. The data generated by different smart terminals are in different forms. Therefore, the data storage method has also become one of the research focuses of technical personnel in this field.

[0003] Traditional file storage systems are not suitable for unstructured data of various forms, while object storage systems can meet the storage needs of various forms of data of various intelligent terminals through simple interfaces, and object storage systems have good scalability and can meet the storage needs of unlimited capacity. However, in related technologies, object storage systems have the problem of a single encoding mode when storing data. Summary of the invention

[0004] The present application provides a data storage method, device, storage medium and terminal, which can solve the technical problem of a single encoding mode when an object storage system stores data in the related art.

[0005] In a first aspect, an embodiment of the present application provides a data storage method, which is applied to an object storage system, and the method includes:

[0006] Acquire data to be stored, and determine a target coding model corresponding to the data to be stored from a set of coding models;

[0007] The data to be stored is stored based on the target coding model.

[0008] Optionally, the method of obtaining the data to be stored and determining the target coding model corresponding to the data to be stored from a coding model set further includes: determining at least one sample data type, wherein the sample data type includes one or more of data size, data structure, and data source; determining at least one sample coding model, wherein the sample coding model includes at least the number of available areas, the coding method, the number of data blocks, and the number of check blocks; and obtaining a coding model set by making one-to-one correspondence between each sample data type and each sample coding model.

[0009] Optionally, determining the target coding model corresponding to the data to be stored from the coding model set includes: performing data analysis on the data to be stored to determine a target data type corresponding to the data to be stored; determining a target sample data type corresponding to the target data type from the coding model set, and using the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0010] Optionally, determining the target coding model corresponding to the data to be stored from the coding model set includes: obtaining a storage instruction input for the data to be stored, the storage instruction carrying a target data type corresponding to the data to be stored; determining a target sample data type corresponding to the target data type from the coding model set, and using the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0011] Optionally, storing the data to be stored based on the target coding model includes: encoding the data to be stored based on the target coding model to obtain available areas, data blocks and check blocks corresponding to the data to be stored; obtaining reliability parameters of the data to be stored, and determining the minimum number of data blocks and check blocks corresponding to the data to be stored that are successfully written to the available areas according to the referenceability parameters; and storing the data to be stored based on the minimum number.

[0012] Optionally, storing the data to be stored based on the minimum number of copies includes: writing the data blocks and check blocks of the data to be stored to the corresponding available areas respectively, and monitoring the target number of copies of the data blocks and check blocks of the data to be stored that are successfully written to the corresponding available areas; if it is monitored that the target number of copies is greater than or equal to the minimum number of copies, it is determined that the reliability of the data to be stored meets the preset requirements and the data to be stored is successfully stored; if it is monitored that the target number of copies is less than the minimum number of copies, it is determined that the reliability of the data to be stored does not meet the preset requirements and the data to be stored fails to be stored.

[0013] Optionally, after determining that the target number of copies is greater than or equal to the minimum number of copies, the method further includes: stopping writing of data blocks and check blocks in the data to be stored, and deleting unwritten data blocks and check blocks in the data to be stored.

[0014] In a second aspect, an embodiment of the present application provides a data storage device, applied to an object storage system, the device comprising:

[0015] A coding model determination module, used to obtain data to be stored, and determine a target coding model corresponding to the data to be stored from a set of coding models;

[0016] A data storage module is used to store the data to be stored based on the target coding model.

[0017] In a third aspect, an embodiment of the present application provides a computer storage medium, wherein the computer storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of the above-mentioned method.

[0018] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0019] The beneficial effects brought about by the technical solutions provided by some embodiments of the present application include at least:

[0020] The embodiment of the present application provides a data storage method, which is applied to an object storage system, firstly obtaining the data to be stored, determining the target coding model corresponding to the data to be stored from the coding model set; and then storing the data to be stored based on the target coding model. Since the corresponding target coding model can be determined according to the data characteristics of the data to be stored or the needs of the user before storing the data to be stored, and then the data to be stored is stored according to the target coding model, the coding mode during data storage can be enriched and the writing efficiency during data storage can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 An exemplary system architecture diagram of a data storage method provided in an embodiment of the present application;

[0023] Figure 2 A system interaction diagram of a data storage method provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of a data storage method provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of an erasure code strategy provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of a data storage method provided in accordance with another embodiment of the present application;

[0027] Figure 6 A structural schematic diagram of a data storage method provided by another embodiment of the present application;

[0028] Figure 7 A structural schematic diagram of a data storage method provided by another embodiment of the present application;

[0029] Figure 8 A structural schematic diagram of a data storage device provided in another embodiment of the present application;

[0030] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0032] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0033] Figure 1 An exemplary system architecture diagram of a data storage method provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the system architecture may include at least one terminal 110, a network 120, and at least one server 130, wherein the network 120 is used to provide a medium for a communication link between each terminal 110 and the server 130. The network 120 may include various types of wired communication links or wireless communication links, for example, the wired communication link includes an optical fiber, a twisted pair, or a coaxial cable, and the wireless communication link includes a Bluetooth communication link, a Wireless-Fidelity (Wi-Fi) communication link, or a microwave communication link.

[0035] Among them, when the distance between each terminal is far, for example, each terminal is in a different country, if each terminal needs to perform long-distance file transmission, in order to ensure the stability of file transmission during long-distance file transmission, the terminals can be connected to each other using the operating network fiber optic line to form a relatively large internal private network of the enterprise. You can also use two fiber optic lines from two different operators to prevent network problems and have a spare line. The bandwidth of each line is 50M or above.

[0036] Each terminal may be hardware or software. When each terminal is hardware, it may be various electronic devices with a display screen, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers. When each terminal is software, it may be installed in the electronic devices listed above. It may be implemented as multiple software or software modules (for example, to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.

[0037] When each terminal is hardware, a display device may be installed thereon. The display device may be any device capable of realizing a display function, such as a cathode ray tube display (CR), a light-emitting diode display (LED), an electronic ink screen, a liquid crystal display (LCD), and a plasma display panel (PDP). Users may use the display device on each terminal to view displayed text, pictures, videos, and other information. In addition, a positioning device, a shooting device, a sound collection device, a sound playback device, a sensor device, and a controller may be installed in the terminal to meet the needs of the terminal user.

[0038] Furthermore, the server 130 may be a business server that provides various services, and a database may be provided in the server 130. It should be noted that the server 130 may be hardware, i.e., a physical server; or software, i.e., a non-physical virtual server, such as a virtual machine on a physical server, a virtual machine built on a cloud platform, etc., which can provide a computer with the same functions as a server or a virtual machine. When the server 130 is hardware, it may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server 130 is software, it may be implemented as multiple software or software modules (e.g., for providing distributed services), or as a single software or software module, which is not specifically limited here.

[0039] Each terminal 110 can interact with the server 130 through the network 120 to receive data from the server 130 or send data to the server 130. Various remote client applications can be installed on each terminal, such as: cloud game remote client, drawing application, video recording application, video playback application, voice collection application, voice interaction application, search application, instant communication tool, mailbox remote client and social platform software.

[0040] Furthermore, in the present application, the server 130 can be used to store the data sent by the above-mentioned terminals 110. The server 130 can also use object storage technology to store the data. Therefore, an object storage system 131 can be set in the server 130. The server 130 stores the above-mentioned data based on the object storage system 131. Of course, the terminal 110 can also read the stored data from the server 130 at any time.

[0041] It should be understood that Figure 1 The number of terminals, networks and servers in the example is only for illustration purposes and may be any number of terminals, networks and servers as required.

[0042] See also Figure 2 , Figure 2 A system interaction diagram of a data storage method provided in an embodiment of the present application is shown below. Figure 1 and Figure 2 This paper introduces the system interaction process in a data storage method.

[0043] S201. The terminal sends data to be stored to the object storage system.

[0044] S202: The object storage system obtains data to be stored, and determines a target coding model corresponding to the data to be stored from a coding model set.

[0045] Optionally, before obtaining the data to be stored and determining the target coding model corresponding to the data to be stored from the coding model set, it also includes: determining at least one sample data type, wherein the sample data type includes one or more of data size, data structure, and data source; determining at least one sample coding model, wherein the sample coding model includes at least the number of available areas, the coding method, the number of data blocks, and the number of check blocks; and obtaining a coding model set by making one-to-one correspondence between each sample data type and each sample coding model.

[0046] Optionally, determining a target coding model corresponding to the data to be stored from a coding model set includes: performing data analysis on the data to be stored to determine a target data type corresponding to the data to be stored; determining a target sample data type corresponding to the target data type from the coding model set, and using the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0047] Optionally, determining a target coding model corresponding to the data to be stored from a coding model set includes: obtaining a storage instruction input for the data to be stored, the storage instruction carrying a target data type corresponding to the data to be stored; determining a target sample data type corresponding to the target data type from the coding model set, and using the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0048] S203: The object storage system stores the data to be stored based on the target coding model.

[0049] Optionally, the data to be stored is stored based on the target coding model, including: encoding the data to be stored based on the target coding model to obtain the available area, data blocks and check blocks corresponding to the data to be stored; obtaining reliability parameters of the data to be stored, and determining the minimum number of data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area according to the referenceability parameters; and storing the data to be stored based on the minimum number of copies.

[0050] Optionally, storing the data to be stored based on the minimum number of copies includes: writing the data blocks and check blocks of the data to be stored to the corresponding available areas respectively, and monitoring the target number of copies of the data blocks and check blocks of the data to be stored that are successfully written to the corresponding available areas; if it is monitored that the target number of copies is greater than or equal to the minimum number of copies, it is determined that the reliability of the data to be stored meets the preset requirements and the data to be stored is stored successfully; if it is monitored that the target number of copies is less than the minimum number of copies, it is determined that the reliability of the data to be stored does not meet the preset requirements and the data to be stored fails to be stored.

[0051] Optionally, if it is determined that the target number of copies is greater than or equal to the minimum number of copies, it also includes: stopping writing to data blocks and check blocks in the data to be stored, and deleting unwritten data blocks and check blocks in the data to be stored.

[0052] In the embodiment of the present application, the data to be stored is first obtained, and the target coding model corresponding to the data to be stored is determined from the coding model set; then the data to be stored is stored based on the target coding model. Since the corresponding target coding model can be determined according to the data characteristics of the data to be stored or the needs of the user before the data to be stored is stored, and then the data to be stored is stored according to the target coding model, the coding mode during data storage can be enriched and the writing efficiency during data storage can be improved.

[0053] See also Figure 3 , Figure 3 A flowchart of a data storage method provided in an embodiment of the present application.

[0054] like Figure 3 As shown, the method includes:

[0055] S301: Acquire data to be stored, and determine a target coding model corresponding to the data to be stored from a coding model set.

[0056] It can be understood that in the embodiment of the present application, the execution subject can be a server or an object storage system in the server. For the convenience of description, the specific execution process of the data storage method is introduced below by taking the execution subject as an object storage system as an example.

[0057] First, let's introduce the concepts of Region and AZ in the object storage system. Region refers to a region or area, and Region can be understood as a computer room in different cities. Regions in the object storage system are spread all over the world, and different regions can generally only be connected through the public network.

[0058] AZ, also known as Availability-Zone, can be understood as different computer rooms in the same city, which are included in the Region. For example, the Beijing Region will include multiple AZs, and a single AZ will consist of one or more IDC computer room parks. AZs are connected through a high-bandwidth, low-latency, and highly redundant dedicated network to ensure secure and reliable data transmission across AZs.

[0059] A Region is composed of multiple AZs. The power supply and network of the computer room are used to ensure that a failure in one AZ will not affect another AZ.

[0060] Furthermore, object storage systems use disks as storage media. In large-scale storage systems, it is common for disk damage to make stored data inaccessible. In more serious cases, there may even be various factors such as power supply system failure in the computer room, explosion in the computer room, and air pollution that damage all disks in the computer room, making the data in the entire availability zone inaccessible. In order to ensure the high availability and high reliability of object storage system data, the current mainstream object storage systems will use a storage mode with multiple availability zones for redundant data backup, ensuring that even if a single availability zone is damaged, it will not affect the storage system's normal provision of data read and write services.

[0061] Object storage systems can use multi-copy strategies and erasure code strategies to achieve redundant backup of stored data. The multi-copy strategy is that the object storage system copies a piece of data into multiple copies and stores them in different areas. When a copy of the data fails, the data can be read from other copies. The erasure code strategy uses coding calculations to achieve storage reliability consistent with multiple copies with fewer redundant backups, greatly improving storage utilization in object storage systems and saving storage costs in object storage systems.

[0062] See also Figure 4 , Figure 4 A schematic diagram of an erasure coding strategy provided in an embodiment of the present application.

[0063] like Figure 4 As shown, in the related art, in the erasure code strategy of the object storage system, the data is generally stored in an asynchronous manner by calculating the erasure code, that is, the data to be stored is first written into multiple temporary files through a multi-copy system, and then the erasure code of the data to be stored is calculated based on a fixed encoding method after the data volume of the temporary file reaches the size of a stripe, so as to obtain the data blocks (D1-D6 in the figure) and check blocks (P1 to P6 in the figure) of the data to be stored, and finally the data blocks and check blocks are written into different available areas (AZ1-AZ3 in the figure), respectively.

[0064] Problems such as disk damage and machine failure in large-scale clusters may cause temporary file data loss. Therefore, the asynchronous calculation of erasure codes requires a multi-copy strategy to ensure the security of temporary file data. Implementing both a multi-copy system and an erasure code system in an object storage system will increase system complexity, increase the difficulty of system management and operation, and not take advantage of system expansion. Secondly, the asynchronous calculation of erasure codes generally fixes the stripe size and encoding mode, and cannot adapt different encoding modes for different types of files, and cannot select different erasure codes for adaptation based on regional characteristics. For example, foreign computer rooms need to implement LRC (Local-Reconstruction-Codes) encoding to reduce the cost of data repair, but the asynchronous calculation of erasure codes often cannot meet this requirement.

[0065] In response to the above problems, in an embodiment of the present application, data is stored by calculating erasure codes in real time. First, a coding model set can be pre-set, where the coding model set includes at least one coding model. Then, after the terminal sends the data to be stored to the server, the object storage system in the server can obtain the data to be stored and determine the target coding model corresponding to the data to be stored from the coding model set according to the data type of the data to be stored or the needs or active modifications of the administrator.

[0066] S302: Store the data to be stored based on the target coding model.

[0067] After determining the target coding model corresponding to the data to be stored, the erasure code calculation can be performed directly on the data to be stored based on the target coding model, without having to write the data to be stored into multiple temporary files through a multi-copy system, thus achieving real-time calculation of erasure codes for data storage. Specifically, the target coding model includes the number of available areas, the coding method, the number of data blocks, and the number of check blocks, so the data to be stored can be encoded by the coding method according to the target coding model to obtain the data blocks and check blocks corresponding to the data to be stored, and the data blocks and check blocks are stored in the corresponding available areas respectively.

[0068] In the embodiment of the present application, the data to be stored is first obtained, and the target coding model corresponding to the data to be stored is determined from the coding model set; then the data to be stored is stored based on the target coding model. Since the corresponding target coding model can be determined according to the data characteristics of the data to be stored or the needs of the user before the data to be stored is stored, and then the data to be stored is stored according to the target coding model, the coding mode during data storage can be enriched and the writing efficiency during data storage can be improved.

[0069] See also Figure 5 , Figure 5 A flowchart of a data storage method provided in another embodiment of the present application.

[0070] like Figure 5 As shown, the method includes:

[0071] S501 , analyzing the data to be stored to determine the target data type corresponding to the data to be stored.

[0072] It can be understood that in an embodiment of the present application, a coding model set can be set in advance so that at least one coding model is set in the coding model set. Specifically, at least one sample data type can be determined, wherein the sample data type includes one or more of data size, data structure, and data source. The data size may refer to the storage space occupied by the original data, the data structure may refer to the structure in which the data is stored, and the data source may refer to the terminal or application scenario that sends the data for storage.

[0073] Then, at least one sample coding model is determined, wherein the sample coding model includes at least the number of available areas, the coding method, the number of data blocks and the number of check blocks. The number of available areas is at least one, and the coding method is any erasure code applied to the object storage system. For example, the erasure code may be LRC (Local-Reconstruction-Codes) coding or RS (Reed-Solomon) coding, wherein RS coding is the most commonly used coding in the object storage system. The implementation principle of RS coding is to divide the data to be stored into N data blocks, and then select a suitable check matrix to generate M check blocks by matrix operation, and write the N+M data blocks to different storage nodes respectively, so that data damage of up to M storage nodes can be tolerated, that is, the original data to be stored can be restored by matrix operation through at least N surviving data blocks (N and M are positive integers greater than 1).

[0074] Finally, each sample data type and each sample coding model are matched one by one to obtain a coding model set. Specifically, in the process of determining the sample coding model corresponding to each sample data type, if the corresponding coding method in a certain coding model can divide the original data to be stored into N data blocks and M check blocks, then the N value affects the data recovery cost. The smaller the N value, the smaller the data dispersion, the larger the fault impact area, and the greater the reconstruction cost; the M value affects the reliability and storage cost. The larger the M value, the higher the fault tolerance, and the smaller the M value, the lower the data redundancy, where the data redundancy is between the minimum redundancy (N / N) and the maximum redundancy ((N+M) / N). Therefore, the sample coding models corresponding to each sample data type can be determined according to one or more of the data size, data structure, and data source. For example, when the data in a certain sample data model is large, the redundancy in the sample coding model corresponding to the sample data type can be set to be smaller, that is, the number of data blocks and the number of check blocks in the sample coding model are larger, so that the fan-out of the data to be stored is higher, thereby saving the storage cost of the system; for another example, when the data in a certain sample data model is small, the number of data blocks and the number of check blocks in the sample coding model corresponding to the sample data type can be set to be smaller, so that the fan-out of the data to be stored is smaller, which can reduce the write delay of small files and improve the write performance.

[0075] After obtaining the coding model set, when determining the target coding model corresponding to the data to be stored from the coding model set, the target data type corresponding to the data to be stored can be determined first. One way is to perform data analysis on the data to be stored through machine learning or a preset algorithm to obtain the target data type of the data to be stored, wherein the target data type includes one or more of data size, data structure, and data source.

[0076] S502: Obtain a storage instruction input for the data to be stored, where the storage instruction carries a target data type corresponding to the data to be stored.

[0077] Another way is that the user can also directly input storage instructions to the object storage system as needed. For example, when the data to be stored is relatively important, a storage instruction can be input to the storage system. The storage instruction carries a data type whose source is important, so that the object storage system can obtain the storage instruction input for the data to be stored and determine that the target data type corresponding to the data to be stored is the data source is important.

[0078] S503: Determine a target sample data type corresponding to the target data type from the coding model set, and use the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0079] Since the coding model set includes the corresponding situations of various sample data types and various sample coding models, after determining the target data type of the data to be stored, the target sample data type corresponding to the target data type can be first determined from the coding model set, and then the target sample coding model corresponding to the target sample data type can be used as the target coding model corresponding to the data to be stored.

[0080] See also Figure 6 , Figure 6 A structural schematic diagram of a data storage method provided in another embodiment of the present application.

[0081] like Figure 6 As shown, after obtaining the data to be stored, the corresponding encoding model can be determined according to the data type of the data to be stored or the needs of the administrator or actively modified. For example, Figure 6In different cases, the data to be stored can correspond to three coding models. In the first coding model (3AZ, N12M15L0), the number of available areas is 3, the coding method is RS coding, the number of data blocks is 12, and the number of check blocks is 15; in the second coding model (3AZ, N6M6L6), the number of available areas is 3, the coding method is LRC coding, the number of data blocks is 6, and the number of check blocks is 12 (including 6 global check blocks and 6 local check blocks); in the third coding model (2AZ, N6M10L2), the number of available areas is 2, the coding method is LRC coding, the number of data blocks is 6, and the number of check blocks is 10 (including 10 global check blocks and 2 local check blocks). The data to be stored can be stored based on the coding model.

[0082] S504 , encoding the data to be stored based on the target coding model to obtain the available area, data block and check block corresponding to the data to be stored.

[0083] Since the target coding model at least includes the number of available areas, the coding method, the number of data blocks and the number of check blocks, the data to be stored can be encoded based on the target coding model to obtain the available areas, data blocks and check blocks corresponding to the data to be stored, as well as the data blocks and check blocks in each available area in the available areas corresponding to the data to be stored.

[0084] S505: Obtain reliability parameters of the data to be stored, and determine the minimum number of data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area based on the reference parameters.

[0085] The principle of erasure code is to divide the data to be stored into N data blocks, and then select a suitable check matrix to generate M check blocks through matrix operation, and write N+M data blocks to different storage nodes respectively. In this way, data damage of up to M storage nodes can be tolerated, that is, the original data to be stored can be restored through matrix operation through the surviving N data blocks. Therefore, the minimum number of data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area is set according to needs. In this way, redundancy can be dynamically adjusted on the basis of meeting the minimum data reliability, thereby improving the availability of the system.

[0086] Specifically, the reliability parameters of the data to be stored can be obtained, and the minimum number of data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area can be determined according to the reference parameters. For example, when a certain data to be stored is divided into 12 data blocks, and then a suitable check matrix is ​​selected to generate 15 check blocks through matrix operations, and the number of available areas for the data to be stored is three, then when the reliability parameter of the data to be stored is the AZ-level reliability requirement, then the object storage system can be required to support AZ-level disaster recovery, and ensure that the data of the object storage system can be accessed normally under the premise that one AZ is unavailable. Theoretically, the threshold of the minimum number of successful writing of data blocks and check blocks is set to 21. Then, when the number of successful writing of data blocks and check blocks is 21, the remaining 6 data blocks can all fail to be written, and it can also be guaranteed that the data of the object storage system can be accessed normally under the premise that one AZ is unavailable.

[0087] See also Figure 7 , Figure 7 A structural schematic diagram of a data storage method provided in another embodiment of the present application.

[0088] like Figure 7 As shown, after obtaining the data to be stored, the target encoding model of the data to be stored is determined to be (3AZ, N12M15L0), that is, the available partitions are 3, the encoding method is RS encoding, the data blocks are 12, and the check blocks are 15. When storing the data to be stored, if the reliability parameter of the data to be stored is determined to be the AZ-level reliability requirement, then the threshold of the minimum number of successful writes of data blocks and check blocks is set to 21. Even if D1, D2, D3, D4, P1 and P2 in AZ1 fail to be written, it can still be guaranteed that the data of the object storage system can be accessed normally under the premise that one AZ is unavailable.

[0089] S506. Store the data to be stored based on the minimum number of copies.

[0090] Specifically, when storing the data to be stored based on the minimum number of copies, the data blocks and check blocks of the data to be stored can be written to the corresponding available areas respectively, and the target number of copies of the data blocks and check blocks of the data to be stored that are successfully written to the corresponding available areas can be monitored. If the target number of copies is monitored to be greater than or equal to the minimum number of copies, it is determined that the reliability of the data to be stored meets the preset requirements and the data to be stored is successfully stored. At this time, in order to reduce the storage pressure of the object storage system, it can be determined that the target number of copies is greater than or equal to the minimum number of copies, then the writing of the data blocks and check blocks in the data to be stored can be stopped, and the unwritten data blocks and check blocks in the data to be stored can be deleted. If the target number of copies is monitored to be less than the minimum number of copies, it is determined that the reliability of the data to be stored does not meet the preset requirements and the storage of the data to be stored has failed.

[0091] In the embodiment of the present application, the data to be stored is first obtained, and the target coding model corresponding to the data to be stored is determined from the coding model set; then the data to be stored is stored based on the target coding model. Since the corresponding target coding model can be determined according to the data characteristics of the data to be stored or the needs of the user before the data to be stored is stored, and then the data to be stored is stored according to the target coding model, the coding mode during data storage can be enriched and the writing efficiency during data storage can be improved.

[0092] Please participate Figure 8 , Figure 8 A schematic diagram of the structure of a data storage device provided in another embodiment of the present application.

[0093] like Figure 8 As shown, the data storage device 800 includes:

[0094] The coding model determination module 810 is used to obtain the data to be stored and determine the target coding model corresponding to the data to be stored from the coding model set.

[0095] The data storage module 820 is used to store the data to be stored based on the target coding model.

[0096] The data storage device 800 further includes:

[0097] The first sample determination module is used to determine at least one sample data type, wherein the sample data type includes one or more of data size, data structure, and data source.

[0098] The second sample determination module is used to determine at least one sample coding model, wherein the sample coding model at least includes the number of available areas, the coding method, the number of data blocks and the number of check blocks.

[0099] The third sample determination module is used to obtain a coding model set by matching each sample data type and each sample coding model one by one.

[0100] The coding model determination module 810 is also used to perform data analysis on the data to be stored, determine the target data type corresponding to the data to be stored; determine the target sample data type corresponding to the target data type from the coding model set, and use the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0101] The coding model determination module 810 is also used to obtain a storage instruction for the data to be stored, the storage instruction carrying a target data type corresponding to the data to be stored; determine a target sample data type corresponding to the target data type from the coding model set, and use the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0102] The data storage module 820 is also used to encode the data to be stored based on the target coding model to obtain the available area, data blocks and check blocks corresponding to the data to be stored; obtain the reliability parameters of the data to be stored, and determine the minimum number of data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area based on the reference parameters; and store the data to be stored based on the minimum number of copies.

[0103] Among them, storing the data to be stored based on the minimum number of copies includes: writing the data blocks and check blocks of the data to be stored to the corresponding available areas respectively, and monitoring the target number of copies of the data blocks and check blocks of the data to be stored that are successfully written to the corresponding available areas; if it is monitored that the target number of copies is greater than or equal to the minimum number of copies, it is determined that the reliability of the data to be stored meets the preset requirements and the data to be stored is successfully stored; if it is monitored that the target number of copies is less than the minimum number of copies, it is determined that the reliability of the data to be stored does not meet the preset requirements and the data to be stored fails to be stored.

[0104] Among them, if it is determined that the target number of copies is greater than or equal to the minimum number of copies, it also includes: stopping the writing of data blocks and check blocks in the data to be stored, and deleting the unwritten data blocks and check blocks in the data to be stored.

[0105] In an embodiment of the present application, a data storage device includes: a coding model determination module, which is used to obtain data to be stored and determine a target coding model corresponding to the data to be stored from a set of coding models; and a data storage module, which is used to store the data to be stored based on the target coding model. Since the corresponding target coding model can be determined according to the data characteristics of the data to be stored or the needs of the user before the data to be stored is stored, and then the data to be stored is stored according to the target coding model, the coding mode during data storage can be enriched and the writing efficiency during data storage can be improved.

[0106] An embodiment of the present application further provides a computer storage medium, which can store multiple instructions, and the instructions are suitable for being loaded by a processor and executing the steps of any method in the above embodiments.

[0107] See also Fig. 9 , Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9As shown, the electronic device 900 may include: at least one processor 901 , at least one network interface 904 , a user interface 903 , a memory 905 , and at least one communication bus 902 .

[0108] The communication bus 902 is used to realize the connection and communication between these components.

[0109] The user interface 903 may include a display screen (Display) and a camera (Camera), and the optional user interface 903 may also include a standard wired interface and a wireless interface.

[0110] The network interface 904 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0111] Among them, the processor 901 may include one or more processing cores. The processor 901 uses various interfaces and lines to connect various parts of the entire electronic device 900, and executes various functions and processes data of the electronic device 900 by running or executing instructions, programs, code sets or instruction sets stored in the memory 905, and calling data stored in the memory 905. Optionally, the processor 901 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 901 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 901, and it can be implemented separately through a chip.

[0112] Among them, the memory 905 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory, ROM). Optionally, the memory 905 includes a non-transitory computer-readable storage medium. The memory 905 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 905 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned method embodiments, etc.; the data storage area may store data involved in the above-mentioned method embodiments, etc. The memory 905 may optionally be at least one storage device located away from the aforementioned processor 901. As Fig. 9 As shown, the memory 905 as a computer storage medium may include an operating system, a network communication module, a user interface module and a data storage program.

[0113] exist Fig. 9 In the electronic device 900 shown, the user interface 903 is mainly used to provide an input interface for the user and obtain data input by the user; and the processor 901 can be used to call the data storage program stored in the memory 905 and specifically perform the following operations:

[0114] The data to be stored is obtained, and a target coding model corresponding to the data to be stored is determined from a set of coding models; and the data to be stored is stored based on the target coding model.

[0115] Optionally, before obtaining the data to be stored and determining the target coding model corresponding to the data to be stored from the coding model set, it also includes: determining at least one sample data type, wherein the sample data type includes one or more of data size, data structure, and data source; determining at least one sample coding model, wherein the sample coding model includes at least the number of available areas, the coding method, the number of data blocks, and the number of check blocks; and obtaining a coding model set by making one-to-one correspondence between each sample data type and each sample coding model.

[0116] Optionally, determining a target coding model corresponding to the data to be stored from a coding model set includes: performing data analysis on the data to be stored to determine a target data type corresponding to the data to be stored; determining a target sample data type corresponding to the target data type from the coding model set, and using the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0117] Optionally, determining a target coding model corresponding to the data to be stored from a coding model set includes: obtaining a storage instruction input for the data to be stored, the storage instruction carrying a target data type corresponding to the data to be stored; determining a target sample data type corresponding to the target data type from the coding model set, and using the target sample coding model corresponding to the target sample data type as the target coding model corresponding to the data to be stored.

[0118] Optionally, the data to be stored is stored based on the target coding model, including: encoding the data to be stored based on the target coding model to obtain the available area, data blocks and check blocks corresponding to the data to be stored; obtaining reliability parameters of the data to be stored, and determining the minimum number of data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area according to the referenceability parameters; and storing the data to be stored based on the minimum number of copies.

[0119] Optionally, storing the data to be stored based on the minimum number of copies includes: writing the data blocks and check blocks of the data to be stored to the corresponding available areas respectively, and monitoring the target number of copies of the data blocks and check blocks of the data to be stored that are successfully written to the corresponding available areas; if it is monitored that the target number of copies is greater than or equal to the minimum number of copies, it is determined that the reliability of the data to be stored meets the preset requirements and the data to be stored is stored successfully; if it is monitored that the target number of copies is less than the minimum number of copies, it is determined that the reliability of the data to be stored does not meet the preset requirements and the data to be stored fails to be stored.

[0120] Optionally, if it is determined that the target number of copies is greater than or equal to the minimum number of copies, it also includes: stopping writing to data blocks and check blocks in the data to be stored, and deleting unwritten data blocks and check blocks in the data to be stored.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0122] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0124] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0125] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0126] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0127] The above is a description of a data storage method, device, storage medium and electronic device provided by the present application. For technicians in this field, according to the ideas of the embodiments of the present application, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data storage method, characterized in that: Applied to an object storage system, the method includes: Acquire data to be stored, and determine a target coding model corresponding to the data to be stored from a coding model set based on a target data type of the data to be stored, wherein the coding model set includes a correspondence between each sample data type and a sample coding model, and the number of data blocks and the number of check blocks in different sample coding models are different. The target coding model includes the number of available areas, a coding method, the number of data blocks, and the number of check blocks, and the coding method is a different coding method in an erasure code strategy; Storing the data to be stored based on the target coding model; The storing the data to be stored based on the target coding model includes: Encode the data to be stored based on the target coding model to obtain the available area, data block, check block corresponding to the data to be stored, and data blocks and check blocks in each available area in the available area corresponding to the data to be stored; Obtaining the reliability parameters of the data to be stored, and determining the minimum number of copies of the data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area according to the reliability parameters; wherein, when the reliability parameter is the reliability requirement at the available area level, the minimum number of copies refers to the number of copies of the data that can be guaranteed to be normally accessed to the object storage system under the premise that one available area is unavailable; The data to be stored is stored based on the minimum number of copies.

2. The method according to claim 1, characterized in that Before obtaining the data to be stored and determining the target coding model corresponding to the data to be stored from the coding model set, the method further includes: Determine at least one of the sample data types, wherein the sample data type includes one or more of data size, data structure, and data source; Determine at least one of the sample coding models, wherein the sample coding model includes at least the number of available areas, the coding mode, the number of data blocks, and the number of check blocks; The coding model set is obtained by making a one-to-one correspondence between each sample data type and each sample coding model.

3. The method according to claim 2, characterized in that The determining, from a set of coding models, a target coding model corresponding to the data to be stored comprises: Performing data analysis on the data to be stored to determine the target data type corresponding to the data to be stored; A target sample data type corresponding to the target data type is determined from the coding model set, and a target sample coding model corresponding to the target sample data type is used as the target coding model corresponding to the data to be stored.

4. The method according to claim 2, characterized in that: The determining, from a set of coding models, a target coding model corresponding to the data to be stored comprises: Obtaining a storage instruction input for the data to be stored, wherein the storage instruction carries a target data type corresponding to the data to be stored; A target sample data type corresponding to the target data type is determined from the coding model set, and a target sample coding model corresponding to the target sample data type is used as a target coding model corresponding to the data to be stored.

5. The method according to claim 1, characterized in that The storing the data to be stored based on the minimum number of copies includes: Writing the data blocks and the check blocks of the data to be stored into the corresponding available areas respectively, and monitoring the target number of the data blocks and the check blocks of the data to be stored that are successfully written into the corresponding available areas; If it is monitored that the target number of copies is greater than or equal to the minimum number of copies, it is determined that the reliability of the data to be stored meets the preset requirements and the data to be stored is successfully stored; If it is monitored that the target number of copies is less than the minimum number of copies, it is determined that the reliability of the data to be stored does not meet the preset requirements and the storage of the data to be stored fails.

6. The method according to claim 5, characterized in that If it is detected that the target number of copies is greater than or equal to the minimum number of copies, the method further includes: Stop writing data blocks and check blocks in the data to be stored, and delete the unwritten data blocks and check blocks in the data to be stored.

7. A data storage device, characterized in that: Applied to an object storage system, the device comprises: A coding model determination module is used to obtain data to be stored, and determine a target coding model corresponding to the data to be stored from a coding model set based on a target data type of the data to be stored, wherein the coding model set includes a correspondence between each sample data type and a sample coding model, and the number of data blocks and the number of check blocks in different sample coding models are different. The target coding model includes the number of available areas, a coding method, the number of data blocks, and the number of check blocks, and the coding method is a different coding method in an erasure code strategy; A data storage module, used for storing the data to be stored based on the target coding model; The data storage module is specifically used to encode the data to be stored based on the target coding model to obtain the available area, data block, check block corresponding to the data to be stored and the data block and check block in each available area in the available area corresponding to the data to be stored; Obtaining the reliability parameters of the data to be stored, and determining the minimum number of copies of the data blocks and check blocks corresponding to the data to be stored that are successfully written to the available area according to the reliability parameters; wherein, when the reliability parameter is the reliability requirement at the available area level, the minimum number of copies refers to the number of copies of the data that can be guaranteed to be normally accessed to the object storage system under the premise that one available area is unavailable; The data to be stored is stored based on the minimum number of copies.

8. A computer storage medium, characterized in that: The computer storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor and executing the steps of the method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 6 when executing the program.

Images