A data encoding and decoding method and system

Abstract

The embodiment of the specification discloses a data coding and decoding method and system. The coding method comprises: obtaining instance data; and coding the instance data based on ontology definition data to obtain coded data for storage. The coded data comprises a description part and a data area, wherein the number of elements of the data area is related to the number of valid fields, and the elements are values of the corresponding valid fields. The invalid fields comprise null fields and / or logically deleted fields in the ontology definition data. The decoding method comprises: obtaining coded data from a storage area; and decoding the coded data based on the ontology definition data.

Description

TECHNICAL FIELD

[0001] The present specification relates to the field of databases, and in particular to a data encoding and decoding method and system. BACKGROUND

[0002] In some cases, data needs to be encoded before being written into a database, and correspondingly, after reading the encoded data from the database, the encoded data needs to be decoded to obtain the data content. Improving the storage efficiency of a database is one of the important directions of database optimization, that is, it is hoped that the same or more content can be stored in less storage space.

[0003] Therefore, it is necessary to provide a data encoding and decoding scheme to improve the storage efficiency of a database. SUMMARY

[0004] One of the embodiments of the present specification provides an adaptive data encoding method, executed by one or more processors, comprising: obtaining instance data; encoding the instance data based on ontology definition data to obtain encoded data for storage. The ontology definition data is used to define one or more fields and the data type of each field. The encoded data comprises a description part and a data area, wherein the number of elements of the data area is related to the number of valid fields, and the elements are values of the corresponding valid fields, which are obtained from the instance data. Wherein, the invalid fields include null fields and / or logically deleted fields in the ontology definition data. When the number of invalid fields meets a first set condition, the description part describes the distribution of valid fields in the one or more fields through the validity information of the one or more fields, and describes the position of the values of the valid fields in the data area through the size information and / or type information of the valid fields; otherwise, the description part describes the distribution of valid fields in the one or more fields and the position of the values of the valid fields in the data area through the size information and / or type information of the one or more fields.

[0005] One of the embodiments of the present specification provides a data encoding system, comprising: an obtaining module, configured to obtain instance data; an encoding module, configured to encode the instance data based on ontology definition data to obtain encoded data for storage. The ontology definition data is used to define one or more fields and the data type of each field. The encoded data comprises a description part and a data area, wherein the number of elements of the data area is related to the number of valid fields, and the elements are values of corresponding valid fields, which are obtained from the instance data. Wherein, the invalid fields include null fields and / or logically deleted fields in the ontology definition data. When the number of invalid fields meets a first set condition, the description part describes the distribution of valid fields in the one or more fields through the validity information of the one or more fields, and describes the position of the values of each valid field in the data area through the size information and / or type information of the valid fields; otherwise, the description part describes the distribution of valid fields in the one or more fields through the size information and / or type information of the one or more fields, and describes the position of the values of each valid field in the data area.

[0006] One of the embodiments of the present specification provides a data encoding method, executed by one or more processors, comprising: obtaining instance data; encoding the instance data based on ontology definition data to obtain encoded data for storage; the ontology definition data is used to define one or more fields and the data type of each field. Wherein, the encoded data comprises: a size list, the elements of which correspond to the one or more fields one by one, and the elements indicate the storage space size of the values of the corresponding fields in the data area; a type list, the elements of which correspond to the one or more fields one by one, and the elements indicate the data type of the corresponding fields; a data area, the number of elements of which is related to the number of valid fields, and the elements are values of corresponding valid fields, which are obtained from the instance data. Wherein, the invalid fields include null fields and / or logically deleted fields in the ontology definition data.

[0007] One of the embodiments of the present specification provides a data encoding method, executed by one or more processors, comprising: obtaining instance data; encoding the instance data based on ontology definition data to obtain encoded data for storage. The ontology definition data is used to define one or more fields and data types of each field. Wherein, the encoded data comprises: a bit table, elements of which correspond to the one or more fields one by one, and the elements are used to indicate whether the corresponding field is valid; a size list comprising elements corresponding to the valid fields, and the elements are used to indicate the storage space size of the value of the corresponding field in the data area; a type list comprising elements corresponding to the valid fields, and the elements are used to indicate the data type of the corresponding field; a data area, the number of elements of which is related to the number of valid fields, and the elements are values of the corresponding valid fields, which are obtained from the instance data. Wherein, the invalid field includes the null field and / or the field logically deleted in the ontology definition data.

[0008] One of the embodiments of the present specification provides a data encoding device, comprising a processor and a storage device, the storage device is used to store instructions. When the processor executes the instructions, the data encoding method as described in any embodiment of the present specification is implemented.

[0009] One of the embodiments of the present specification provides a data decoding method, executed by one or more processors, comprising: reading encoded data from a storage area, the encoded data is obtained based on the data encoding method as described in any embodiment of the present specification; decoding the encoded data based on ontology definition data. The decoding further comprises: obtaining the value of the corresponding field from the data area based on the description part of the encoded data; writing the value into a storage unit, the storage space size of the storage unit is determined based on the data type of the field in the ontology definition data.

[0010] One of the embodiments of the present specification provides a data decoding system, comprising: a reading module, used to read encoded data from a storage area, the encoded data is obtained based on the data encoding method as described in any embodiment of the present specification; a decoding module, used to decode the encoded data based on ontology definition data. The decoding module is further used to: obtain the value of the field from the data area based on the description part of the encoded data; write the value into a storage unit, the storage space size of the storage unit is determined based on the data type of the field in the ontology definition data.

[0011] One of the embodiments of the present specification provides a data decoding device, comprising a processor and a storage device, the storage device is used to store instructions. When the processor executes the instructions, the data decoding method as described in any embodiment of the present specification is implemented.

[0012] One of the embodiments of the present specification provides an encoded data updating method, executed by one or more processors, comprising: generating new ontology definition data when the number of logically deleted fields in the ontology definition data meets a fourth set condition, wherein the ontology definition data is used to define one or more fields and the data type of each field, and the new ontology definition data is obtained by removing the logically deleted fields from the old ontology definition data; for the stored encoded data, decoding the encoded data based on the old ontology definition data to obtain decoded data, and encoding the decoded data based on the new ontology definition data according to the adaptive data encoding method, to obtain new encoded data and store it. Wherein the stored encoded data is obtained by encoding the instance data based on the old ontology definition data according to the adaptive data encoding method.

[0013] One of the embodiments of the present specification provides an encoded data updating system, comprising a generation module and a re-encoding module. The generation module is used to generate new ontology definition data when the number of logically deleted fields in the ontology definition data meets a fourth set condition. Wherein the ontology definition data is used to define one or more fields and the data type of each field, and the new ontology definition data is obtained by removing the logically deleted fields from the old ontology definition data. The re-encoding module is used to decode the encoded data based on the old ontology definition data to obtain decoded data, and encode the decoded data based on the new ontology definition data according to the adaptive data encoding method, to obtain new encoded data and store it. Wherein the stored encoded data is obtained by encoding the instance data based on the old ontology definition data according to the adaptive data encoding method.

[0014] One of the embodiments of the present specification provides an encoded data updating device, comprising a processor and a storage device, wherein the storage device is used to store instructions, and when the processor executes the instructions, the encoded data updating method as described in any embodiment of the present specification is implemented.

[0015] One of the embodiments of the present specification provides an encoded data updating method, executed by a database cluster, the database cluster comprising a center server and a plurality of data servers, the plurality of data servers respectively storing a copy of a same encoded data. The method comprises: generating, by the center server, new ontology definition data when a number of fields logically deleted in ontology definition data reaches a fourth setting condition, wherein the ontology definition data is used to define one or more fields and data types of each field, and the new ontology definition data is obtained by removing the logically deleted fields from old ontology definition data; and performing, by the plurality of data servers, updating processing on the encoded data in the local copy in batches, wherein the encoded data in the local copy is obtained by encoding instance data based on the old ontology definition data according to the adaptive data encoding method as described above. The updating processing comprises: decoding the encoded data based on the old ontology definition data to obtain decoded data; and encoding the decoded data according to the adaptive data encoding method as described above based on the new ontology definition data to obtain new encoded data and store the new encoded data. The method further comprises: deleting the old ontology definition data when each data server updates the encoded data based on the new ontology definition data. BRIEF DESCRIPTION OF DRAWINGS

[0016] The present specification will be further described in the manner of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0017] Figure 1 is a schematic diagram of a data table;

[0018] Figure 2 is a schematic diagram of an application scenario of a data service system according to some embodiments of the present specification;

[0019] Figure 3 is a schematic diagram of a composition of a distributed database system according to some embodiments of the present specification;

[0020] Figure 4 is an exemplary flowchart of a data encoding method according to some embodiments of the present specification;

[0021] Figure 5 shows exemplary indication contents of SIZE under different values;

[0022] Figure 6 shows exemplary indication contents of TYPE under different values;

[0023] Figure 7 is a schematic diagram of the composition of encoded data according to some embodiments of the present specification;

[0024] Figure 8is an exemplary flowchart of a method of decoding data according to some embodiments shown in the specification;

[0025] Figure 9 is an exemplary flowchart of a method of updating encoded data according to some embodiments shown in the specification;

[0026] Figure 10 is an exemplary flowchart of a method of updating encoded data according to yet some embodiments shown in the specification;

[0027] Figure 11 is an exemplary block diagram of an encoding system according to some embodiments shown in the specification;

[0028] Figure 12 is an exemplary block diagram of a decoding system according to some embodiments shown in the specification;

[0029] Figure 13 is an exemplary block diagram of a system of updating encoded data according to some embodiments shown in the specification. DETAILED DESCRIPTION

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some examples or embodiments of the present specification, and for those skilled in the art, without paying creative labor, the present specification can also be applied to other similar scenarios according to these drawings. Unless it is clear from the language environment or otherwise stated, the same reference numbers in the drawings represent the same structure or operation.

[0031] It should be understood that the "system", "device", "unit" and / or "module" used herein is a method for distinguishing different components, elements, parts, sections or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0032] As shown in the specification, unless the context clearly indicates otherwise, the words "one", "a", "an" and / or "the" do not refer to the singular, but can also include the plural. Generally speaking, the terms "include" and "contain" only indicate the inclusion of the steps and elements explicitly identified, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements.

[0033] Flowcharts are used in the specification to illustrate the operation of embodiments of systems in accordance with the specification. It should be understood that the operations in the figures do not necessarily have to be performed in the precise order shown. Rather, various steps can be handled in an inverse order, or simultaneously, or can be omitted entirely. Other operations can also be added to the processes, or some operations can be performed more efficiently by being split up, or combined. Various embodiments can be implemented in hardware, software, or a combination thereof.

[0034] First, some related terms of database are introduced.

[0035] A database is an efficient form of storing and maintaining computer data, which usually uses two-dimensional tables (hereinafter referred to as data tables) to store data. Each row of a data table is usually used to store an instance of data (also referred to as a record), and each column of the data table is used to record a constituent field of the instance of data.

[0036] Figure 1 is a schematic diagram of a data table. As shown in Figure 1 each row of the order data table is used to store an order data. Each order data includes four fields, namely order number, order time, product name and product amount. Correspondingly, the number of columns of the data table is 4, and each column is used to record a field of the order data.

[0037] A data table can have ontology definition data (also referred to as schema) for defining constituent fields of instance data in the data table and data types of the constituent fields. The fields can be defined by naming (numbering) the fields, describing the functions of the fields, etc. The definition / classification of data types in the specification is not limited in any way.

[0038] In some embodiments, the ontology definition information of a data table can include one or more of the following: the table name of the data table (such as the order data table), the column name (i.e. the field name) of each column (such as the order number, the order time, etc.), the data type of each column (i.e. the data type of each field), the status of each column (for example, whether it is logically deleted, whether it is a null column), the primary key, etc. The primary key refers to a field that can be used to uniquely identify an instance of data, for example, the order number can be used as the primary key of the order data table.

[0039] The instance data in a data table can be stored in the form of encoded data (referred to as encoding for short). Correspondingly, after a processor reads the encoding of an instance of data, it can decode the encoding, so as to parse the value of any field in the instance of data.

[0040] The conventional encoding method still retains the value of the invalid field in the encoded data, and the storage efficiency of the database needs to be improved. In view of this, some new encoding methods are provided in embodiments of the specification, which can effectively improve the storage efficiency of the database.

[0041] Figure 2 is a schematic diagram of an application scenario of a data service system according to some embodiments of the present specification. As shown in Figure 2 , the system 200 can include a user terminal 210, a database system 220, and a network 230.

[0042] A user can send a request to the database system 220 through the user terminal 210.

[0043] In some embodiments, the user can send a data table creation request to the database system 220 through the user terminal 210 to create a data table. Creating a data table can refer to generating ontology definition data of the data table. The data table creation request can include a table name (such as an order data table) of the data table to be created, column names (such as order number, order time, product name, product amount, etc.) of each column, data types of each column, and a primary key (such as order number) to indicate the generation of ontology definition data containing these information.

[0044] In some embodiments, the user can send a data table deletion request to the database system 220 through the user terminal 210 to delete a data table. Deleting a data table can refer to deleting ontology definition data of the data table. The data table deletion request can include a table name of the data table to be deleted.

[0045] In some embodiments, the user can send a table structure update request to the database system 220 through the user terminal 210 to update a data table. Updating a data table can refer to modifying ontology definition data of the data table. Specifically, the table structure update request can include one or more of a table name (such as an order table) of the data table to be updated, a column name (such as product quantity) and a data type of an inserted (newly added) column, a column name (such as product amount) of a deleted column, and the like.

[0046] In some embodiments, the user can send a data insertion request to the database system 220 through the user terminal 210 to insert instance data into a data table. Specifically, the data insertion request can include instance data to be inserted (newly added) and a table name (such as an order data table) of the data table where the instance data is to be inserted.

[0047] In some embodiments, the user can send a data deletion request to the database system 220 through the user terminal 210 to delete instance data in a data table. Specifically, the data deletion request can include a value of a primary key (such as order number 0001) of the instance data to be deleted and a table name (such as an order data table) of the data table where the instance data is located.

[0048] It should be noted that, in this specification, the term "deletion" should be understood as logical deletion (e.g. marked deletion) unless otherwise specified in a specific context or otherwise specified. For data that is logically deleted, although the data still physically exists, the database can refuse to provide the data as a query result, for example, prompting the user that the data queried has been deleted.

[0049] In some embodiments, a user can send a data update request to the database system 220 through the user terminal 210 to update the instance data in the database, for example, modify the value of at least part of the fields in a piece of instance data. Specifically, the data update request can include the value of the field to be modified (after modification) of the instance data (e.g. the modified commodity amount), the table name of the data table where the instance data is located (e.g. the order data table), and the value of the primary key of the instance data (e.g. order number 0001).

[0050] In some embodiments, a user can send a data query request to the database system 220 through the user terminal 210 to query the instance data in the database. The user can query the value of all fields in a piece of instance data, or query the value of a specified field in a piece of instance data. Specifically, the query request can include the value of the primary key of the instance data to be queried (e.g. order number 0001), the field name of the field to be queried (e.g. order amount), and the table name of the data table where the instance data is located (e.g. the order data table).

[0051] Data insertion, data deletion, and data update can be collectively referred to as write operation, and correspondingly, data insertion request, data deletion request, and data update request can be collectively referred to as write request. Data query is read operation, and correspondingly, data query request is read request. It should be noted that read / write operation can be triggered by read / write request from the user terminal 210, or can be performed spontaneously by the database system 200 (e.g. updating encoding data).

[0052] In some embodiments, the user terminal 210 can include a smartphone 210-1, a tablet computer 210-2, a laptop computer 210-3, a desktop computer 210-4, or the like, or any combination thereof.

[0053] The database system 220 can process the request from the user terminal 210.

[0054] In some embodiments, in response to the data table creation request from the user terminal 210, the database system 220 can generate and store the ontology definition data of the data table.

[0055] In some embodiments, in response to the data table deletion request from the user terminal 210, the database system 220 can delete the ontology definition data of the data table.

[0056] In some embodiments, in response to a table structure update request from the user end 210, the database system 220 can modify the ontology definition data of the data table.

[0057] In some embodiments, in response to a write request from the user end 210, the database system 220 can obtain instance data, encode the instance data based on the ontology definition data, and store the obtained encoded data.

[0058] In some embodiments, in response to a data query request (i.e., a read request) from the user end 210, the database system 220 can read encoded data from the storage area, decode the encoded data based on the ontology definition data to obtain a data query result.

[0059] For more details about the encoding and decoding, please refer to Figure 4 and the related descriptions.

[0060] In some embodiments, the database system 220 can be a centralized database system, for example, the database system 220 can be implemented on a single server.

[0061] In some embodiments, the database system 220 can be a distributed database system, for example, the database system 220 can be implemented on a server cluster. Among them, multiple copies of data are distributedly stored on multiple machine nodes, for example, stored on machine nodes in different geographical locations (such as different cities). The advantages of the distributed database system include but are not limited to: 1) improve the reliability and availability of the system, when a node fails, the system can operate on the same copy on another node, and a single point failure will not cause the entire system to be paralyzed; 2) improve system performance, the user end can select the nearest database copy for access according to the distance, reduce communication cost, and improve the overall performance of the system.

[0062] It should be noted that the database copies stored on different nodes can be the same for all data, or at least partially the same for some data, which is not specifically limited in the present specification.

[0063] The network 230 connects the components of the system 200 so that the components can communicate with each other. The network between the components in the system can include wired and / or wireless networks. For example, the network 230 can include a cable network, a wired network, a fiber optic network, a telecommunication network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, near field communication (NFC), a bus within a device, a wire within a device, a cable connection, or any combination thereof. The network connection between each two components can be in one of the above manners or in multiple manners.

[0064] Figure 3 is a schematic diagram of a distributed database system according to some embodiments of the present specification. The distributed database system can be implemented on the server cluster 300.

[0065] As shown in Figure 3 , the server cluster 300 can include a central server 310 and a plurality of data servers 320. With reference to the foregoing, the plurality of data servers 320 can each store a copy of the same encoded data.

[0066] The central server 310 is responsible for managing the global ontology definition data (hereinafter referred to as a schema set). Requests related to the ontology definition data (schema) can be forwarded by the data servers to the central server. In some embodiments, the database system 220 can forward a data table creation request from the user end 210 to the central server 310, and in response to the request, the central server 310 can generate a schema and add it to the schema set. In some embodiments, the database system 220 can forward a data table deletion request from the user end 210 to the central server 310, and in response to the request, the central server 310 can delete a schema from the schema set. In some embodiments, the database system 220 can forward a table structure update request from the user end 210 to the central server 310, and in response to the request, the central server 310 can modify a schema in the schema set.

[0067] The data servers 320 are responsible for processing data requests, such as read requests and write requests, from the user end (e.g., the user end 210). Details of the processing of the data requests can be found elsewhere in the present specification.

[0068] The data servers 320 can periodically or aperiodically obtain the latest version of the schema from the central server and re-encode the instance data based on the latest version of the schema. For more details of the re-encoding, reference can be made toFigures 9-10 and related descriptions.

[0069] Figure 4 is an exemplary flowchart of a data encoding method according to some embodiments illustrated in the present specification. The flow 400 can be executed by one or more processors (e.g., the one or more processors of the database system 220 as illustrated in Figure 2 , specifically, the flow 400 can be executed by the data encoding system 1100 implemented in the database system 220 as illustrated in Figure 11 . As illustrated in Figure 4 , the flow 400 can include the following steps.

[0070] Step 410, obtaining instance data. In some embodiments, the step 410 can be executed by the obtaining module 1110.

[0071] With reference to the foregoing, the encoding can be bound to a write operation, accordingly, the instance data in the step 410 can be instance data targeted by the write operation.

[0072] In some embodiments, when a user wants to insert (add) an instance data in a data table, the user can initiate a data insertion request to the database system 220 through the user terminal 210, the data insertion request can include the instance data. Accordingly, the obtaining module 1110 can obtain the instance data to be inserted from the data insertion request.

[0073] In some embodiments, when a user wants to update an instance data in a data table, the user can initiate a data update request to the database system 220 through the user terminal 210, the data update request can include the value of the field to be modified (after modification) of the instance data. Accordingly, the obtaining module 1110 can obtain the instance data after modification according to the data insertion request and the instance data before modification. For the obtaining method of the instance data before modification, please refer to the decoding content (e.g., the flow 300 and related descriptions) in the present specification. Figure 8 and related descriptions.

[0074] In some embodiments, the instance data in the step 410 can be obtained by decoding, for example, by re-encoding the decoding before (for more details, please refer to Figure 9 and related descriptions).

[0075] Step 420, encoding the instance data based on the ontology definition data to obtain encoded data for storage. In some embodiments, the step 410 can be executed by the encoding module 1120.

[0076] The ontology definition data is used to define one or more fields and the data type of each field. The one or more fields are all the fields that constitute the instance data, for example, Figure 1The schema of the order data table shown can include field names and data types of the four fields that make up the order data.

[0077] The format (or encoding method) of the encoded data is described in detail below.

[0078] The encoded data can include a description section and a data section. The description section and the data section can be arranged adjacently, which means that the description section and the data section are stored contiguously in a storage area. It can be understood that the encoding (noun) of data can refer to the encoded data obtained by encoding the data.

[0079] The data section (denoted as [DATA]) is used only to store values of valid fields. Invalid fields, as opposed to valid fields, can include null fields and / or logically deleted fields. The schema can record whether each field is an invalid field, for example, the schema can indicate whether any field is a null field and / or indicate whether any field is a logically deleted field. In some embodiments, the values of the fields in the data section can be stored contiguously, and thus the location of a corresponding field in the data section can be determined by the size information of the field.

[0080] The encoding and decoding method provided in the specification provides support for null values, which can represent unknown values. Null fields (columns) can be used as reserved fields (columns). Taking a null column of integer type (denoted as int) as an example, when a user needs to insert (add) a column of known integer values (such as the number of goods) in a data table, the null value in the null column can be directly replaced with the known value, without the need to first update the schema of the data table (define the added column and its data type).

[0081] In some embodiments, when the user inserts instance data that lacks the value of a certain field that is not logically deleted nor defined as a null value in the schema, the missing value of the field can also be recorded as a null value. At this time, the field in the instance data can also be regarded as an invalid field. For example, the user needs to insert instance data “0004…file bag 12 yuan” in the order data table, where the order time is unknown, and the value of the “order time” field in the “0004” record can be set to a null value.

[0082] The description section is used to describe the attributes of the fields, such as the number of fields, data size (such as the number of bytes), data type (such as integer, floating point, string), validity (whether it is a logically deleted field, whether it is a null field), and the like.

[0083] It should be noted that the information used to describe the attributes of a field (e.g., data size information, data type information, validity information) can be organized in any form. For illustrative purposes, this specification mainly uses tables (such as size lists, type lists, bit tables) as examples to illustrate this information.

[0084] For valid fields, the description portion can use a size list (which may be denoted as [SIZE_ARR]) to describe the position of each valid field in the data area, so that the value of each valid field can be read from the data area. Each element of the size list (denoted as SIZE) can correspond one-to-one with a field of the instance data, and the element indicates the storage space occupied by the value of the corresponding field in the data area. The data size (e.g., number of bits) of each SIZE in [SIZE_ARR] can be equal, so that SIZEs in [SIZE_ARR] can be read one by one in a fixed data size (e.g., 4 bits). In practical applications, to be compatible with multiple (e.g., at least 3) data sizes, a SIZE typically occupies at least 2 bits.

[0085] by Figure 1 For example, in the encoding of order data, [SIZE_ARR] can include four SIZE values ​​that correspond one-to-one with the four fields of the order data. Each SIZE indicates the number of bytes occupied by the corresponding field value in the data area. Assuming each SIZE occupies 4 bits, [SIZE_ARR] = 0100100011111001, and the order of SIZE is consistent with the field order of the order data table, then [SIZE_ARR] can indicate: order number occupies 4 (0100) bytes, order time occupies 8 (1000) bytes, product name occupies 15 (1111) bytes, and product amount occupies 9 (1001) bytes.

[0086] In practice, the data size (such as the number of bytes) indicated by the element (SIZE) in the size list does not necessarily have to be equal to the value of that element (SIZE); a certain mapping (conversion) relationship between the two is sufficient. For example, Figure 5 As shown, when SIZE = 0b0000 (0b represents binary), it indicates that the value of the corresponding field occupies 1 byte in the data area; when SIZE = 0b0001, it indicates that the value of the corresponding field occupies 2 bytes in the data area; when SIZE = 0b0010, it indicates that the value of the corresponding field occupies 4 bytes in the data area; when SIZE = 0b0011, it indicates that the value of the corresponding field occupies 8 bytes in the data area; ...; when SIZE = 0b1111, it indicates that the value of the corresponding field occupies 0 bytes in the data area.

[0087] For the valid fields, the description part can describe data types of the valid fields by a type list (denoted as [TYPE_ARR]). Elements of the type list (denoted as TYPE) can correspond to the fields of the instance data one by one, and each element can indicate a data type of the corresponding field. The data size (such as the number of bits) of each TYPE in [TYPE_ARR] can be equal, so as to read the elements in the type list one by one according to a fixed data size (such as 2 bits). In actual application, in order to be compatible with multiple (for example, at least three) data types, TYPE usually occupies at least 2 bits.

[0088] In some embodiments, the data types described by the type list can be specific data types (which can be regarded as small categories). In some embodiments, the data types described by the type list can be summarized data types (which can be regarded as large categories). For example only, data types can be summarized into three categories: integer types (referred to as integer types, including int8, int32, int64 and the like small categories), floating point types (referred to as floating point types, including float, double and the like small categories) and string types (referred to as string types, including string, varchar, date, datetime and the like small categories).

[0089] For example only, as shown in Figure 6 when TYPE = 0b00, it indicates that the corresponding field is an integer type; when TYPE = 0b01, it indicates that the corresponding field is a floating point type; and when TYPE = 0b10, it indicates that the corresponding field is a string type.

[0090] In some embodiments, the data type can also reflect the data size or the storage space size of the numerical value, such as the storage space occupied by the numerical value of the data type int8 is 1 byte, and the storage space occupied by the numerical value of the data type float is 4 bytes. Therefore, in some embodiments, only by the type information of the field, the position of the corresponding field in the data area can be determined.

[0091] The data area only stores the values of valid fields can greatly improve the storage efficiency of the database, but further consideration is needed on how to mark the invalid fields so that the values of any field (whether valid or not) can be correctly parsed. The embodiments of the present specification provide two encoding ideas for marking invalid fields: Idea one, the description part can describe the distribution of invalid fields in all fields through a bit table (which can be denoted as [BIT_MAP]), wherein the elements (which can be denoted as BIT) of the bit table correspond to all fields in the ontology definition data one by one, and the element indicates whether the corresponding field is an invalid field (thus, each element occupies 1 bit). Idea two, the description part can describe the distribution of invalid fields through a size list and / or a type list, i.e., the size list and / or the type list are not limited to describing the data size and / or data type of the valid fields. At this time, the elements of the size list or the type list correspond to all fields in the ontology definition data one by one, and the element indicates the size of the storage space occupied by the value of the corresponding field in the data area or indicates the data type of the corresponding field. Further, in the size list, the element corresponding to the null value field (if any) is a preset value. For example, when SIZE = 0b1110, it indicates that the corresponding field is a null value field. In the type list, the element corresponding to the logically deleted field (if any) is a preset value. For example, when TYPE = 0b11, it indicates that the corresponding field is a logically deleted field. In order to further improve the storage efficiency of the database, when encoding, the encoding idea can be selected according to the number of invalid columns. The general principle is that the more invalid columns (such as the higher the proportion), the more inclined to choose idea one, i.e., the fewer invalid columns (such as the lower the proportion), the more inclined to choose idea two.

[0092] Based on the definition of invalid fields, the marking of invalid fields can be divided into three cases, and the two encoding ideas mentioned above will be discussed under each case.

[0093] Case one (invalid fields only include null value fields)

[0094] When the null value fields are so many that the first set condition is met, the description part can describe the distribution of null value fields in all fields through a null value field bit table (which can be denoted as [NULL_BIT_MAP]), otherwise, the description part can describe the distribution of null value fields through a size list and / or a type list.

[0095] The elements (which can be denoted as NULL_BIT) of the null value field bit table can correspond to all fields one by one, and the element indicates whether the corresponding field is a null value field, for example, when NULL_BIT = 1, it indicates that the corresponding field is a null value field; when NULL_BIT = 0, it indicates that the corresponding field is not a null value field. The first set condition can include that the proportion of null value fields in all fields reaches a first proportion threshold, for example, 30%, 50%, etc.

[0096] When the number of null fields is not large enough to satisfy the first setting condition, the elements (SIZE and / or TYPE) of the size list and / or the type list can be one-to-one corresponding to all fields, wherein the element corresponding to the null field (if exists) is a preset value. For example, referring to Figure 5 When SIZE = 0b1110, it indicates that the corresponding field is a null field. For another example, referring to Figure 5 and Figure 6 When SIZE = 0b1110 and TYPE ≠ 0b11, it indicates that the corresponding field is a null field.

[0097] For example only, assuming that SIZE occupies 2 bits and TYPE also occupies 2 bits, when there are 99 null fields in 100 fields: if the distribution of the null fields in all fields is described by the null field bit table, only 100*1 bit is consumed; if the distribution of the null fields is described by the size list and / or the type list, 99*2 bits or 99*4 bits are consumed. When there is only 1 null field in 100 fields: if the distribution of the null fields is described by the size list and / or the type list, only 1*2 bits or 1*4 bits are consumed; if the distribution of the null fields in all fields is described by the null field bit table, 100*1 bit is consumed. Through the two examples, it can be seen that when the proportion of the null fields is too high, it is a better choice to describe the distribution of the null fields by the null field bit table, otherwise, it is a better choice to describe the distribution of the null fields by the size list and / or the type list.

[0098] Case Two (Invalid fields only include logically deleted fields)

[0099] When the number of logically deleted fields is large enough to satisfy the first setting condition (which can be different from or the same as the first setting condition in Case One), the description part can describe the distribution of the logically deleted fields in all fields by the deleted field bit table (which can be denoted as [DEL_BIT_MAP]), otherwise, the description part can describe the distribution of the logically deleted fields by the size list and / or the type list.

[0100] The elements (which can be denoted as DEL_BIT) of the deleted field bit table can be one-to-one corresponding to all fields in the body definition data, and the element indicates whether the corresponding field is a logically deleted field, for example, when DEL_BIT = 1, it indicates that the corresponding field is a logically deleted field; when DEL_BIT = 0, it indicates that the corresponding field is not a logically deleted field. The second setting condition can include that the proportion of the logically deleted fields in all fields reaches a first proportion threshold, for example, 20%, 50%, etc.

[0101] When the number of logically deleted fields is not large enough to satisfy the first preset condition, the elements (SIZE and / or TYPE) of the size list and / or the type list can correspond to all the fields one by one, wherein the element corresponding to the logically deleted field (if any) is a preset value. For example, referring to Figure 6 When TYPE = 0b11, it indicates that the corresponding field is a logically deleted field. For another example, referring to Figure 5 and Figure 6 When SIZE = 0b1111 and TYPE = 0b11, it indicates that the corresponding field is a logically deleted field.

[0102] For example only, assuming that SIZE occupies 2 bits and TYPE also occupies 2 bits, when there are 99 logically deleted fields in 100 fields: if the logically deleted fields are described in the deletion field bit table, only 100*1 bit is consumed; if the logically deleted fields are described in the size list and / or the type list, 99*2 bits or 99*4 bits are consumed. When there is only 1 logically deleted field in 100 fields: if the logically deleted field is described in the size list and / or the type list, only 1*2 bits or 1*4 bits are consumed; if the logically deleted field is described in the deletion field bit table, 100*1 bit is consumed. Through the two examples, it can be seen that when the proportion of logically deleted fields is too high, it is a better choice to describe the distribution of logically deleted fields by the deletion field bit table, otherwise, it is a better choice to describe the distribution of logically deleted fields by the size list and / or the type list.

[0103] Case three (invalid fields include null fields and logically deleted fields)

[0104] When the invalid fields include more than two kinds, the bit table in the description part can include one or more bit tables corresponding to one or more specific invalid fields one by one, for example, in case three, the bit table in the description part can include a null field bit table and / or a deletion field bit table.

[0105] On the one hand, when the number of null fields is large enough to satisfy the second preset condition (which can be the same as the first preset condition in case one), the description part can describe the distribution of null fields in all fields by the null field bit table, otherwise, the description part can describe the distribution of null fields by the size list and / or the type list.

[0106] The elements (NULL_BIT) of the null field bit table can correspond to all the fields in the ontology definition data one by one, and the elements indicate whether the corresponding field is a null field. For example, when NULL_BIT = 1, it indicates that the corresponding field is a null field; when NULL_BIT = 0, it indicates that the corresponding field is not a null field. The second setting condition can include that the proportion of null fields in all fields reaches a second proportion threshold.

[0107] When the null fields are not too many to meet the second setting condition, the elements (SIZE and / or TYPE) of the size list and / or the type list can correspond to at least part of the fields (including valid fields) in the ontology definition data one by one, wherein the element corresponding to the null field (if any) is a preset value. For example, referring to Figure 5 When SIZE = 0b1110, it indicates that the corresponding field is a null field. For another example, in combination with reference to Figure 5 and Figure 6 When SIZE = 0b1110 and TYPE ≠ 0b11, it indicates that the corresponding field is a null field.

[0108] On the other hand, when the logically deleted fields are too many to meet the third setting condition (which can be the same as the first preset condition in case two), the description part can describe the distribution of the logically deleted fields in all fields by deleting the field bit table, otherwise, the description part can describe the distribution of the logically deleted fields by the size list and / or the type list.

[0109] The elements (DEL_BIT) of the deleted field bit table can correspond to all the fields one by one, and the elements indicate whether the corresponding field is a logically deleted field. For example, when DEL_BIT = 1, it indicates that the corresponding field is a logically deleted field; when DEL_BIT = 0, it indicates that the corresponding field is not a logically deleted field. The third setting condition can include that the proportion of logically deleted fields in all fields reaches a third proportion threshold.

[0110] When the logically deleted fields are not too many to meet the third setting condition, the elements (SIZE and / or TYPE) of the size list and / or the type list can correspond to at least part of the fields (including valid fields) in the ontology definition data one by one, wherein the element corresponding to the logically deleted field (if any) is a preset value. For example, referring to Figure 6 When TYPE = 0b11, it indicates that the corresponding field is a logically deleted field. For another example, in combination with reference to Figure 5 and Figure 6 When SIZE = 0b1111 and TYPE = 0b11, it indicates that the corresponding field is a logically deleted field.

[0111] For case three, the following situations can occur: 1. The ontology definition data contains a small number of logically deleted fields, but the instance data contains more null fields. In this case, the encoded data includes the null field bit table and does not include the deleted field bit table. In addition to the elements corresponding to the valid fields, the size list or type list also includes elements corresponding to the deleted logical fields. Only the values of the elements corresponding to the deleted logical fields in the size list or type list are preset values to distinguish them from the valid fields. 2. The instance data contains fewer null fields, but the ontology definition data contains more logically deleted fields. In this case, the encoded data includes the deleted field bit table and does not include the null field bit table. In addition to the elements corresponding to the valid fields, the size list or type list also includes elements corresponding to the null fields. Only the values of the elements corresponding to the null fields in the size list or type list are preset values to distinguish them from the valid fields. 3. Both the null fields and the logically deleted fields are more. In this case, the encoded data includes the deleted field bit table and the null field bit table. The size list or type list only contains elements corresponding to the valid fields.

[0112] In summary of the above cases, in some embodiments, when the number of invalid fields does not meet the first set condition: in the size list, the element corresponding to the invalid field can be (or indicate) 0 or null; in the type list, the element corresponding to the invalid field can be an invalid identifier (such as 0b11).

[0113] In some embodiments, when the value of a valid field is a special value (such as zero), the data area can not be stored during encoding. That is, the data area is only used to store the values of valid fields that are not special values (such as non-zero values). Further, the description part can mark the special values of the valid fields through the size list and / or type list. For example, in combination with reference to Figure 5 and Figure 6 When SIZE = 0b1111 and TYPE ≠ 0b11, it indicates that the value of the corresponding field (valid field) is zero. Of course, in [SIZE_ARR], the SIZE corresponding to the valid field with a zero value can also be equal to other preset values, such as 0b1110.

[0114] In some embodiments, in order to save storage space as much as possible, for the value of an effective field, the encoding module 1120 can select the minimum storage space size that can store the valid bits of the value from two or more storage space sizes corresponding to the element of the type list, as the storage space size of the value of the effective field in the data area. Among them, the two or more storage spaces corresponding to the element can refer to two or more storage space sizes corresponding to a certain summary data type, for example, int type can include int8, int32 and int64, that is, int type can correspond to 1 byte, 4 bytes and 8 bytes. Just as an example, when the data type of the effective field is int32, but the value of the effective field can be stored with 1 byte, the SIZE corresponding to the effective field can be set to 0b0000 according to the indication relationship shown in the following table. Figure 5

[0115] In some embodiments, the encoded data can also include the field number (which can be denoted as [CNT]) and / or the encoding identifier (which can be denoted as [VER]). Among them, the field number can be determined based on the total number of the one or more fields, for example, when the one or more fields in the ontology definition data are n fields, the field number can be equal to n. The encoding identifier can be used to distinguish different encoding methods (formats), that is, to indicate which way (such as 3 in case one or case three) the current encoded data is encoded. [CNT] and / or [VER] can be set to a fixed data size (such as bit number, byte number), for example, [VER] can occupy 4 bits, and [CNT] can occupy 2 bytes (allowing to create at most 65535 fields). Just as an example, referring to the foregoing embodiments, the encoding formats indicated by different encoding identifiers can be as shown in the following table. Figure 7 Figure 7 Among them, ROW represents the encoded data.

[0116] It should be understood that in alternative embodiments, it is not necessary to choose between the two encoding ideas described above, but to choose one as the unified encoding format of the example data.

[0117] For example, according to idea one, the encoded data can include: a bit table, the elements of which correspond one-to-one to the one or more fields (i.e. all fields in the ontology definition data), and the elements are used to indicate whether the corresponding field is an invalid field, further, the bit table can be a null field bit table, or a deleted field bit table, or both; a size list, the elements of which correspond one-to-one to the effective fields, and the elements are used to indicate the storage space size occupied by the value of the corresponding field in the data area; a type list, the elements of which correspond one-to-one to the effective fields, and the elements are used to indicate the data type of the corresponding field.

[0118] ​​For example, according to the second idea, the encoded data can include: a size list, elements of which correspond to the one or more fields one by one, and the elements are used to indicate the size of the storage space occupied by the value of the corresponding field (valid field) in the data area or indicate that the corresponding field is an invalid field; a type list, elements of which correspond to the one or more fields one by one, and the elements are used to indicate the data type of the corresponding field or indicate that the corresponding field is an invalid field.

[0119] Figure 8 is an exemplary flowchart of a data decoding method according to some embodiments of the present specification. The flow 800 can be executed by one or more processors (for example, Figure 2 one or more processors of the database system 220 shown), and specifically, the flow 800 can be executed by Figure 12 the data decoding system 1200 implemented in the database system 220 shown. As Figure 8 shown, the flow 800 can include the following steps.

[0120] Step 810, reading the encoded data from the storage area. In some embodiments, step 810 can be executed by the reading module 1210.

[0121] The encoded data can be obtained by the data encoding method as described in any embodiment of the present specification, and specific details can be found elsewhere in the present specification and will not be repeated here.

[0122] Step 820, obtaining the value of the field to be parsed according to the description part of the encoded data. In some embodiments, step 820 can be executed by the decoding module 1220.

[0123] According to the relevant content of the encoding, the decoding module 1220 can determine the situation of the field to be parsed according to the description part of the encoded data. For example, when the field to be parsed is a valid field, the decoding module 1220 can determine the position of the value of the field to be parsed in the data area according to the size list, and then the decoding module 1220 can obtain the value of the field to be parsed from the data area. For another example, the decoding module 1220 can determine that the field to be parsed is an invalid field or the value of the field to be parsed is a special value according to the description part of the encoded data (such as the bit table, the size list, the type list, etc.).

[0124] The data decoding process will be illustrated intuitively through specific examples below.

[0125] For example only, reference is made to Figure 7, the decoding module 1220 can first read [VER] and [CNT], i.e., determine the encoding format satisfied by the encoding data (ROW) and the total number of all fields in the ontology definition data (denoted as n). [VER] determines whether the description part of the ROW contains [NULL_BIT_MAP] and / or [DEL_BIT_MAP]. [CNT] determines the number of elements (DEL_BIT) of [NULL_BIT_MAP] / [DEL_BIT_MAP]. When the ROW does not contain [NULL_BIT_MAP] or [DEL_BIT_MAP], the number of elements (SIZE) of [SIZE_ARR] is equal to n, and the number of elements (TYPE) of [TYPE_ARR] is also equal to n. For reference, please refer to Figure 5 and Figure 6 When the ROW contains [NULL_BIT_MAP] but does not contain [DEL_BIT_MAP], the number of elements (SIZE) of [SIZE_ARR] is equal to n-j, and the number of elements (TYPE) of [TYPE_ARR] is also equal to n-j, where j represents the number of elements (NULL_BIT) of [NULL_BIT_MAP] with 1 (indicating that the corresponding field is a null field). When the ROW contains [DEL_BIT_MAP] but does not contain [NULL_BIT_MAP], the number of elements (SIZE) of [SIZE_ARR] is equal to n-k, and the number of elements (TYPE) of [TYPE_ARR] is also equal to n-k, where k represents the number of elements (DEL_BIT) of [DEL_BIT_MAP] with 1 (indicating that the corresponding field is a logically deleted field). When the ROW contains [NULL_BIT_MAP] and contains [DEL_BIT_MAP], the number of elements (SIZE) of [SIZE_ARR] is equal to n-j-k, and the number of elements (TYPE) of [TYPE_ARR] is also equal to n-j-k. After determining the number of elements of [NULL_BIT_MAP], [DEL_BIT_MAP], [SIZE_ARR], [TYPE_ARR] ([NULL_BIT_MAP] and / or [DEL_BIT_MAP] may not exist), the decoding module 1220 can read [NULL_BIT_MAP], [DEL_BIT_MAP], [SIZE_ARR], [TYPE_ARR] in turn. Further, the decoding module 1220 can obtain the value of any field according to [NULL_BIT_MAP], [DEL_BIT_MAP], [SIZE_ARR], [TYPE_ARR].

[0126] Decoding is the inverse process of encoding, and details about determining the value of the effective field, the deleted field, the null value field, and the effective 0 value field can be determined according to the aforementioned encoding principle, and will not be described here.

[0127] Step 830, write the value of the field to be parsed into a storage unit. In some embodiments, step 820 can be performed by the decoding module 1220.

[0128] In some embodiments, the storage space size of the storage unit can be determined based on the specific data type of the field to be parsed. Referring to the foregoing, when the description part of the encoded data (such as the type list) describes the aggregate data type of the field, the decoding module 1220 can select the minimum storage space size capable of storing the valid bits of the corresponding field value from the storage space sizes of two or more specific data types corresponding to the corresponding element in the type list, as the storage space size of the value of the effective field in the data area. Therefore, in the decoding stage, the specific data type of the field to be parsed needs to be determined from the ontology definition data, that is, the decoding module 1220 can decode the encoded data based on the ontology definition data. Specifically, the decoding module 1220 can write the value of the field to be parsed into a storage unit of a corresponding size according to the specific data type of the field to be parsed defined by the ontology definition data. For example, the corresponding element of the type list indicates that the data type of a certain field to be parsed is an integer, and the value of the field to be parsed may only occupy 1 byte of storage space in the data area in the encoded data, but the specific data type of the field in the ontology definition data is int64 under the integer type. Therefore, the decoding module 1220 writes the value of the field into a storage unit of 8 bytes. When the description part of the encoded data (such as the type list) describes the specific data type of the field, the specific data type of the field to be parsed can also be determined from the description part.

[0129] As can be seen from the foregoing embodiments, the user can modify the data table structure or the ontology definition data, specifically, logically delete the definition of some fields or columns, and accordingly, some embodiments of the present specification can select a more space-saving encoding mode to encode the instance data according to the number of logically deleted fields. However, when a large number of fields in the ontology definition data have been deleted, even if the encoding mode of the foregoing embodiments is used, storage space cannot be effectively saved. For example, 99 of the 100 fields have been logically deleted, and a 100-bit bit table is still needed to describe the distribution of the valid fields in the 100 fields in the encoded data. For this purpose, some embodiments of the present specification propose that when most of the fields in the ontology definition data are logically deleted, new ontology definition data can be generated, wherein the new ontology definition data is substantially the same as the old ontology definition data, except that the logically deleted fields in the old ontology definition data are removed. In some scenarios, the data table needs to strictly conform to the ontology definition data, and therefore, after the ontology definition data is updated, the encoded data stored in the data table needs to be scanned one by one to be re-encoded based on the new ontology definition data.

[0130] Figure 9 is an exemplary flowchart of an encoding data updating method according to some embodiments of the present specification. The flow 900 can be executed by one or more processors (for example, the one or more processors of the database system 220 shown in Figure 2 , specifically, can be executed by the encoding data updating system 1300 implemented in the database system 220 shown in Figure 13 . As shown in Figure 9 , the flow 900 can include the following steps.

[0131] Step 910, when the number of logically deleted fields in the ontology definition data meets a fourth set condition, new ontology definition data is generated. In some embodiments, step 910 can be executed by the generation module 1310.

[0132] The new ontology definition data is obtained by removing the logically deleted fields from the old ontology definition data. Specifically, the generation module X10 can remove the information of the logically deleted fields, such as the definition, data type, state, etc. of the logically deleted fields, from the old ontology definition data to obtain the new ontology definition data.

[0133] It can be understood that the fields in the ontology definition data are the fields defined by the ontology definition data, i.e., the constituent fields of the instance data. When too many fields defined by the ontology definition data are logically deleted, the ontology definition data and the encoded data are filled with a large amount of useless information (or "garbage information"), which wastes a lot of time in encoding and decoding, thereby affecting the I / O (read / write) performance of the system. Therefore, it is necessary to remove these garbage information (which can be called garbage collection) to ensure the I / O performance of the system.

[0134] In some embodiments, the fourth condition can include that the proportion of the logically deleted fields in all fields reaches a fourth proportion threshold. In some embodiments, the fourth condition can include that the proportion of the logically deleted fields in the non-empty value fields reaches a fourth proportion threshold.

[0135] For the stored encoded data, the re-encoding module 1320 can perform steps 920 and 930 to obtain and store new encoded data. The stored encoded data is obtained by encoding the instance data based on the old ontology definition data according to the adaptive encoding method, where the adaptive encoding method refers to the encoding method provided in the foregoing embodiments for selecting between the two encoding ideas.

[0136] In step 920, the encoded data is decoded to obtain decoded data.

[0137] For more details about decoding, please refer to Figure 8 and the related descriptions.

[0138] In step 930, the decoded data is encoded based on the new ontology definition data to obtain and store new encoded data.

[0139] The re-encoding module 1130 can perform step 930 according to the adaptive encoding method.

[0140] In some embodiments, the database system 220 can also delete the old ontology definition data. Before deleting the old ontology definition data, when receiving a read request for the encoded data, for example, when a user wants to query the value of a specified field in a piece of instance data, the database system 220 can refuse to respond. Before deleting the old ontology definition data, when receiving a write request (for example, a data insertion request or a data update request) for the instance data, the database system 220 directly encodes the instance data based on the new ontology definition data to obtain and store new encoded data. Here, the deletion can refer to logical deletion or physical deletion. The logically deleted old ontology definition data can be eventually physically deleted. It can be understood that physically deleting the old ontology definition data is beneficial to saving storage space.

[0141] Figure 10is an exemplary flowchart of a method of updating encoded data according to yet some embodiments of the present specification. The flow 1000 can be performed by the server cluster 300 as shown in Figure 3 The flow 900 can include the following steps as shown in Figure 9

[0142] At step 1010, a new ontology definition data is generated by the central server when the number of fields logically deleted in the ontology definition data meets a fourth setting condition.

[0143] Referring to Figure 3 and the related description, the central server is responsible for managing the ontology definition data, and thus the ontology definition data can be updated by the central server. For implementation details of step 1010, reference can be made to the detailed description of step 910.

[0144] At step 1020, the encoded data in the local copy of each data server is updated in batches.

[0145] The encoded data in the local copy is encoded from the instance data according to the adaptive encoding method.

[0146] For example only, each batch can update the encoded data in the local copy of one data server, and when the data server of the current batch completes the update of the encoded data, the data server of the next batch can start the update of the encoded data, and so on, until each data server completes the update of the encoded data.

[0147] Through the update in batches, it can be ensured that there is always a data server capable of providing data services (such as responding to I / O requests from the user end) to the outside.

[0148] For more details of step 1020, reference can be made to the detailed description of steps 920 and 930. That is, for each data server, the update can include steps 920 and 930.

[0149] At step 1030, the old ontology definition data is deleted when each data server updates the encoded data based on the new ontology definition data.

[0150] When each data server updates the encoded data based on the new ontology definition data, the central server and the data servers can delete the old ontology definition data. For more details of deleting the old ontology definition data, reference can be made to the related content of the flow 900.

[0151] In some embodiments, different read-write strategies can be adopted for data servers in different update states.

[0152] ​In particular, for a data server (denoted as S_A) that is updating the encoded data based on the new ontology definition data, when receiving a read request for the encoded data, S_A can reject the response. When receiving a write request for the instance data, S_A can encode the instance data based on the new ontology definition data according to the adaptive encoding method, and store. A The instance data can be encoded based on the new ontology definition data according to the adaptive encoding method, and stored.

[0153] For a data server (denoted as S_B) that has not updated the encoded data based on the new ontology definition data, when receiving a read request for the encoded data, S_B can decode the encoded data based on the old ontology definition data. When receiving a write request for the instance data, S_B can encode the instance data based on the new ontology definition data according to the adaptive encoding method, and store.

[0154] For a data server (denoted as S_C) that has updated the encoded data based on the new ontology definition data, when receiving a read request for the encoded data, S_C can decode the encoded data based on the new ontology definition data. When receiving a write request for the instance data, S_C can encode the instance data based on the new ontology definition data according to the adaptive encoding method, and store.

[0155] It should be noted that the above description of the flow is merely for example and illustration, and does not limit the scope of the present specification. Various modifications and changes can be made to the flow under the guidance of the present specification by those skilled in the art. However, these modifications and changes are still within the scope of the present specification.

[0156] Figure 11 is an exemplary block diagram of an encoding system according to some embodiments of the present specification. The system 1100 can be implemented in a data server, such as the data server 210 shown in Figure 2 .

[0157] As shown in Figure 11 , the system 1100 can include an obtaining module 1110 and an encoding module 1120.

[0158] The obtaining module 1110 can be configured to obtain instance data.

[0159] The encoding module 1120 can be configured to encode the instance data based on ontology definition data to obtain encoded data for storage.

[0160] For more details about the system 1100 and its modules, reference can be made to Figure 4 and related descriptions.

[0161] Figure 12 is an exemplary block diagram of a decoding system according to some embodiments of the present specification. The system 1200 can be implemented in a data server, such as the data server 210 shown in Figure 2 .

[0157] As shown in Figure 11 , the system 1200 can include a decoding module 1210 and a storing module 1220.

[0158] The decoding module 1210 can be configured to decode encoded data based on ontology definition data to obtain instance data.

[0159] The storing module 1220 can be configured to store the instance data.

[0160] For more details about the system 1200 and its modules, reference can be made to Figure 4 and related descriptions.Figure 2 implemented in the database system 220 as shown.

[0162] As shown in FIG. 12, the system 1200 can include a reading module 1210 and a decoding module 1220. Figure 12

[0163] The reading module 1210 can be configured to read the encoded data from the storage area. The decoding module 1220 can be configured to decode the encoded data based on the ontology definition data, which can be further configured to: obtain the value of the field from the data area based on the description part of the encoded data; and write the value to the storage unit, the storage space size of the storage unit can be determined based on the data type of the field in the ontology definition data.

[0164] For more details of the system 1200 and its modules, please refer to Figure 8 and the related descriptions.

[0165] Figure 13 is an exemplary module diagram of an encoded data updating system according to some embodiments of the present specification. The system 1300 can be implemented in the database system 220 as shown. Figure 2 implemented in the database system 220 as shown.

[0166] As shown in FIG. 13, the system 1300 can include a generating module 1310 and a re-encoding module 1320. Figure 13 The generating module 1310 can be configured to generate new ontology definition data when the number of fields that are logically deleted in the ontology definition data meets a fourth set condition.

[0167] The re-encoding module 1320 can be configured to, for the stored encoded data, decode the encoded data based on the old ontology definition data, and encode the obtained decoded data based on the new ontology definition data to obtain new encoded data and store.

[0168]

[0169] For more details of the system 1300 and its modules, please refer to Figure 9 and the related descriptions.

[0170] It should be understood that, Figures 11-13 ​​The illustrated system and its modules can be implemented in various ways. For instance, in some embodiments, the system and its modules can be implemented in hardware, software, or a combination of software and hardware. The hardware portion can be implemented with special logic, while the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art can understand that the above-described method and system can be implemented using computer-executable instructions and / or included in processor control code, such as provided on a carrier medium, such as a disk, CD or DVD-ROM, programmable memory, such as read-only memory (firmware), or data carrier, such as an optical or electronic signal carrier. The system and its modules of the present specification can be implemented not only in hardware circuitry, such as very large scale integrated circuits or gate arrays, semiconductors, such as logic chips, transistors, or programmable hardware devices, such as field programmable gate arrays, programmable logic devices, etc., but also in software, for example, executed by various types of processors, and also by a combination of the above hardware circuitry and software (e.g., firmware).

[0171] It should be noted that the above description of the system and its modules is for convenience of description only, and does not limit the present specification to the scope of the embodiments. It can be understood that, for those skilled in the art, after understanding the principles of the system, various modules can be combined or connected to form subsystems without departing from the principles. For example, in some embodiments, the acquisition module 1110 and the encoding module 1120 can be different modules in a system, or they can be a module that implements the functions of the two modules. Variations such as this are within the scope of protection of the present specification.

[0172] The beneficial effects that the embodiments of the present specification can bring include but are not limited to: (1) storing only the values of valid fields in the data area of the encoded data, and marking the invalid columns in the description part of the encoded data, which can improve the storage efficiency of the database on the premise of ensuring the integrity of the data content, and the higher storage efficiency also means better I / O performance; (2) adding support for null fields in database encoding and decoding, which can meet more user needs; (3) adaptively adjusting the encoding format (i.e., whether to enable a bit table to mark invalid fields), which can further improve storage efficiency and optimize I / O performance; (4) when the value of the valid field is a special value (such as zero), the data area can not be stored during encoding, and the special value of the valid field is marked in the description part of the encoded data, which can further improve storage efficiency and optimize I / O performance on the premise of ensuring the integrity of the data content; (5) further saving storage space and optimizing I / O performance through garbage collection; (6) further providing a garbage collection mechanism for a multi-copy architecture. It should be noted that different embodiments can have different beneficial effects, and in different embodiments, the beneficial effects that can be produced can be any one or a combination of the above, or any other beneficial effects that can be obtained.

[0173] The above has described the basic concepts, and it is obvious that the above detailed disclosure is only as an example and does not constitute a limitation on the embodiments of the present specification. Although it is not explicitly stated here, those skilled in the art can make various modifications, improvements and corrections to the embodiments of the present specification. Such modifications, improvements and corrections are suggested in the embodiments of the present specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of the present specification.

[0174] At the same time, specific words are used in the present specification to describe the embodiments of the present specification. As "one embodiment", "an embodiment" and / or "some embodiments" mean a certain feature, structure or characteristic related to at least one embodiment of the present specification. Therefore, it should be emphasized and noted that the "an embodiment" or "one embodiment" or "one alternative embodiment" mentioned in the present specification twice or more in different positions does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of the present specification can be properly combined.

[0175] Moreover, those skilled in the art will appreciate that the various aspects of the disclosure can be illustrated and described in connection with a number of various kinds of systems and / or methods, including any new and useful processes, machines, manufacture, or compositions of matter, or any new and useful improvements thereof, as described and claimed. Accordingly, the various aspects of the disclosure can be embodied in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on hardware, which can all be generically referred to as a "circuit," "module," "engine," "unit," "component," or "system." Each and every aspect of the disclosure can be realized by a "data block," which can be implemented in hardware and / or in software (including firmware, resident software, micro-code, etc.) that runs on hardware. These data blocks can be combined with other data blocks to form larger data blocks or systems. Furthermore, aspects of the disclosure can be implemented as a computer program product, which can include a computer-readable medium having stored computer program codes that can be executed by one or more computer processors.

[0176] Computer storage media can include a propagated data signal with the computer program code embodied therein, e.g., in baseband or as part of a carrier wave. Such a propagated signal can take any of a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. Computer storage media can be any media that can be accessed by a computer to provide the computer program code to the computer processor for execution. The computer storage media can be any available media that can be accessed by a computer, including magnetic and optical storage media such as compact disc (CD) and / or digital versatile disc (DVD) that can be located either inside of or outside of a computer processing unit. The computer storage media can be any media that can be accessed by a computer to provide the computer program code to the computer processor for execution, including any suitable type of memory, including volatile memory (e.g., RAM), non-volatile memory (e.g., ROM, flash, solid state disk, etc.), or any suitable type of storage device including a disk drive, diskette, tape, or any other storage device.

[0177] The computer program code for carrying out the operations of the aspects of the disclosure can be written in any one or more of a variety of programming languages, including an object-oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, and the like, conventional procedural programming languages, such as the "C" programming language, Visual Basic, Fortran 2003, Perl, COBOL 2002, PHP, ABAP, dynamic programming languages such as Python, Ruby, and Groovy, or other programming languages. The program code can execute entirely on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic program code can be downloaded from an on-demand computing platform, such as Amazon Web Services, Microsoft Azure, or Google Cloud Platform, or a similar on-demand computing platform.

[0178] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in the embodiments of this specification are not intended to limit the order of the processes and methods of the embodiments of this specification. Although some inventive embodiments that are currently considered useful have been discussed by way of various examples in the foregoing disclosure, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments. Rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented solely by software solutions, such as installing the described system on existing processing devices or mobile devices.

[0179] Similarly, it should be noted that, in order to simplify the description of the embodiments disclosed in this specification and thereby aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the objects of the embodiments in this specification require more features than those mentioned in the claims. In fact, the embodiments have fewer features than all the features of the single embodiments disclosed above.

[0180] For each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, and documents, referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with this specification, as well as documents that limit the broadest scope of the claims of this application (currently or subsequently appended to this application). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0181] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of the embodiments described herein. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A data encoding method performed by one or more processors, comprising: obtaining instance data; encoding the instance data based on ontology definition data to obtain encoded data for storage; the ontology definition data is used to define one or more fields and data types of the fields; the encoded data comprises a description part and a data area, wherein a number of elements of the data area is related to a number of valid fields, and each element is a value of a corresponding valid field, the value being obtained from the instance data; wherein an invalid field comprises a null value field and / or a logically deleted field in the ontology definition data; when the number of invalid fields meets a first preset condition, the description part describes a distribution of the valid fields in the one or more fields by validity information of the one or more fields, and describes positions of the values of the valid fields in the data area by size information and / or type information of the valid fields; otherwise, the description part describes the distribution of the valid fields in the one or more fields by size information and / or type information of the one or more fields, and describes the positions of the values of the valid fields in the data area. 2.The method of claim 1, wherein when the number of invalid fields does not meet the first preset condition, the description part comprises a size list and a type list, wherein: elements of the size list correspond to the one or more fields one by one, and each element indicates a storage space size of a value of a corresponding field in the data area; elements of the type list correspond to the one or more fields one by one, and each element indicates a data type of a corresponding field; when the number of invalid fields meets the first preset condition, the description part comprises a bit table, a size list and a type list, wherein: elements of the bit table correspond to the one or more fields one by one, and each element indicates whether a corresponding field is valid; the size list comprises elements corresponding to the valid fields, and each element indicates a storage space size of a value of a corresponding field in the data area; the type list comprises elements corresponding to the valid fields, and each element indicates a data type of a corresponding field.

3. The method of claim 2, wherein, when a number of logically deleted fields in the ontology definition data meets a second preset condition, the bit table comprises a deleted field bit table, elements of the deleted field bit table correspond to the one or more fields one by one, and each element indicates whether a corresponding field is logically deleted; when a number of null value fields meets a third preset condition, the bit table comprises a null value field bit table, elements of the null value field bit table correspond to the one or more fields one by one, and each element indicates whether a value of a corresponding field is a null value.

4. The method of claim 2, wherein, when the number of invalid fields does not meet the first preset condition: in the size list, elements corresponding to the invalid fields are 0 or null values.

5. The method of claim 2, wherein, in the size list, an element corresponding to a valid field with a zero value is a preset value; the data area is used to store values of valid fields with non-zero values.

6. The method of claim 2, wherein, elements of the type list are integer type identifiers, character type identifiers, floating point type identifiers or invalid identifiers; when the number of invalid fields does not meet the first preset condition: in the type list, elements corresponding to the invalid fields are invalid identifiers.

7. The method of claim 6, wherein, In order to determine the storage space size of the value of the effective field in the data area, the encoding of the instance data based on the ontology definition data obtains encoded data, comprising: selecting, based on the value of the effective field, the minimum storage space size capable of storing the valid bits of the value from two or more storage space sizes corresponding to the elements in the type list, as the storage space size of the value of the effective field in the data area.

8. The method of claim 1, wherein, The encoded data further comprises a field number and an encoding identifier; the field number is determined based on the total number of the one or more fields, and the encoding identifier is used to distinguish different encoding modes of the encoded data.

9. A data encoding system, comprising: an acquisition module configured to acquire instance data; an encoding module configured to encode the instance data based on ontology definition data to obtain encoded data for storage; the ontology definition data is used to define one or more fields and the data type of each field; the encoded data comprises a description part and a data area, wherein the number of elements of the data area is related to the number of effective fields, and the elements are values corresponding to the effective fields, which are obtained from the instance data; wherein the invalid fields include null fields and / or fields logically deleted in the ontology definition data; when the number of invalid fields meets a first set condition, the description part describes the distribution of the effective fields in the one or more fields through the validity information of the one or more fields, and describes the position of the values of the effective fields in the data area through the size information and / or type information of the effective fields; otherwise, the description part describes the distribution of the effective fields in the one or more fields through the size information and / or type information of the one or more fields, and describes the position of the values of the effective fields in the data area.

10. A data encoding method performed by one or more processors, comprising: acquiring instance data; encoding the instance data based on ontology definition data to obtain encoded data for storage; the ontology definition data is used to define one or more fields and the data type of each field; wherein the encoded data comprises: a size list, the elements of which correspond one-to-one to the one or more fields, and the elements indicate the storage space size of the values of the corresponding fields in the data area; a type list, the elements of which correspond one-to-one to the one or more fields, and the elements indicate the data type of the corresponding fields; a data area, the number of elements of which is related to the number of effective fields, and the elements are values corresponding to the effective fields, which are obtained from the instance data; wherein the invalid fields include null fields and / or fields logically deleted in the ontology definition data.

11. A data encoding method performed by one or more processors, comprising: acquiring instance data; encoding the instance data based on ontology definition data to obtain encoded data for storage; the ontology definition data is used to define one or more fields and the data type of each field; wherein the encoded data comprises: a bit table, the elements of which correspond one-to-one to the one or more fields, and the elements are used to indicate whether the corresponding fields are effective. a size list including elements corresponding to the valid fields, and each element is used to indicate the size of the storage space of the value of the corresponding field in the data area; a type list including elements corresponding to the valid fields, and each element is used to indicate the data type of the corresponding field; a data area including elements corresponding to the valid fields, and each element is the value of the corresponding valid field, which is obtained from the instance data; wherein the invalid fields include the null value fields and / or the logically deleted fields in the ontology definition data.

12. The method of claim 11, wherein, the bit table includes a deleted field bit table and / or a null value field bit table; wherein the elements of the deleted field bit table correspond to the one or more fields one by one, and each element indicates whether the corresponding field is logically deleted; the elements of the null value field bit table correspond to the one or more fields one by one, and each element is used to indicate whether the value of the corresponding field is null.

13. A data encoding apparatus, wherein, a processor and a storage device, wherein the storage device is used to store instructions, and when the processor executes the instructions, the method of any one of claims 1-8, 10-12 is implemented.

14. A data decoding method, executed by one or more processors, comprising: reading encoded data from a storage area; the encoded data is obtained based on the method of any one of claims 1-8, 10-12; decoding the encoded data based on the ontology definition data, which further comprises: obtaining the value of the corresponding field from the data area based on the description part of the encoded data; writing the value into a storage unit; the storage space size of the storage unit is determined based on the data type of the field in the ontology definition data.

15. A data decoding system, comprising: a reading module, configured to read encoded data from a storage area; the encoded data is obtained based on the method of any one of claims 1-8, 10-12; a decoding module, configured to decode the encoded data based on the ontology definition data, which is further configured to: obtain the value of the field from the data area based on the description part of the encoded data; write the value into a storage unit; the storage space size of the storage unit is determined based on the data type of the field in the ontology definition data.

16. A data decoding apparatus, wherein, a processor and a storage device, wherein the storage device is used to store instructions, and when the processor executes the instructions, the method of claim 14 is implemented.

17. A method for updating encoded data, executed by one or more processors, comprising: generating new ontology definition data when the number of logically deleted fields in the ontology definition data meets a fourth set condition; the ontology definition data is used to define one or more fields and the data type of each field; the new ontology definition data is obtained by removing the logically deleted fields from the old ontology definition data; for the stored encoded data, wherein the stored encoded data is obtained by encoding the instance data based on the old ontology definition data according to the method of any one of claims 1-8: decoding the encoded data based on the old ontology definition data to obtain decoded data; encoding the decoded data according to the method of any one of claims 1-8 based on the new ontology definition data, to obtain new encoded data and store.

18. The method of claim 17, wherein, Further comprising deleting the old ontology definition data; and further comprising, before deleting the old ontology definition data: when receiving an encoded data read request, refusing to respond; when receiving an instance data write request, encoding the instance data corresponding to the request according to the method of any one of claims 1-8 based on the new ontology definition data, and storing.

19. An encoded data updating system, comprising: a generation module configured to generate new ontology definition data when the number of logically deleted fields in the ontology definition data meets a fourth set condition; the ontology definition data is used to define one or more fields and the data type of each field; the new ontology definition data is obtained by removing the logically deleted fields from the old ontology definition data; a re-encoding module configured to, for the stored encoded data, wherein the stored encoded data is obtained by encoding instance data according to the method of any one of claims 1-8 based on the old ontology definition data: decoding the encoded data based on the old ontology definition data, to obtain decoded data; encoding the decoded data according to the method of any one of claims 1-8 based on the new ontology definition data, to obtain new encoded data and store.

20. An encoded data updating apparatus, wherein, a processor and a storage device, the storage device being configured to store instructions, when the processor executes the instructions, the method of claim 17 or 18 is implemented.

21. An encoded data updating method, performed by a database cluster, the database cluster comprising a center server and a plurality of data servers, the plurality of data servers respectively storing a copy of the same encoded data, the method comprising: generating new ontology definition data by the center server when the number of logically deleted fields in the ontology definition data meets a fourth set condition; the ontology definition data is used to define one or more fields and the data type of each field; the new ontology definition data is obtained by removing the logically deleted fields from the old ontology definition data; updating the encoded data in the local copy by the plurality of data servers in batches, wherein the encoded data in the local copy is obtained by encoding instance data according to the method of any one of claims 1-8 based on the old ontology definition data: decoding the encoded data based on the old ontology definition data, to obtain decoded data; encoding the decoded data according to the method of any one of claims 1-8 based on the new ontology definition data, to obtain new encoded data and store; deleting the old ontology definition data after each data server updates the encoded data based on the new ontology definition data.

22. The method of claim 21, wherein, for the data server that is updating the encoded data based on the new ontology definition data: when receiving an encoded data read request, refusing to respond; When receiving an instance data write request, the instance data corresponding to the request is encoded based on the new ontology definition data according to the method in any one of claims 1-8, and stored; For the data server that has not updated the encoded data based on the new ontology definition data: When receiving an encoded data read request, the encoded data corresponding to the request is decoded based on the old ontology definition data; When receiving an instance data write request, the instance data corresponding to the request is encoded based on the new ontology definition data according to the method in any one of claims 1-8, and stored; For the data server that has updated the encoded data based on the new ontology definition data: When receiving an encoded data read request, the encoded data corresponding to the request is decoded based on the new ontology definition data; When receiving an instance data write request, the instance data corresponding to the request is encoded based on the new ontology definition data according to the method in any one of claims 1-8, and stored.

Images