Data processing method and device

Abstract

The invention provides a data processing method and device, and relates to the technical field of cloud computing, and the method comprises the steps: receiving a first data writing request, the first data writing request comprises to-be-written first data, and the first data comprises a first key, first description information and first attribute information; according to the first data writing request, first data to be written is written into the key value database, an index record is generated, the index record comprises an index of first description information and an index of first attribute information, the index of the first description information is used for indexing the first description information based on the first key, and the index of the first attribute information is used for indexing the first attribute information based on the first key. The index of the first attribute information is used for indexing the first attribute information based on the first key. The index record comprises the index of the description information and the index of the attribute information, so that the user can independently access the description information or the attribute information, and the access overhead is reduced on the basis of supporting independent access to the attribute information.

Description

[0001] This application claims priority to a Chinese patent application filed with the Chinese Patent Office on June 26, 2023, with application number 202310761932.7 and application name “A method, device and other equipment for storing data”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and more specifically, to a data processing method and device. Background Art

[0003] An application system file can be divided into content files and description files. The organizational model of content files is "full file path + content data"; the organizational model of description files is "full file path + description information + attribute information". When using cloud services to store application system files, you can choose a key-value database to store description files. For example, you can choose an unstructured key-value database or a semi-structured key-value database to store description files.

[0004] In the description file, the description information and the attribute information often have independent update requirements. In other words, when the user wants to update the description information, it is often not necessary to update the attribute information. When the user wants to update the attribute information, it is often not desirable to update the description information.

[0005] Currently, in an unstructured key-value database or a semi-structured key-value database, when a user wants to access attribute information separately, he needs to access the description information and the attribute information at the same time first, and then find the required attribute information, which results in high access overhead.

[0006] Therefore, reducing access overhead while supporting independent access to attribute information is an urgent problem to be solved. Summary of the invention

[0007] The present application provides a data processing method and device, which can reduce access overhead while supporting independent access to attribute information.

[0008] In a first aspect, a data processing method is provided, the method being applied to a key-value database, the key-value database being used to store a description file in the form of a key-value pair, the description file comprising a key, description information and attribute information, the method comprising: receiving a first data write request, wherein the first data write request comprises first data to be written, the first data comprising a first key, first description information and first attribute information; according to the first data write request, writing the first data to be written into the key-value database, and generating an index record, wherein the index record comprises an index of the first description information and an index of the first attribute information, wherein the index of the first description information is used to index the first description information based on the first key, and the index of the first attribute information is used to index the first attribute information based on the first key.

[0009] Based on the above scheme, since the index record includes the index of the description information and the index of the attribute information, the user can access the description information or the attribute information separately, without having to access the description information and the attribute information as a whole, thereby reducing the access overhead while supporting independent access to the attribute information. On the other hand, the embodiment of the present application only needs one key-value storage file. Compared with the "two key-value" scheme, the data processing method provided by the embodiment of the present application can reduce the occupied storage space.

[0010] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: receiving a data read request, wherein the data read request includes the first key; obtaining the index record based on the first key in the data read request; and reading a first target value based on the index record, wherein the first target value includes the first descriptive information and / or the first attribute information.

[0011] Based on the above scheme, the description information or attribute information can be read separately according to the index record, or the description information and attribute information can be read at the same time. Compared with the scheme of only reading the description information and attribute information first and then extracting the required target value from the description information and attribute information, this application can reduce the reading overhead.

[0012] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: receiving a data update request, wherein the data update request includes the first key; updating a second target value based on the data update request, wherein the second target value includes the first description information and / or the first attribute information; and updating the index record based on the updated second target value.

[0013] Based on the above scheme, the description information or attribute information can be updated separately according to the index record. Compared with the scheme of only accessing the description information and attribute information first and then updating the target value from the description information and attribute information, this application can reduce the update overhead and improve the update speed.

[0014] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: receiving a data deletion request, the data deletion request including the first key; and deleting the index record according to the first key in the data deletion request.

[0015] In combination with the first aspect, in certain implementations of the first aspect, the method also includes: receiving a second data write request, wherein the second data write request includes second data to be written, and the second data includes the second key, second description information, and second attribute information; when the number of bytes of the second description information is less than or equal to a first threshold, writing the second description information to the index of the second description information; or when the number of bytes of the second attribute information is less than or equal to a second threshold, writing the second attribute information to the index of the second attribute information.

[0016] Based on the above scheme, when the description information is small, the description information can be written directly into the index record. On the one hand, since the number of bytes of the description information is small, it may even be smaller than the number of bytes occupied by the storage address of the description information. Compared with the scheme of writing the description information and its storage address in the database, only writing the description information into the index record in the database can save storage space and increase the storage speed. On the other hand, when accessing the description information, you can directly "access in one step" in the index record. Compared with the "two-step access" scheme of first obtaining the storage address of the description information and then accessing it according to the storage address, the data processing method provided by this application reduces one access from two accesses, improves efficiency by 50%, and reduces access delays. The above analysis is also applicable to attribute information, which will not be repeated here.

[0017] In a second aspect, an embodiment of the present application provides a data processing device, which includes a module for implementing the first aspect or any possible implementation method of the first aspect.

[0018] In a third aspect, a computing device cluster is provided, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the data processing method of the first aspect or any possible implementation of the first aspect.

[0019] In a fourth aspect, a computer program product including instructions is provided. When the instructions are executed by a computing device cluster, the computing device cluster executes the data processing method of the first aspect or any possible implementation of the first aspect.

[0020] In a fifth aspect, a computer-readable storage medium is provided, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the data processing method of the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a system diagram based on a cloud server system according to an embodiment of the present application.

[0022] Figure 2 It is a schematic flow chart of a data processing method provided in an embodiment of the present application.

[0023] Figure 3 It is a schematic block diagram of a data processing system provided in an embodiment of the present application.

[0024] Figure 4 It is a schematic diagram of a data processing method provided in an embodiment of the present application.

[0025] Figure 5 It is a schematic diagram of data update in two situations provided in the embodiments of the present application.

[0026] Figure 6 It is a schematic diagram of two data updating methods provided in the embodiments of the present application.

[0027] Figure 7 It is a schematic block diagram of a data processing device provided in an embodiment of the present application.

[0028] Figure 8 An architectural diagram of a computing device provided in an embodiment of the application.

[0029] Fig. 9 It is a schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application.

[0030] Fig.10 It is a schematic diagram of a connection between two computing devices via a network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0032] The present application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0033] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0034] The business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0035] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0036] Figure 1 It is a system diagram based on a cloud server system according to an embodiment of the present application.

[0037] like Figure 1 As shown, the cloud management platform 110 is used to manage the infrastructure that provides multiple cloud services, and the infrastructure includes multiple cloud data centers, each of which includes multiple cloud servers, and each cloud server includes cloud service resources to provide corresponding cloud services for tenants. For example, a cloud server can store user application system files, and a cloud service resource can be a cloud database.

[0038] The cloud management platform 110 provides an access interface (such as an application program interface (API)). The tenant can operate the client remote access access interface to register a cloud account and password on the cloud management platform and log in to the cloud management platform. After the cloud management platform successfully authenticates the cloud account and password, the tenant can further pay to select and purchase a virtual machine with specific specifications (processor, memory, disk) on the cloud management platform. After the payment is successful, the cloud management platform provides the remote login account and password of the purchased virtual machine. The client can remotely log in to the virtual machine and install and run the tenant's application in the virtual machine. Tenants of cloud services can be individuals, enterprises, schools, hospitals, administrative agencies, etc.

[0039] The functions of the cloud management platform 110 include, but are not limited to, user console, computing management service, network management service, storage management service, authentication service, and image management service. The user console provides an API to interact with tenants, the computing management service is used to manage servers running virtual machines and containers and bare metal servers, the network management service is used to manage network services (such as gateways, firewalls, etc.), the storage management service is used to manage storage services (such as data bucket services), the authentication service is used to manage tenant accounts and passwords, and the image management service is used to manage virtual machine images. Tenants can use the client 130 to log in to the cloud management platform 110 through the Internet 120 to manage the rented cloud services.

[0040] An application system file can be divided into a content file and a description file. The organizational model of the content file is "full file path + content data"; the organizational model of the description file is "full file path + description information + attribute information". The size of the content file is usually in the megabyte (MB) level, and the size of the description file is usually in the kilobyte (KB) level. For example, a user can write a content file and a description file to a cloud server through the cloud management platform 110.

[0041] Among them, the full path of the file can also be called the absolute path, which refers to the logical location of the application system file on the disk. For example, "E:\ABC\D" can be a full file path, indicating that the logical location of file D is in the E drive, ABC folder. The content data can be the content of the file. For example, the D file is a patent application, and the text, drawings and other data in the patent application are the content data of the D file. The description information can be customized by the user, and the description information can facilitate the user to manage files. For example, the user can label the D file as "patent document", and for another example, the user can mark the D file as "important". The aforementioned "patent document" and "important" can be understood as the description information of the D file. Attribute information is used to reflect the attributes of the file. For example, data such as the size, creation date, author or permissions of the D file can be used as attribute information of the D file.

[0042] It should be noted that the present application does not limit the description information to only include user-defined tags, but may also include other information. For example, the description information may include summary information, etc.

[0043] It is understandable that if the full file path of the content file and the description file is the same, the content file and the description file as a whole can be regarded as an "application system file". For the sake of simplicity, when referring to the content file and the description file below, unless otherwise specified, the content file and the description file can be regarded as belonging to one application system file.

[0044] When using cloud services to store application system files, content files and description files often use different storage systems. For content files, since the size of content files is usually in the MB level, the total data volume is large, and the access to content files is bandwidth-oriented, requiring a high bandwidth per second, an object storage system is selected. For description files, since the size of description files is usually in the KB level, the total data volume is small, and the access to description files is throughput-oriented, requiring stable latency, a key-value database is selected.

[0045] In the description file, the description information and the attribute information often have independent update requirements. In other words, when the user wants to update the description information, it is often not necessary to update the attribute information. When the user wants to update the attribute information, it is often not desirable to update the description information.

[0046] For example, if a user modifies a content file, the size of the application system file (i.e., attribute information) may change, but this does not mean that the importance of the application system file (i.e., description information) also changes. For another example, if a user thinks that the application system file needs a new label (i.e., description information), this does not mean that the attribute information such as the size, creation date, author, or permissions of the application system file also needs to be modified.

[0047] There are two types of key-value databases that can store description files, namely unstructured key-value databases and semi-structured key-value databases.

[0048] Unstructured key-value databases store description files based on the "key + value" data model. There is a corresponding relationship between key and value. Key can store the full path of the file, and value can store description information and attribute information. Unstructured key-value databases process key and value as binary data. Unstructured key-value databases can provide key-value access interfaces, such as write (put), update (update), batch (batch), read (get) and scan (scan).

[0049] A semi-structured key-value database can perceive the data type and structure of a value. For example, the data type of a value is an integer (int) or a floating point (float), and the data structure of a value is a linear list (list), a set (set), or a map (map).

[0050] If the description information and attribute information corresponding to the full path of the file are stored as two key-value pairs respectively, the description information and attribute information can be accessed independently. However, storing a description file requires two key-value pairs, which takes up a large storage space. In addition, using two key-value pairs will also increase the access cost. For example, if a user wants to access the attribute information separately, two key-value pairs may be queried based on one key, which increases the scanning delay. Of the two key-value pairs, one belongs to the description information and the other belongs to the attribute information. The user needs to filter out the key-value pair that belongs to the description information, thereby increasing the access cost.

[0051] Moreover, as the number of application system files increases, compared with the solution of one key-value storage, using two key-value storages will cause the storage space to grow at twice the rate and the read volume will double.

[0052] If the description information and attribute information corresponding to the full path of the file are stored as a key-value, the storage space occupied can be reduced compared to the aforementioned "two key-value storage" solution. However, in the key-value database, the "one key-value storage" solution increases the access overhead. The following uses the unstructured key-value database and the semi-structured key-value database as examples to illustrate.

[0053] In an unstructured key-value database, when a user wants to access attribute information separately, he needs to access both the description information and the attribute information at the same time, and then find the required attribute information, which results in high access overhead. In addition, it is difficult for an unstructured key-value database to distinguish between the description information and the attribute information in the value. If you do not want to distinguish manually, you need to set tags for the description information and the attribute information respectively, and use additional programs to identify these tags, which further increases the access overhead.

[0054] In a semi-structured key-value database, although a value can include two fields, description information and attribute information, since the semi-structured key-value database does not know the storage location of the description information and attribute information, it is still necessary to access the description information and attribute information at the same time. Therefore, in a semi-structured key-value database, the access overhead is also large. Moreover, when updating attribute information separately, although the user only updates the attribute information in the value, since a value can only be accessed as a whole, updating this part of the attribute information is equivalent to updating the entire value, thereby increasing the user's update cost.

[0055] Therefore, reducing access overhead while supporting independent access to attribute information is an urgent problem to be solved.

[0056] Figure 2 2 is a schematic flow chart of a data processing method 200 provided in an embodiment of the present application. The data processing method 200 can reduce access overhead on the basis of supporting independent access to description information and attribute information. The method 200 can be executed by any device with computing function (such as cloud management platform 110).

[0057] The method 200 is applied to a key-value database, which is used to store description files in the form of key-value pairs, and the description files include keys, description information and attribute information. In other words, the description files can be stored in a data model of "key", "description information" and "attribute information".

[0058] "Description information" and "attribute information" can be regarded as "values". The key, description information, and attribute information of a description file can be stored as a key-value in a key-value database. This application does not limit the specific form of the key-value database, for example, it can be an unstructured key-value database or a semi-structured key-value database.

[0059] The key may be an identifier of an application system file, such as a full file path or other identifier. The meaning of the description information and attribute information can be found in the previous text and will not be repeated here.

[0060] Combine the following Figure 2 and Figure 3 , describe the method embodiments of the present application in detail.

[0061] S210, receiving a first data write request, wherein the first data write request includes first data to be written, and the first data includes a first key, first description information, and first attribute information.

[0062] The first data write request may come from a client, and the first data write request may be used to write the first data to be written into the database. The first data may be any data, for example, data of any description file. The first key may be an identifier of a description file, the first description information includes description information of the description file, and the first attribute information includes attribute information of the description file.

[0063] The above S210 can also be understood as that the first data write request may include a triple of the first key, the first description information, and the first attribute information. Exemplarily, the API of the data write request may be put-kv(key, value, attr). The above data model including the first key, the first description information, and the first attribute information may be visible to the user.

[0064] S220, according to the first data write request, write the first data to be written into the key-value database, and generate an index record, wherein the index record includes an index of the first description information and an index of the first attribute information, wherein the index of the first description information is used to index the first description information based on the first key, and the index of the first attribute information is used to index the first attribute information based on the first key.

[0065] Combine the following Figure 3 and Figure 4 Some embodiments of S220 are described in detail.

[0066] Figure 3 3 is a schematic block diagram of a data processing system 300 provided in an embodiment of the present application. The data processing system 300 can serve as a server to receive requests from a client, such as data write requests, data read requests, data update requests, data delete requests, etc.

[0067] The data processing system 300 includes: a data unit 310 for reading and writing keys, description information and attribute information; an index unit 320 for managing index records; and a storage unit 330 for storing description information and attribute information.

[0068] Writing the first data into the key-value database may be understood as writing the first description information and the first attribute information in the first data into the storage unit 330. Generating an index record may include: generating an index of the first description information and an index of the first attribute information, and inserting the first key, the index of the first description information, and the index of the first attribute information into the index tree of the index unit 320 as index records.

[0069] The storage unit 330 may include a plurality of blocks, each of which has a block identifier, and the block identifier is used to distinguish different blocks.

[0070] The present application does not limit the specific form of the index of the first description information. For example, the index of the first description information may be in the form of a layout. The layout may include a block identifier, an offset, and a length. The offset is used to indicate the starting position of the storage space of the block corresponding to the block identifier, and the length is used to indicate the number of bytes or bits of the first description information. For another example, the index of the first description information may be the first description information itself.

[0071] Similarly, the present application does not limit the specific form of the index of the first attribute information. For example, the index of the first attribute information may be in the form of a layout. For another example, the index of the first attribute information may be the first attribute information itself.

[0072] Based on the above scheme, since the index record includes the index of the description information and the index of the attribute information, the user can access the description information or the attribute information separately, without having to access the description information and the attribute information as a whole, thereby reducing the access overhead while supporting independent access to the attribute information. On the other hand, the embodiment of the present application only needs one key-value storage file. Compared with the "two key-value" scheme, the data processing method provided by the embodiment of the present application can reduce the occupied storage space.

[0073] In some embodiments, the method 200 also includes: receiving a second data write request, wherein the second data write request includes second data to be written, and the second data includes the second key, second description information and second attribute information; when the number of bytes of the second description information is less than or equal to a first threshold, writing the second description information to the index of the second description information; or when the number of bytes of the second attribute information is less than or equal to a second threshold, writing the second attribute information to the index of the second attribute information.

[0074] The second data write request may come from the client, and the second data write request may be used to write the second data to be written into the database. The second data may be any data, for example, data of any description file. The second key may be an identifier of a description file, the second description information includes description information of the description file, and the second attribute information includes attribute information of the description file.

[0075] The second data may be the same data as the first data, or may be different data from the first data. For other descriptions of the second data write request, reference may be made to the similar descriptions of the first data write request above, which will not be repeated here.

[0076] The first threshold may be a preset value, for example, the first threshold may be 100 bytes. Thus, when the number of bytes of the second description information is greater than 100 bytes, the second description information will be written into the storage unit 330. In this case, the index of the second description information is the layout of the second description information. The layout may point to the storage location of the second description information to implement operations such as reading and updating the first description information.

[0077] The second threshold may be a preset value, and the second threshold may be the same as the first threshold or may be different from the second threshold.

[0078] Specifically, the above scheme can be divided into four situations. Figure 4 Embodiments of these four cases are described in detail.

[0079] Figure 4 It is a schematic diagram of a data processing method 200 provided in an embodiment of the present application.

[0080] See also Figure 4 In some embodiments, when the number of bytes of the second description information is greater than a first threshold value, and the number of bytes of the second attribute information is greater than a second threshold value, the method 200 may include: writing the second description information to a first storage location, and writing the second attribute information to a second storage location; determining a first layout and a second layout, wherein the first layout is used to indicate the first storage location, and the second layout is used to indicate the second storage location; writing the first layout to an index of the second description information, and writing the second layout to an index of the second attribute information.

[0081] The first storage location may be a location of a block in the storage unit 330. The first layout may be used to describe the first storage location. For example, the first layout may include a block identifier, an offset, and a length. The meanings of the block identifier, the offset, and the length are as described above and are not repeated here.

[0082] The scheme of index record 410 can also be called "external" second description information and "external" second attribute information. "External" means that the second description information and the second attribute information are written into storage unit 330, and a pointer to storage unit 330 is stored in index unit 320. In other words, the second description information and the second attribute information are written outside index record 410. In the scheme of index record 410, the second description information and the second attribute information can be written into the disk space continuously, so that when the second description information and the second attribute information are accessed simultaneously in the future, the access can be realized through one input and output.

[0083] It can be seen that when the bytes of the second description information and the second attribute information are large, the layout of the second description information and the second attribute information can be written into the index, thereby achieving separate access to the second description information or the second attribute information.

[0084] See also Figure 4 Index record 420, in some embodiments, when the number of bytes of the second description information is less than or equal to the first threshold, and the number of bytes of the second attribute information is greater than the second threshold, the method 200 may include: writing the second attribute information to a second storage location; determining a second layout, wherein the second layout is used to indicate the second storage location; writing the second description information to an index of the second description information, and writing the second layout to an index of the second attribute information.

[0085] The scheme of index record 420 can also be called that the second description information is “inline” and the second attribute information is “external.” “Inline” means that the second description information is not written into storage unit 330 , but only into index unit 320 , within index record 420 .

[0086] Based on the above scheme, when the second description information is relatively small, the second description information can be written directly into the index record. On the one hand, since the second description information is relatively small, and may even be smaller than the first layout, compared to the scheme of writing the first layout and the second description information in the database, only writing the second description information into the index record in the database can save storage space and increase the storage speed. On the other hand, when accessing the second description information, you can directly "access in one step" in the index record. Compared with the "two-step access" scheme of first obtaining the first layout and then accessing according to the first layout, the data processing method provided by the present application reduces one access from two accesses, improves efficiency by 50%, and reduces access latency.

[0087] See also Figure 4 Index record 430, in some embodiments, when the number of bytes of the second description information is less than or equal to the first threshold, and the number of bytes of the second attribute information is less than or equal to the second threshold, the method 200 may include: writing the second description information to the index of the second description information, and writing the second attribute information to the index of the second attribute information.

[0088] The scheme of index record 430 may also be referred to as second description information “inline” or second attribute information “inline”.

[0089] Based on the above scheme, when both the second description information and the second attribute information are relatively small, the second description information and the second attribute information can be directly written into the index record. On the one hand, compared with the scheme of writing the first layout, the second layout, the second description information and the second attribute information into the database, only writing the second description information and the second attribute information into the index record in the database can save storage space and increase the storage speed. On the other hand, when accessing the second description information or the second attribute information, you can directly "access in one step" in the index record. Compared with the "two-step access" scheme of first obtaining the layout and then accessing according to the layout, the data processing method provided by the present application reduces one access from two accesses, improves efficiency by 50%, and reduces access latency.

[0090] See also Figure 4 Index record 440, in some embodiments, when the number of bytes of the second description information is greater than the first threshold and the number of bytes of the second attribute information is less than or equal to the second threshold, the method 200 may include: writing the second description information to a first storage location; determining a first layout, wherein the first layout is used to indicate the first storage location; writing the first layout to an index of the second description information, and writing the second attribute information to an index of the second attribute information.

[0091] The scheme of index record 420 may also be referred to as the second description information being “external” and the second attribute information being “inline”.

[0092] Based on the above scheme, when the first attribute information is small, the first attribute information can be written directly into the index record. On the one hand, since the first attribute information is small, and may even be smaller than the second layout, compared with the scheme of writing the second layout and the first attribute information in the database, only writing the first attribute information directly into the index record can save storage space and increase the storage speed. On the other hand, when accessing the first attribute information, you can directly "access in one step" in the index record. Compared with the "two-step access" scheme of first obtaining the second layout and then accessing according to the second layout, the data processing method provided by the present application reduces one access from two accesses, improves efficiency by 50%, and reduces access latency.

[0093] In summary, in an embodiment of the present application, when the description information is small, the description information can be written directly into the index record. On the one hand, since the number of bytes of the description information is small, it may even be smaller than the number of bytes occupied by the storage address of the description information. Compared with the solution of writing the description information and its storage address in the database, only writing the description information into the index record in the database can save storage space and increase the storage speed. On the other hand, when accessing the description information, you can directly "access in one step" in the index record. Compared with the "two-step access" solution of first obtaining the storage address of the description information and then accessing it according to the storage address, the data processing method provided by the present application reduces one access from two accesses, improves efficiency by 50%, and reduces access latency. The above analysis is also applicable to attribute information, which will not be repeated here.

[0094] In some embodiments, the method 200 also includes: receiving a data read request, wherein the data read request includes the first key; obtaining the index record according to the first key in the data read request; and reading a first target value according to the index record, wherein the first target value includes the first description information and / or the first attribute information.

[0095] Exemplarily, the API for data read request may be get-kv-all(key,_out&value,_out&attr), which is used to read the first description information and the first attribute information. The API for data read request may also be get-kv-attr(key,_out&attr), which is used to read the first attribute information. The API for data read request may also be get-kv-value(key,_out&value), which is used to read the first description information.

[0096] The data read request may be issued by the data unit 310. The data read request may carry a first key, and the index record corresponding to the first key may be matched in the index unit 320 through the first key, thereby obtaining the index record. For example, the first target value may be the first description information, and the first description information may be obtained according to the index of the first description information. For another example, the first target value may be the first attribute information, and the first attribute information may be obtained according to the index of the first attribute information. For another example, the first target value may be the first description information and the first attribute information, and the first description information and the first attribute information may be obtained according to the index of the first description information and the index of the first attribute information.

[0097] Based on the above scheme, the description information or attribute information can be read separately according to the index record, or the description information and attribute information can be read at the same time. Compared with the scheme of only reading the description information and attribute information first and then extracting the required target value from the description information and attribute information, this application can reduce the reading overhead.

[0098] In some embodiments, the method 200 also includes: receiving a data update request, wherein the data update request includes the first key; updating a second target value according to the data update request, wherein the second target value includes the first description information and / or the first attribute information; and updating the index record according to the updated second target value.

[0099] Exemplarily, the API for the data update request may be update-kv-all(key, value, attr), which is used to update the first description information and the first attribute information. The API for the data update request may also be update-kv-attr(key, attr), which is used to update the first attribute information. The API for the data update request may also be update-kv-value(key, value), which is used to update the first description information.

[0100] The data update request may be issued by the data unit 310. The data update request may carry the first key, and the index record corresponding to the first key is updated in the index unit 320 through the first key. In this way, the next time the updated second target value needs to be accessed, it can be accessed through the updated index record.

[0101] Based on the above scheme, the description information or attribute information can be updated separately according to the index record. Compared with the scheme of only accessing the description information and attribute information first and then updating the target value from the description information and attribute information, this application can reduce the update overhead and improve the update speed.

[0102] Combine the following Figure 5 and Figure 6 , detailed embodiments of updating the second target value under different circumstances are introduced. The following examples are only for illustration and do not constitute a limitation of the present application.

[0103] Figure 5 It is a schematic diagram of data update in two situations provided in the embodiments of the present application.

[0104] See also Figure 5 , the target threshold value refers to the threshold value corresponding to the second target value. For example, when the second target value is the first description information, the target threshold value is the first threshold value. For another example, when the second target value is the first attribute information, the target threshold value is the second threshold value. Figure 5 The horizontal arrow in the middle indicates the byte number of the updated second target value (hereinafter referred to as “the second byte number”). As the arrow points to the right, the second byte number increases.

[0105] In one embodiment, according to the data update request, the second target value is updated, including: determining that the number of bytes of the updated second target value is less than or equal to the target threshold; writing the updated second target value into the index record, wherein the target threshold is the first threshold or the second threshold corresponding to the second target value.

[0106] Referring to case 510, when the number of bytes of the second target value before updating (hereinafter referred to as "the first number of bytes") is less than or equal to the target threshold, the second target value before updating (hereinafter referred to as "the original target value") is stored in the index unit 320 in an "inline" form. If the second number of bytes is less than or equal to the target threshold, the updated second target value may also be stored in the index unit 320 in an "inline" form. Since case 510 sets the first number of bytes to be less than the target threshold, if the second number of bytes is less than the first number of bytes, the updated second target value may also be stored "inline". In other words, it is still stored "inline" after updating.

[0107] Referring to case 520, when the first byte number is greater than the target threshold, the original target value is stored in the storage unit 330 in an "external" form. If the second byte number is less than or equal to the target threshold, the updated second target value may also be stored in the index unit 320 in an "inline" form. That is, because the second byte number is smaller, it may be updated from "external" storage to "inline" storage.

[0108] In one embodiment, according to the data update request, updating the second target value includes: determining that the number of bytes of the updated second target value is greater than the target threshold; writing the updated second target value into the key-value database, wherein the target threshold is the first threshold or the second threshold corresponding to the second target value.

[0109] Referring to case 510, if the second byte number is greater than the target threshold, it means that the byte number of the updated second target value is relatively long and may not be stored in the index record. Therefore, it needs to be stored in the storage unit 330 in an "external" form. In other words, it needs to be updated from "inline" storage to "external" storage.

[0110] Referring to case 520, if the second byte number is greater than the target threshold, the updated second target value still needs to be stored in an "external" form. On this basis, there are different updating methods according to the size relationship between the first byte number and the second byte number. Figure 6 These embodiments are described in detail.

[0111] Figure 6 It is a schematic diagram of two data updating methods provided in the embodiments of the present application.

[0112] See also Figure 6, the original target value is the first attribute information 612 stored in the block 610. The description information 611, the description information 613, and the attribute information 614 are also stored in the block 610. The block 610 may be a block in the storage unit 330. The description information 611 and the attribute information 612 correspond to one key, and the description information 613 and the attribute information 614 correspond to another key.

[0113] In some embodiments, based on the data update request, the second target value is updated, including: determining that the number of bytes of the updated second target value is less than or equal to the number of bytes of the second target value before the update, and the number of bytes of the updated second target value is greater than the target threshold; determining the write position of the second target value based on the index corresponding to the second target value before the update; and writing the updated second target value at the write position of the second target value.

[0114] As an example of the above solution, please refer to the update method 631. Figure 6 , assuming that the update request 630 is used to update the attribute information 612, that is, the second target value in the above scheme is the attribute information 612. When the number of bytes of the attribute information 612' is less than or equal to the attribute information 612 and greater than the target threshold, the index of the attribute information 612 can be obtained. According to the index, the attribute information 612' can be written to the storage location of the attribute information 612, that is, the attribute information 612' is overwritten with the attribute information 612. The update method 631 can also be understood as updating at the location of the original target value, and replacing the second target value before the update with the updated second target value.

[0115] The second target value may be the first description information or the first attribute information. The embodiment of updating the first description information may refer to the embodiment of the first attribute information, which is not described here. When the second target value is the first description information, the target threshold is the first threshold.

[0116] Based on the above scheme, the updated second target value is stored in the position of the second target value before the update. Therefore, for the updated data, the first description information and the first attribute information corresponding to a key are still stored in the position before the update. For example, when the user wants to access the first description information and the first attribute information at the same time, the first description information and the first attribute information can be accessed through one input and output, which reduces the access overhead and improves the access efficiency.

[0117] In other embodiments, the second target value is updated according to the data update request, including: determining that the number of bytes of the updated second target value is greater than the number of bytes of the second target value before the update; writing the updated second target value at a target position in the key-value database, and the target position is different from the storage position of the second target value before the update.

[0118] As an example of the above solution, please refer to the update method 632. Figure 6 , assuming that the update request 630 is used to update the attribute information 612, that is, the second target value in the above scheme is the attribute information 612. When the number of bytes of the attribute information 612' is greater than the attribute information 612, the attribute information 612' can be written into the block 620. The update method 632 can also be understood as updating at a new location.

[0119] The second target value may be the first description information or the first attribute information. The embodiment of updating the first description information may refer to the embodiment of the first attribute information, which is not described here. When the second target value is the first description information, the target threshold is the first threshold.

[0120] In some further embodiments, updating the second target value according to the update request includes: writing the updated second target value.

[0121] As an example of the above solution, please refer to the update method 632. Figure 6 , assuming that the update request 630 is used to update the attribute information 612, that is, the second target value in the above scheme is the attribute information 612.

[0122] The block 620 before updating includes description information 621 and attribute information 622. For example, the attribute information 612' can be written into the block 620. The updating method 632 can also be understood as updating at a new location. The updating method 632 does not need to determine whether the attribute information 612 can be updated at the location before updating, thereby increasing the speed of updating.

[0123] The present application does not limit the new position to be in a different block from the original data, and the updating method 632 may also write the attribute information 612' into the block 610. The second target value may be the first description information or the first attribute information. The embodiment of updating the first description information may refer to the embodiment of the first attribute information, which is not described here.

[0124] Therefore, the update method 631 and the update method 632 can be combined, for example, the update method 632 can also include the step of determining the number of bytes of the second target value. In this way, when the number of bytes is large, the update method 632 can be executed, and when the number of bytes is small, the update method 631 can be executed. The update method 632 can also write the second target value directly to the new position without any determination.

[0125] After the second target value is updated, the index record may be updated, and the updated index record includes the index corresponding to the updated second target value. For example, in the case where the second target value before the update is stored in the index record in an "inline" manner, the updated second target value may be written into the index record as an index, or the layout corresponding to the updated second target value may be written into the index record as an index. For another example, in the case where the second target value before the update is stored in the index record in an "external" manner, the updated second target value may be written into the index record as an index, or the layout corresponding to the updated second target value may be written into the index record as an index.

[0126] In some embodiments, the method 200 further includes: receiving a data deletion request, the data deletion request including the first key; and deleting the index record according to the first key in the data deletion request.

[0127] Exemplarily, the API of the data deletion request may be delete-kv(key), which is used to delete index records.

[0128] The data deletion request may be issued by the data unit 310. The data deletion request may carry a first key, and the index record corresponding to the first key is matched in the index unit 320 through the first key.

[0129] The method embodiments of the present application are described in detail above. The device embodiments of the present application are described below. The device embodiments correspond to the method embodiments, so the parts not described in detail can refer to the previous method embodiments. The device can implement any possible implementation of the above method.

[0130] Figure 7 1 is a schematic block diagram of a data processing device 700 provided in an embodiment of the present application. The device 700 is applied to a key-value database, which is used to store description files in the form of key-value pairs, and the description files include keys, description information and attribute information.

[0131] Figure 7 As shown, the data processing device 700 includes: a receiving module 710, used to receive a first data write request, wherein the first data write request includes first data to be written, and the first data includes a first key, first description information and first attribute information; a processing module 720, used to write the first data to be written into the key-value database according to the first data write request, and generate an index record, wherein the index record includes an index of the first description information and an index of the first attribute information, wherein the index of the first description information is used to index the first description information based on the first key, and the index of the first attribute information is used to index the first attribute information based on the first key.

[0132] In some embodiments, the receiving module 710 is also used to receive a data read request, wherein the data read request includes the first key; the processing module 720 is also used to obtain the index record according to the first key in the data read request, and to read the first target value according to the index record, wherein the first target value includes the first descriptive information and / or the first attribute information.

[0133] In some embodiments, the receiving module 710 is also used to receive a data update request, wherein the data update request includes the first key; the processing module 720 is also used to update the second target value according to the data update request, wherein the second target value includes the first description information and / or the first attribute information, and to update the index record according to the updated second target value.

[0134] In some embodiments, the receiving module 710 is further used to receive a data deletion request, where the data deletion request includes the first key; the processing module 720 is further used to delete the index record according to the first key in the data deletion request.

[0135] In some embodiments, the receiving module 710 is also used to receive a second data write request, wherein the second data write request includes second data to be written, and the second data includes the second key, second description information and second attribute information; the processing module 720 is also used to write the second description information to the index of the second description information when the number of bytes of the second description information is less than or equal to a first threshold; or, the processing module 720 is also used to write the second attribute information to the index of the second attribute information when the number of bytes of the second attribute information is less than or equal to a second threshold.

[0136] Specifically, the specific implementation of various operations of the data processing method performed by the data processing device 700 can refer to the description of the relevant content in the above method embodiment, which will not be repeated here.

[0137] The above-mentioned receiving module 710 and processing module 720 can be implemented by software or by hardware. Exemplarily, the implementation of the receiving module 710 is introduced below by taking the receiving module 710 as an example. Similarly, the implementation of the processing module 720 can refer to the implementation of the receiving module 710.

[0138] As an example of a software functional unit, the receiving module 710 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the receiving module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple data centers with similar geographical locations. Among them, usually a region may include multiple AZs.

[0139] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0140] As an example of a hardware functional unit, the receiving module 710 may include at least one computing device, such as a server, etc. Alternatively, the receiving module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0141] It should be noted that, in other embodiments, the receiving module 710 can be used to execute any step in the data processing method 200, and the processing module 720 can be used to execute any step in the data processing method 200. The steps that the receiving module 710 and the processing module 720 are responsible for implementing can be specified as needed. The full functions of the data processing device 700 are realized by respectively implementing different steps in the data processing method 200 through the receiving module 710 and the processing module 720.

[0142] The present application also provides a computing device 800. Figure 8 As shown, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate through the bus 802. The computing device 800 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 800.

[0143] The bus 802 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The bus 802 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 802 may include a path for transmitting information between various components of the computing device 800 (eg, the memory 806, the processor 804, and the communication interface 808).

[0144] The processor 804 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0145] The memory 806 may include a volatile memory, such as a random access memory (RAM). The memory 806 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0146] The memory 806 stores executable program codes, and the processor 804 executes the executable program codes to respectively implement the functions of the aforementioned receiving module 710 and processing module 720, thereby implementing the data processing method 200. That is, the memory 806 stores instructions for executing the data processing method 200.

[0147] The communication interface 808 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 800 and other devices or communication networks.

[0148] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0149] like Fig. 9 As shown, the computing device cluster includes at least one computing device 800. The same instructions for executing the data processing method 200 may be stored in the memory 806 in one or more computing devices 800 in the computing device cluster.

[0150] In some embodiments, the memory 806 of one or more computing devices 800 in the computing device cluster may also respectively store partial instructions for executing the data processing method 200. In other words, the combination of one or more computing devices 800 may jointly execute instructions for executing the data processing method 200.

[0151] It should be noted that the memory 806 in different computing devices 800 in the computing device cluster may store different instructions, which are respectively used to execute part of the functions of the data processing apparatus 700. That is, the instructions stored in the memory 806 in different computing devices 800 may implement the functions of one or more modules in the receiving module 710 and the processing module 720.

[0152] In some embodiments, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network.

[0153] Fig.10 A possible implementation method is proposed. Fig.10 As shown, two computing devices 800A and 800B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 806 in the computing device 800A stores instructions for executing the functions of the receiving module 710. At the same time, the memory 806 in the computing device 800B stores instructions for the functions of the processing module 720.

[0154] Fig.10 The connection method between the computing device clusters shown may be based on the consideration that the data processing method 200 provided in the present application needs to receive a large number of various requests, and therefore it is considered to hand over the functions implemented by the receiving module 710 to the computing device 800B for execution.

[0155] It should be understood that Figure 7 The functions of the computing device 800A shown in FIG. 8 may also be completed by multiple computing devices 800. Similarly, the functions of the computing device 800B may also be completed by multiple computing devices 800.

[0156] In some embodiments, the memory 806 of one or more computing devices 800 in the computing device cluster may also respectively store partial instructions for executing the data processing method 200. In other words, the combination of one or more computing devices 800 may jointly execute instructions for executing the data processing method 200.

[0157] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned data processing method 200.

[0158] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by the computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned data processing method 200.

[0159] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0160] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0161] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0162] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0163] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0164] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0165] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: The method is applied to a key-value database, the key-value database is used to store description files in the form of key-value pairs, the description files include keys, description information and attribute information, the method includes: Receiving a first data write request, wherein the first data write request includes first data to be written, and the first data includes a first key, first description information, and first attribute information; According to the first data write request, the first data to be written is written into the key-value database, and an index record is generated, wherein the index record includes an index of the first description information and an index of the first attribute information, wherein the index of the first description information is used to index the first description information based on the first key, and the index of the first attribute information is used to index the first attribute information based on the first key.

2. The method according to claim 1, characterized in that: Also includes: receiving a data read request, wherein the data read request includes the first key; Obtain the index record according to the first key in the data read request; According to the index record, a first target value is read, where the first target value includes the first description information and / or first attribute information.

3. The method according to claim 1 or 2, characterized in that: Also includes: receiving a data update request, wherein the data update request includes the first key; updating a second target value according to the data update request, where the second target value includes the first description information and / or the first attribute information; The index record is updated according to the updated second target value.

4. The method according to any one of claims 1 to 3, characterized in that Also includes: receiving a data deletion request, wherein the data deletion request includes the first key; The index record is deleted according to the first key in the data deletion request.

5. The method according to claim 1, characterized in that Also includes: receiving a second data write request, wherein the second data write request includes second data to be written, and the second data includes the second key, second description information, and second attribute information; When the number of bytes of the second description information is less than or equal to a first threshold, writing the second description information into an index of the second description information; or When the number of bytes of the second attribute information is less than or equal to a second threshold, the second attribute information is written into the index of the second attribute information.

6. A data processing device, characterized in that: The device is applied to a key-value database, the key-value database is used to store description files in the form of key-value pairs, the description files include keys, description information and attribute information, and the device includes: A receiving module, configured to receive a first data writing request, wherein the first data writing request includes first data to be written, and the first data includes a first key, first description information, and first attribute information; A processing module is used to write the first data to be written into the key-value database according to the first data write request, and generate an index record, wherein the index record includes an index of the first description information and an index of the first attribute information, wherein the index of the first description information is used to index the first description information based on the first key, and the index of the first attribute information is used to index the first attribute information based on the first key.

7. The device according to claim 6, characterized in that The receiving module is further used to receive a data reading request, wherein the data reading request includes the first key; The processing module is further used to obtain the index record according to the first key in the data read request, and to read a first target value according to the index record, where the first target value includes the first description information and / or first attribute information.

8. The device according to claim 6 or 7, characterized in that The receiving module is further used to receive a data update request, wherein the data update request includes the first key; The processing module is further used to update a second target value according to the data update request, where the second target value includes the first description information and / or the first attribute information, and to update the index record according to the updated second target value.

9. The device according to any one of claims 6 to 8, characterized in that The receiving module is further used to receive a data deletion request, wherein the data deletion request includes the first key; The processing module is further configured to delete the index record according to the first key in the data deletion request.

10. The device according to claim 6, characterized in that The receiving module is further used to receive a second data writing request, wherein the second data writing request includes second data to be written, and the second data includes the second key, second description information and second attribute information; The processing module is further configured to, when the number of bytes of the second description information is less than or equal to a first threshold, write the second description information into an index of the second description information; or, The processing module is further configured to, when the number of bytes of the second attribute information is less than or equal to a second threshold, write the second attribute information into the index of the second attribute information.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 5.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 5.

13. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 5.

Images