Data processing method, device, equipment and storage medium

Abstract

The present invention provides a data processing method, device, equipment and storage medium, which can be applied to the field of database technology. The method includes: in response to a storage request for a target data block, according to the occupied space of the target data block and the remaining storage space of an initial data management file, storing the target data block in a target data management file, wherein the target data block in the target data management file is associated with a target query attribute; according to the remaining storage entries in the initial data packet, storing the target query attribute in the target data packet; and sending the target data packet and the target data management file to an object storage system in parallel.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] In the design and implementation of modern database systems, a storage-computing separation architecture is generally used to separate data storage and computing processing at the physical level to improve the efficiency and performance of data processing. In database storage systems, data blocks that occupy large space are not suitable for storage in database tables and are generally stored separately in the local file system.

[0003] When processing operations to access a large number of data blocks, the local file system needs to be frequently accessed, thereby reducing the query performance and efficiency of the database. Summary of the invention

[0004] In view of the above problems, the present invention provides a data processing method, apparatus, device and storage medium.

[0005] According to a first aspect of the present invention, there is provided a data processing method, comprising:

[0006] In response to a storage request for a target data block, the target data block is stored in a target data management file according to the occupied space of the target data block and the remaining storage space of the initial data management file, wherein the target data block in the target data management file is associated with a target query attribute; the target query attribute is stored in a target data packet according to the remaining storage entries in the initial data packet; and the target data packet and the target data management file are sent to the object storage system in parallel.

[0007] Optionally, storing the target data block in the target data management file according to the occupied space of the target data block and the remaining storage space of the initial data management file includes:

[0008] In the case where the remaining storage space of the initial data management file is smaller than the occupied space of the target data block, the target data block is divided into a first sub-data block and a second sub-data block according to the remaining storage space of the initial data management file and the arrangement relationship between the multiple data in the target data block; the initial data management file is determined as the first target data management file; the first sub-data block is stored in the first target data management file; the second target data management file is added; and the second sub-data block is stored in the second target data management file, wherein the target data management file includes the first target data management file and the second target data management file.

[0009] Optionally, storing the target data block in the target data management file according to the occupied space of the target data block and the remaining storage space of the initial data management file further includes:

[0010] When the remaining storage space of the initial data management file is greater than or equal to the occupied space of the target data block, the initial data management file is determined as the target data management file; and the target data block is stored in the target data management file.

[0011] Optionally, according to the remaining stored entries in the initial data packet, the target query attributes are stored in the target data packet, including:

[0012] When there are remaining storage entries in the initial data packet, the initial data packet is determined as the first target data packet; when there are no remaining storage entries in the initial data packet, a new data packet is added as the second target data packet; and the target query attribute is stored in the first target data packet or the second target data packet, wherein the target data packet includes the first target data packet and the second target data packet.

[0013] Optionally, the target query attribute includes a target data management file address, a file version number, an offset of the target data block in the target data management file, and a length of the target data block, wherein the file version number represents an operation version number generated in response to the target data block being stored in the target data management file;

[0014] The data processing method also includes: storing the target data management file address, file version number, offset, and target data block length in a target data packet, and generating a storage version number corresponding to the target data packet; determining the key value of the target data block according to the target data packet identifier, storage version number, and row index of the target query attribute in the target data packet; and storing the key value in a transactional key-value storage system so as to obtain the target data block in response to a query operation request.

[0015] Optionally, the data processing method also includes: in response to the scheduled clearing event execution, traversing the metadata information in the parameter file, determining the historical file version number, wherein the metadata information includes the file version number information of the target data management file; and clearing the target data management file corresponding to the historical file version number.

[0016] Optionally, the data processing method also includes: storing the target data management file in a cache system; setting a first priority for target data management files with a lower access rate than a preset access rate in the cache system; and deleting the target data management files with the first priority when the storage space of the cache system is full.

[0017] A second aspect of the present invention provides a data processing device, comprising:

[0018] A first storage module is used to store the target data block in a target data management file in response to a storage request for the target data block according to the occupied space of the target data block and the remaining storage space of the initial data management file, wherein the target data block in the target data management file is associated with the target query attribute;

[0019] A second storage module, configured to store the target query attribute into the target data packet according to the remaining storage entries in the initial data packet; and

[0020] The sending module is used to send the target data packet and the target data management file to the object storage system in parallel.

[0021] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned data processing method.

[0022] The fourth aspect of the present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned data processing method.

[0023] According to the data processing method, device, equipment and storage medium provided by the present invention, the target data block is stored in the target data management file according to the occupied space of the target data block and the remaining storage space of the initial data management file; the target query attribute is stored in the target data packet according to the remaining storage entries in the initial data packet, and finally sent to the object storage system in parallel. Since a data management file that can directly store the target data block is created in the storage architecture of the database, after the target data block is stored in the target data management file, the target query attribute corresponding to the target data block is stored in the target data packet, and finally the data of the target data management file and the target data packet are sent to the object storage system in parallel, the target data block is stored in the object storage system, so that the target data block can be directly obtained according to the target query attribute in the storage architecture of the database when responding to the query operation request, thereby improving the access efficiency and server performance of the data query operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0025] Figure 1 An application scenario of the data processing method according to an embodiment of the present invention is shown;

[0026] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown;

[0027] Figure 3 An example schematic diagram of generating a file version number according to an embodiment of the present invention is shown;

[0028] Figure 4 An example schematic diagram of querying a target data block according to an embodiment of the present invention is shown;

[0029] Figure 5 An example schematic diagram showing a clearing target data management file according to an embodiment of the present invention is shown;

[0030] Figure 6 A structural block diagram of a data processing device according to an embodiment of the present invention is shown;

[0031] Figure 7 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0032] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0033] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0034] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0035] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0036] In the process of implementing the present invention, it is found that the storage and computing separation architecture improves the efficiency and performance of data processing by separating data storage and computing processing at the physical level. In the database storage system, data blocks that occupy a large space are not suitable for storage in the database table and are generally stored separately in the local file system. When processing operations to access a large number of data blocks, it is necessary to frequently access the local file system, thereby reducing the query performance and efficiency of the database.

[0037] In view of this, an embodiment of the present invention provides a data processing method, apparatus, device and storage medium. The method includes: in response to a storage request for a target data block, storing the target data block in a target data management file according to the occupied space of the target data block and the remaining storage space of an initial data management file, wherein the target data block in the target data management file is associated with a target query attribute; storing the target query attribute in a target data packet according to the remaining storage entries in the initial data packet; and sending the target data packet and the target data management file to an object storage system in parallel.

[0038] In the technical solution of the present invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] Figure 1 The application scenario of the data processing method according to the embodiment of the present invention is shown.

[0040] like Figure 1 As shown, the business system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0041] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0042] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0043] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0044] It should be noted that the data processing method provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can generally be set in the server 105. The data processing method provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data processing device provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0045] Alternatively, the data processing method provided in the embodiment of the present invention may also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the data processing apparatus provided in the embodiment of the present invention may also be arranged in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or may be arranged in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0046] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0047] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0048] Figure 2 A flow chart of a data processing method according to an embodiment of the present invention is shown.

[0049] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0050] In operation S210 , in response to a storage request for a target data block, the target data block is stored in a target data management file according to an occupied space of the target data block and a remaining storage space of the initial data management file.

[0051] Optionally, the type of the target data block may be a large object data type (Large Object), and the target data block represents binary data or unstructured data exceeding a preset size threshold, such as video, audio, and high-resolution images.

[0052] Optionally, the database is deployed on a server, and the server determines the occupied space of the target data block in response to receiving a storage request for the target data block, where the occupied space represents the size of the disk space occupied by the target data block.

[0053] Optionally, the storage architecture of the database may be a storage-computing separation architecture, where computing resources and storage resources are independent, and data management files are created in the storage-computing separation architecture, each data management file is set with a corresponding storage space threshold, and the data management file is used to store target data blocks. As target data blocks are continuously stored, the number of data management files also increases. The initial data management file represents the latest data management file added in the storage-computing separation architecture at the moment of responding to a storage request for a target data block.

[0054] Optionally, the target data management file represents a data management file storing the target data block. The occupied space of the target data block and the remaining storage space of the initial data management file can be compared to determine one or more target data management files, and then all the target data blocks can be stored in one target data management file or the target data blocks can be divided and stored in multiple target data management files.

[0055] For example, when the occupied space of the target data block is smaller than the remaining storage space of the initial data management file, the initial data management file may be determined as the target data management file and stored.

[0056] Optionally, the target data block in the target data management file is associated with a target query attribute. The target query attribute represents an attribute related to the target data management file in which the target data block is stored. For example, the target query attribute may be an attribute such as an offset or a file address in the target data management file in which the target data block is stored. The target query attribute is used to query the target data block.

[0057] In operation S220 , the target query attribute is stored in the target data packet according to the remaining storage entries in the initial data packet.

[0058] Optionally, the storage and computing separation architecture includes a data packet (DataCell), which is the smallest data storage unit. Each storage entry in the data packet can be used to store data that does not exceed a preset size threshold, structured data, and target query attributes related to the target data block.

[0059] Optionally, in response to a data storage request, as the target query attribute of the data or the target data block is stored, the data packet in the storage-computation separation architecture also increases. The initial data packet represents the latest added data packet in the storage-computation separation architecture at the moment of responding to the storage request for the target data block.

[0060] Optionally, the target data packet is determined according to the remaining storage entries in the initial data packet. For example, when there are remaining storage entries in the initial data packet, the initial data packet can be determined as the target data packet to store the target query attribute of the target data block.

[0061] In operation S230, the target data packet and the target data management file are sent to the object storage system in parallel.

[0062] Optionally, the target data packet and the target data management file are stored in the same thread pool, and the thread pool is used to send the target data packet and the target data management file to the object storage system in parallel in the form of multiple threads, thereby improving thread sending efficiency.

[0063] Optionally, the server finally stores the target data block and the target query attribute in a separate object storage system, so that the read operation and the write operation on the cluster server do not affect each other.

[0064] Optionally, since a data management file that can directly store the target data block is created in the storage architecture of the database, after the target data block is stored in the target data management file, the target query attribute corresponding to the target data block is stored in the target data packet, and finally the data of the target data management file and the target data packet are sent to the object storage system in parallel, thereby achieving the storage of all target data blocks in the object storage system, so that when responding to a query operation request, the target data block can be obtained according to the target query attribute in the storage architecture of the database, thereby improving the access efficiency of data query operations and server performance.

[0065] Optionally, storing the target data block in the target data management file according to the occupied space of the target data block and the remaining storage space of the initial data management file includes:

[0066] In the case where the remaining storage space of the initial data management file is smaller than the occupied space of the target data block, the target data block is divided into a first sub-data block and a second sub-data block according to the remaining storage space of the initial data management file and the arrangement relationship between the multiple data in the target data block; the initial data management file is determined as the first target data management file; the first sub-data block is stored in the first target data management file; the second target data management file is added; and the second sub-data block is stored in the second target data management file, wherein the target data management file includes the first target data management file and the second target data management file.

[0067] Optionally, the arrangement relationship between the multiple data in the target data block represents the arrangement order of the multiple data in the target data block. The arrangement relationship between the multiple data in the first sub-data block and the second sub-data block is the same as the arrangement relationship between the multiple data in the target data block.

[0068] For example, the target data block is "0011011011", the space occupied by the target data block is 500M, the storage space of the initial data management file is 1GB, and the remaining storage space is 100M. The remaining storage space of the initial data management file is less than the space occupied by the target data block, so the target data block needs to be split. The target data block "0011011011" contains the arrangement order between the data. According to the amount of data corresponding to the occupied space of 100M and the amount of data corresponding to 400M, the first sub-data block "0011" occupying the space of 100M and the second sub-data block "011011" occupying the space of 400M can be obtained.

[0069] Optionally, when the remaining storage space of the initial data management file is smaller than the occupied space of the target data block, the initial data management file is determined as the first target data management file, and a second target data management file is added to the storage and computing separation architecture.

[0070] For example, taking the above-mentioned first sub-data block "0011" and the second sub-data block "011011" as an example, the first sub-data block "0011" is stored in the first target data management file, and the second sub-data block "011011" is stored in the second target data management file.

[0071] Optionally, a storage space threshold is set for each data management file to avoid the problem of data management files being too large and affecting read and write performance. When the remaining storage space of the initial data management file is less than the space occupied by the target data block, the target data management file can be flexibly added to improve the manageability of the data management file and the scalability of the system.

[0072] Optionally, storing the target data block in the target data management file according to the occupied space of the target data block and the remaining storage space of the initial data management file further includes:

[0073] When the remaining storage space of the initial data management file is greater than or equal to the occupied space of the target data block, the initial data management file is determined as the target data management file; and the target data block is stored in the target data management file.

[0074] For example, the target data block is "110100", the space occupied by the target data block is 300M, and the remaining storage space of the initial data management file is 400. The remaining storage space of the initial data management file is larger than the space occupied by the target data block. Therefore, the initial data management file is directly used as the target data management file, and the target data block "110100" is stored in the target data management file.

[0075] Optionally, according to the remaining stored entries in the initial data packet, the target query attributes are stored in the target data packet, including:

[0076] When there are remaining storage entries in the initial data packet, the initial data packet is determined as the first target data packet; when there are no remaining storage entries in the initial data packet, a new data packet is added as the second target data packet; and the target query attribute is stored in the first target data packet or the second target data packet, wherein the target data packet includes the first target data packet and the second target data packet.

[0077] Optionally, the initial data packet includes multiple storage entries, and the target query attribute is stored in a storage entry in the initial data packet according to the order of the remaining storage entries.

[0078] For example, 6344 storage entries are set in the initial data packet. When the 6343rd and 6344th storage entries exist in the initial data packet, the initial data packet is determined as the first target data packet, and the target query attribute is stored in the 6343rd storage entry in the first target data packet.

[0079] For example, 6344 storage entries are set in the initial data packet. When there are no remaining storage entries in the initial data packet, a second target data packet is added to the storage and computing separation architecture. The second target data packet can be set to have 6344 storage entries and store the target query attributes in the first storage entry in the second target data packet.

[0080] Optionally, since the target query attribute occupies less space, it can be directly stored in the target data packet, and then the target data block can be queried in the target data management file directly through the target query attribute in the data packet when a read operation is requested, which solves the problem that the target data block cannot be directly queried in the storage and computing separation architecture due to the large space occupied by the target data block, and optimizes the read and write performance; in addition, the target query attribute that occupies a small storage space makes full use of the storage space of the data packet, reduces the frequency of adding new data packets, and thus improves storage efficiency.

[0081] Optionally, the target query attributes include a target data management file address, a file version number, an offset of a target data block in the target data management file, and a target data block length, wherein the file version number represents an operation version number generated in response to the target data block being stored in the target data management file.

[0082] Optionally, the file version number represents an operation version identifier generated when storing the target data block in the target data management file in response to a storage request for the target data block. A new file version number is generated for each storage operation.

[0083] For example, taking the above-mentioned first sub-data block "0011" and the second sub-data block "011011" as an example, before responding to the storage request for the target data block, the file version number of the first target data management file is "1111", and after responding to the storage request for the target data block, the first sub-data block "0011" is stored in the first target data management file and the file version number "1112" is generated, and the second sub-data block "011011" is stored in the second target data management file and the file version number "1112" is generated.

[0084] Optionally, the target data block is split into a first sub-data block and a second sub-data block, and the file version numbers generated when the first target data management file stores the first sub-data block "0011" and the second target data management file stores the second sub-data block "011011" are the same, so that the database can support version control and tracking of historical data.

[0085] Figure 3 An example schematic diagram of generating a file version number according to an embodiment of the present invention is shown.

[0086] like Figure 3 As shown, in response to a storage request 310 for a target data block, when the remaining storage space of the initial data management file is smaller than the occupied space of the target data block, the target data block is segmented to obtain a first sub-data block and a second sub-data block; the initial data management file is determined as the first target data management file; the first sub-data block is stored in the first target data management file 320; the second target data management file is added, and the second sub-data block is stored in the second target data management file 330, the file version number 340 of the history of the first target data management file is "1111", and a new file version number 350 "1112" is generated for the storage operation of the first sub-data block, and a new file version number 360 "1112" is generated for the second target data management file.

[0087] Optionally, the offset of the target data block in the target data management file represents the starting position of the target data block in the target data management file. The complete target data block can be read according to the offset of the target data block in the target data management file and the length of the target data block.

[0088] Optionally, the data processing method also includes: storing the target data management file address, file version number, offset, and target data block length in a target data packet, and generating a storage version number corresponding to the target data packet; determining the key value of the target data block according to the target data packet identifier, storage version number, and row index of the target query attribute in the target data packet; and storing the key value in a transactional key-value storage system to obtain the target data block in response to a query operation request.

[0089] Optionally, the storage version number represents an operation version identifier generated when storing the target query attribute in the target data packet. The storage version number can be used to quickly determine the target data packet corresponding to the target storage version number when performing a query operation request. A new storage version number is generated for each storage operation in the data packet.

[0090] Optionally, the row index of the target query attribute in the target data packet can be used to quickly determine the storage entry of the target query attribute in the target data packet. For example, a row index of 2 indicates the second storage entry of the target query attribute in the target data packet.

[0091] Optionally, the key value of the target data block is a structured identifier, which can be used to uniquely identify the target data packet where the target query attribute corresponding to the target data block is located. The key value of the target data block is composed of the target data packet identifier, the storage version number, and the row index of the target query attribute in the target data packet.

[0092] Optionally, the key value is stored in a transactional key-value storage system. After the server responds to the query operation request, the target data packet identifier, the storage version number, and the row index of the target query attribute in the target data packet are first determined from the transactional key-value storage system based on the key value, and then the target data packet and the storage version number are determined from the object storage system, and the target query attribute is obtained from the target data packet corresponding to the storage version number based on the row index, and the target data management file and the file version number are determined from the object storage system based on the target query attribute, and the target data block is obtained from the target data management file corresponding to the file version number.

[0093] Optionally, by storing the correspondence between the target query attributes and the target data packet in a transactional key-value storage system in a key-value manner, the uniqueness of the target data packet and the target data management file is ensured, which facilitates the object storage system to quickly locate and access the target data packet and the target data management file, helps to effectively manage large object storage systems, improve performance and reduce storage overhead.

[0094] Figure 4 An example schematic diagram of querying a target data block according to an embodiment of the present invention is shown.

[0095] like Figure 4 As shown, each storage entry in the target data packet 410 stores a target query attribute. According to the first target query attribute "Addr1-1112-offset 1-1", the target data block 1 is queried in the target data management file 420. The target data management file address of the target data block 1 is Addr1, the file version number is "1112", the offset is "offset 1", and the target data block length is "1". According to the second target query attribute "Addr1-1112-offset 2-2", the target data management file 420 is queried. 0 queries the target data block 2, the target data management file address where the target data block 2 is located is Addr1, the file version number is "1112", the offset is "Offset 2", and the target data block length is "2"; according to the third target query attribute "Addr1-1112-Offset 4-3", query the target data block 3 in the target data management file 420, the target data management file address where the target data block 3 is located is Addr1, the file version number is "1112", the offset is "Offset 4", and the target data block length is "3".

[0096] Optionally, the data processing method also includes: in response to the scheduled clearing event execution, traversing the metadata information in the parameter file, determining the historical file version number, wherein the metadata information includes the file version number information of the target data management file; and clearing the target data management file corresponding to the historical file version number.

[0097] Optionally, the metadata information represents information such as attributes and data structure, and the metadata information of the data management file and multiple file version numbers are stored in the parameter file.

[0098] Optionally, the scheduled clearing event represents clearing invalid target data management files at a predetermined time or time interval. The server responds to the scheduled clearing event, traverses the metadata information in the parameter file, and determines the historical file version number. For example, the target data management file corresponds to two file version numbers, "111" and "112", so the file version number "111" is determined to be the historical file version number.

[0099] Optionally, the target data management file is used to incrementally store target data blocks in response to storage requests. Each storage operation generates a new file version number. Therefore, the data in the target data management file corresponding to the latest file version number is complete and valid, and the target data management file corresponding to the historical file version number needs to be cleared.

[0100] Figure 5 An example schematic diagram of clearing a target data management file according to an embodiment of the present invention is shown.

[0101] like Figure 5 As shown, in response to the scheduled clearing event running, the metadata information in parameter file 1, parameter file 2, parameter file 3, and parameter file 4 are traversed, the historical file version numbers "123" and "34" corresponding to the target data management file identifiers are determined, the target data management files corresponding to the historical file version numbers "123" and "34" are cleared, and the target data management files corresponding to the file version number "56789" are retained.

[0102] Optionally, the data processing method also includes: storing the target data management file in a cache system; setting a first priority for target data management files in the cache system with an access rate less than a preset rate; and deleting the target data management files with the first priority when the storage space of the cache system is full.

[0103] Optionally, the target data management file needs to be stored in the cache system for backup. If the storage space of the cache system is limited, the cache system can set a priority based on the access rate of the data in the target data management file. For target data management files with a lower access rate than the preset rate, a first priority is set. When the storage space of the cache system is full, the target data management files with the first priority are deleted first, so as to ensure efficient data access.

[0104] Optionally, for large data that is accessed temporarily, the cache system may not set a priority. When the read operation is completed or the response data is cleaned up, the corresponding data in the cache system is directly deleted to avoid long-term occupation of the cache system storage space.

[0105] Based on the above data processing method, the present invention also provides a data processing device. Figure 6 The device is described in detail.

[0106] Figure 6 A structural block diagram of a data processing device according to an embodiment of the present invention is shown.

[0107] like Figure 6 As shown, the data processing device 600 of this embodiment includes a first storage module 610 , a second storage module 620 and a sending module 630 .

[0108] The first storage module 610 is used to store the target data block in the target data management file in response to the storage request for the target data block according to the occupied space of the target data block and the remaining storage space of the initial data management file, wherein the target data block in the target data management file is associated with the target query attribute. In one embodiment, the first storage module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0109] The second storage module 620 is used to store the target query attribute into the target data packet according to the remaining storage entries in the initial data packet. In one embodiment, the second storage module 620 can be used to perform the operation S220 described above, which will not be described in detail here.

[0110] The sending module 630 is used to send the target data packet and the target data management file to the object storage system in parallel. In one embodiment, the sending module 630 can be used to perform the operation S230 described above, which will not be described in detail here.

[0111] Optionally, the first storage module 610 includes a first storage submodule, a second storage submodule, a third storage submodule, a fourth storage submodule and a fifth storage submodule.

[0112] The first storage submodule is used to divide the target data block into a first sub-data block and a second sub-data block according to the arrangement relationship between the remaining storage space of the initial data management file and multiple data in the target data block when the remaining storage space of the initial data management file is smaller than the occupied space of the target data block.

[0113] The second storage submodule is used to determine the initial data management file as the first target data management file.

[0114] The third storage submodule is used to store the first sub-data block in the first target data management file.

[0115] The fourth storage submodule is used to add a second target data management file.

[0116] The fifth storage submodule is used to store the second sub-data block in a second target data management file, wherein the target data management file includes a first target data management file and a second target data management file.

[0117] Optionally, the first storage module 610 further includes a sixth storage submodule and a seventh storage submodule.

[0118] The sixth storage submodule is used to determine the initial data management file as the target data management file when the remaining storage space of the initial data management file is greater than or equal to the occupied space of the target data block.

[0119] The seventh storage submodule is used to store the target data block in the target data management file.

[0120] Optionally, the second storage module 620 includes an eighth storage sub-module, a ninth storage sub-module and a tenth storage sub-module.

[0121] An eighth storage submodule is configured to determine the initial data packet as the first target data packet when there are remaining storage entries in the initial data packet.

[0122] The ninth storage submodule is used to add a new data packet as a second target data packet when there are no remaining storage entries in the initial data packet.

[0123] The tenth storage submodule is used to store the target query attribute in the first target data packet or the second target data packet, wherein the target data packet includes the first target data packet and the second target data packet.

[0124] Optionally, the data processing device 600 further includes a generating module, a determining module and a third storage module.

[0125] The generation module is used to store the target data management file address, file version number, offset, and target data block length in the target data packet, and generate a storage version number corresponding to the target data packet.

[0126] The determination module is used to determine the key value of the target data block according to the target data packet identifier, the storage version number, and the row index of the target query attribute in the target data packet.

[0127] The third storage module is used to store the key value in the transactional key value storage system so as to obtain the target data block in response to the query operation request.

[0128] Optionally, the data processing device 600 further includes a traversal module and a clearing module.

[0129] The traversal module is used to respond to the timing clearing event and run, traverse the metadata information in the parameter file, and determine the historical file version number, wherein the metadata information includes the file version number information of the target data management file.

[0130] The clearing module is used to clear the target data management file corresponding to the historical file version number.

[0131] Optionally, the data processing device 600 further includes a fourth storage module, a setting module and a deleting module.

[0132] The fourth storage module is used to store the target data management file in the cache system.

[0133] The setting module is used to set a first priority for a target data management file with a less than preset access rate in the cache system.

[0134] The deletion module is used to delete the target data management file of the first priority when the storage space of the cache system is full.

[0135] Optionally, any multiple modules of the first storage module 610, the second storage module 620 and the sending module 630 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. Optionally, at least one of the first storage module 610, the second storage module 620 and the sending module 630 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the first storage module 610, the second storage module 620 and the sending module 630 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0136] Figure 7 A block diagram of an electronic device suitable for implementing a data processing method according to an embodiment of the present invention is shown.

[0137] Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0138] like Figure 7 As shown, the computer electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 709 to the random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0139] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in one or more memories.

[0140] Optionally, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0141] Optionally, the method flow according to the embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. Optionally, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0142] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0143] Optionally, the computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0144] For example, optionally, the computer-readable storage medium may include the ROM 702 and / or the RAM 703 described above and / or one or more memories other than the ROM 702 and the RAM 703 .

[0145] An embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the data processing method provided by the embodiment of the present invention.

[0146] When the computer program is executed by the processor 701, the above functions defined in the system / device of the embodiment of the present invention are performed. Optionally, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0147] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0148] Optionally, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages, specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, such as Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. It can be understood by those skilled in the art that the features recorded in the various embodiments and / or claims of the present invention can be combined and / or combined in various ways, even if such a combination or combination is not explicitly recorded in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0150] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: In response to a storage request for a target data block, storing the target data block in a target data management file according to an occupied space of the target data block and a remaining storage space of an initial data management file comprises: In a case where the remaining storage space of the initial data management file is smaller than the occupied space of the target data block, the target data block is segmented according to the remaining storage space of the initial data management file and the arrangement relationship between the plurality of data in the target data block to obtain a first sub-data block and a second sub-data block; Determining the initial data management file as a first target data management file; storing the first sub-data block in the first target data management file; Add a second target data management file; The second sub-data block is stored in the second target data management file, wherein the target data management file includes the first target data management file and the second target data management file, wherein the target data block in the target data management file is associated with a target query attribute, the target data block represents binary data or unstructured data exceeding a preset size threshold, and the initial data management file represents the latest added data management file existing in the storage-computation separation architecture at the time of responding to the storage request; According to the remaining storage entries in the initial data packet, the target query attribute is stored in the target data packet, wherein the initial data packet represents the latest added data packet existing in the storage-computation separation architecture at the time of responding to the storage request, the target query attribute represents the attribute related to the target data management file in which the target data block is stored, the target query attribute includes the target data management file address, the file version number, the offset of the target data block in the target data management file, and the target data block length, and the file version number represents the operation version number generated in response to the target data block being stored in the target data management file; and Sending the target data packet and the target data management file to the object storage system in parallel; The target data management file address, the file version number, the offset, and the target data block length are stored in the target data packet, and a storage version number corresponding to the target data packet is generated; Determine the key value of the target data block according to the target data packet identifier, the storage version number, and the row index of the target query attribute in the target data packet; The key value is stored in a transactional key-value storage system, so that the target data block is obtained based on the transactional key-value storage system and the object storage system in response to a query operation request.

2. The method according to claim 1, characterized in that According to the occupied space of the target data block and the remaining storage space of the initial data management file, the target data block is stored in the target data management file, further comprising: In a case where the remaining storage space of the initial data management file is greater than or equal to the occupied space of the target data block, determining the initial data management file as the target data management file; The target data block is stored in the target data management file.

3. The method according to claim 1, characterized in that According to the remaining storage entries in the initial data packet, storing the target query attribute into the target data packet includes: In the case where there are remaining storage entries in the initial data packet, determining the initial data packet as a first target data packet; In the case that there are no remaining storage entries in the initial data packet, adding a new data packet as a second target data packet; The target query attribute is stored in the first target data packet or the second target data packet, wherein the target data packet includes the first target data packet and the second target data packet.

4. The method according to claim 1, characterized in that The method further comprises: In response to the scheduled clearing event running, traverse the metadata information in the parameter file to determine the historical file version number, wherein the metadata information includes the file version number information of the target data management file; The target data management file corresponding to the historical file version number is cleared.

5. The method according to claim 1, characterized in that: The method further comprises: Storing the target data management file in a cache system; For a target data management file in the cache system whose access rate is less than a preset rate, a first priority is set; When the storage space of the cache system is full, the target data management file of the first priority is deleted.

6. A data processing device, characterized in that: The device comprises: A first storage module, configured to, in response to a storage request for a target data block, store the target data block in a target data management file according to an occupied space of the target data block and a remaining storage space of an initial data management file, the first storage module comprising a first storage submodule, a second storage submodule, a third storage submodule, a fourth storage submodule and a fifth storage submodule; A first storage submodule is used for, when the remaining storage space of the initial data management file is smaller than the occupied space of the target data block, dividing the target data block according to the arrangement relationship between the remaining storage space of the initial data management file and the plurality of data in the target data block to obtain a first sub-data block and a second sub-data block; A second storage submodule, used to determine the initial data management file as a first target data management file; A third storage submodule, used for storing the first sub-data block in the first target data management file; A fourth storage submodule, used for adding a second target data management file; a fifth storage submodule, configured to store the second sub-data block in the second target data management file, wherein the target data management file includes the first target data management file and the second target data management file, wherein the target data block in the target data management file is associated with a target query attribute, the target data block represents binary data or unstructured data exceeding a preset size threshold, and the initial data management file represents the latest added data management file existing in the storage-computation separation architecture at the time of responding to the storage request; a second storage module, configured to store the target query attribute into the target data packet according to the remaining storage entries in the initial data packet, wherein the initial data packet represents the latest added data packet existing in the storage-computation separation architecture at the time of responding to the storage request, the target query attribute represents the attribute related to the target data management file in which the target data block is stored, the target query attribute includes the target data management file address, the file version number, the offset of the target data block in the target data management file, and the target data block length, and the file version number represents the operation version number generated in response to the target data block being stored in the target data management file; and A sending module, used for sending the target data packet and the target data management file to the object storage system in parallel; A generating module, used for storing the target data management file address, the file version number, the offset, and the target data block length in the target data packet, and generating a storage version number corresponding to the target data packet; A determination module, configured to determine a key value of the target data block according to a target data packet identifier, the storage version number, and a row index of the target query attribute in the target data packet; The third storage module is used to store the key value in a transactional key-value storage system, so as to obtain the target data block based on the transactional key-value storage system and the object storage system in response to a query operation request.

7. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Executable instructions are stored thereon, and when the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

Images