Data object storage method, device, apparatus, and storage medium

Abstract

The application discloses a data object storage method and device, equipment and a storage medium, and relates to the technical field of data lake. The method comprises the following steps: acquiring at least one to-be-stored object and object numbers of the to-be-stored objects, the object numbers corresponding to the to-be-stored objects one by one, and the at least one to-be-stored object being a data object obtained according to a to-be-stored data file; storing the at least one to-be-stored object and the object numbers of the to-be-stored objects in an object storage server, and obtaining metadata of the to-be-stored objects, wherein each piece of metadata is used for representing a storage position of each to-be-stored object; generating submission applications of the metadata, and sending the submission applications to a submission queue, wherein the submission queue indicates an execution sequence of the submission applications; and based on the execution sequence indicated by the submission queue, performing a submission operation of each piece of metadata in response to the submission applications, wherein each submission operation is used for submitting each piece of metadata to the object storage server.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data lake, and particularly relates to a data object storage method and device, equipment and a storage medium. BACKGROUND

[0002] A data lake is an evolving and scalable big data storage, processing and analysis infrastructure that can be data-oriented to achieve full acquisition, full storage, multi-mode processing and full life cycle management of data of any source, speed, size and type. It can support various enterprise-level applications through interaction and integration with various external heterogeneous data sources. However, when multiple processes simultaneously operate and write metadata in the existing distributed object storage system as the underlying storage of the data lake, the object storage system that finally succeeds in writing may not include the previously written metadata, that is, the metadata writing is inconsistent, resulting in loss of the previously written data. SUMMARY

[0003] The embodiments of the application provide a data object storage method, device, equipment and storage medium, which can reduce the loss of previously written data.

[0004] In a first aspect, the embodiments of the application provide a data object storage method applied to a cluster server, wherein the cluster server is connected with an object storage server, and the method comprises the following steps.

[0005] At least one to-be-stored object and object numbers of each to-be-stored object are obtained, each object number corresponds to each to-be-stored object in one-to-one correspondence, and the at least one to-be-stored object is a data object obtained according to a to-be-stored data file;

[0006] The at least one to-be-stored object and the object numbers of each to-be-stored object are stored in the object storage server, and metadata of each to-be-stored object is obtained, wherein each metadata is used to represent a storage location of each to-be-stored object;

[0007] A submission application of each metadata is generated, and each submission application is sent to a submission queue, wherein the submission queue indicates an execution order of each submission application;

[0008] Based on the execution order indicated by the submission queue, a submission operation of each metadata is executed in response to each submission application, and each submission operation is used to submit each metadata to the object storage server.

[0009] In a second aspect, the embodiments of the application provide a data object storage device applied to a cluster server, wherein the cluster server is connected with an object storage server, and the device comprises the following steps.

[0010] The first obtaining module is configured to obtain at least one to-be-stored object and object numbers of the to-be-stored objects, each object number corresponding to a to-be-stored object, and the at least one to-be-stored object being a data object obtained according to a to-be-stored data file;

[0011] The storage module is configured to store the at least one to-be-stored object and the object numbers of the to-be-stored objects into the object storage server, and obtain metadata of the to-be-stored objects, each metadata being used to represent a storage location of the to-be-stored object;

[0012] The generation module is configured to generate submission applications of the metadata, and send the submission applications to a submission queue, the submission queue indicating an execution order of the submission applications;

[0013] The execution module is configured to execute, in response to the submission applications, submission operations of the metadata according to the execution order indicated by the submission queue, each submission operation being used to submit the metadata to the object storage server.

[0014] In a third aspect, an electronic device is provided, and the device includes a processor and a memory storing computer program instructions; the processor implements the data object storage method in any one of the above aspects when executing the computer program instructions.

[0015] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer program instructions; the computer program instructions are executed by a processor to implement the data object storage method in any one of the above aspects.

[0016] The data object storage method, device, equipment and storage medium provided by the embodiment of the present application are applied to a cluster server, the cluster server is connected with an object storage server, at least one to-be-stored object and object numbers of the to-be-stored objects can be stored in the object storage server first, each object number corresponds to each to-be-stored object one by one, the at least one to-be-stored object is a data object obtained according to a to-be-stored data file, and metadata of each to-be-stored object is obtained, each metadata is used to represent a storage position of each to-be-stored object; then, each metadata submission application is generated, and each submission application is sent to a submission queue, the submission queue indicates an execution order of each submission application; finally, based on the execution order indicated by the submission queue, a submission operation of each metadata is executed in response to each submission application, and each submission operation is used to submit each metadata to the object storage server. In this way, the embodiment of the present application can obtain a submission queue of each metadata submission application according to the submission application of the metadata of each to-be-stored object when storing the at least one to-be-stored object, and execute the submission operation of each metadata according to the execution order indicated by the submission queue, so as to realize the submission of each metadata to the object storage server, thereby ensuring the consistency of metadata writing and avoiding the loss of data written first. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiments of the present application will be briefly introduced below. Those skilled in the art can also obtain other drawings according to these drawings without creating any creative labor.

[0018] Figure 1 is a flowchart of a data object storage method provided by an embodiment of the present application;

[0019] Figure 2 is a data lake system architecture diagram provided by an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of submitting metadata of a to-be-stored object provided by an embodiment of the present application;

[0021] Figure 4 is a flowchart of an execution step of metadata submission provided by an embodiment of the present application;

[0022] Figure 5 is an additional upload schematic diagram provided by an embodiment of the present application;

[0023] Figure 6 is a structural schematic diagram of a data object storage device provided by another embodiment of the present application;

[0024] Figure 7 is a structural schematic diagram of an electronic device provided by still another embodiment of the present application. DETAILED DESCRIPTION

[0025] The features and exemplary embodiments of the various aspects of the present application will be described in detail below with reference to the drawings. The following detailed description is merely intended to explain the present application, and is not intended to limit the present application. The present application can be implemented without some of the specific details. The following description of the embodiments is merely provided to give a better understanding of the present application by showing examples of the present application.

[0026] It should be noted that the terms such as first and second, etc., are merely intended to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the elements defined by the statement "include" do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0027] A data lake is an evolving and scalable big data storage, processing and analysis infrastructure that can be data-oriented to achieve full acquisition, full storage, multi-mode processing and full life cycle management of data of any source, any speed, any size and any type. Through interaction and integration with various external heterogeneous data sources, it supports various enterprise-level applications. However, when the existing distributed object storage system is used as the underlying storage of the data lake, if multiple processes simultaneously operate to write metadata, the object storage system that finally succeeds in writing may not include the previously written metadata, that is, the metadata writing is inconsistent, resulting in loss of the data written first.

[0028] To solve the problems in the prior art, the embodiments of the present application provide a data object storage method, device, equipment and storage medium. First, the data object storage method provided by the embodiments of the present application will be introduced.

[0029] Figure 1 The flowchart of the data object storage method provided by an embodiment of the present application is shown. As shown in Figure 1 The data object storage method is applied to a cluster server, the cluster server is connected with an object storage server, and the method can include the following steps S101-S104:

[0030] S101: Obtain at least one object to be stored and an object number of each object to be stored, wherein each object number corresponds to each object to be stored, and at least one object to be stored is a data object obtained according to a data file to be stored;

[0031] S102: Store at least one object to be stored and the object number of each object to be stored in the object storage server, and obtain metadata of each object to be stored, where each metadata is used to represent the storage location of each object to be stored;

[0032] S103: Generate a submission application for each metadata, and send each submission application to a submission queue, which indicates the execution order of each submission application;

[0033] S104 . Based on the execution order indicated by the submission queue, in response to each submission application, execute a submission operation of each metadata, where each submission operation is used to submit each metadata to the object storage server.

[0034] The data object storage method of the embodiment of the present application is applied to a cluster server, which is connected to an object storage server. It can first store at least one object to be stored and the object number of each object to be stored in the object storage server, where each object number corresponds to each object to be stored. At least one object to be stored is a data object obtained based on the data file to be stored, and metadata of each object to be stored is obtained, each metadata is used to characterize the storage location of each object to be stored; then a submission application for each metadata is generated, and each submission application is sent to a submission queue, and the submission queue indicates the execution order of each submission application; finally, based on the execution order indicated by the submission queue, in response to each submission application, a submission operation of each metadata is executed, and each submission operation is used to submit each metadata to the object storage server. In this way, the embodiment of the present application can obtain the submission queue of each metadata submission application according to the submission application of the metadata of each object to be stored when storing at least one object to be stored, and execute the submission operation of each metadata according to the execution order indicated by the submission queue, so as to submit each metadata to the object storage server, thereby ensuring the consistency of metadata writing and avoiding the loss of the first written data.

[0035] The specific implementation methods of the above steps are introduced below.

[0036] In S101, the at least one to-be-stored object is a data object obtained according to a to-be-stored data file. Exemplarily, the data object can be an entity for storing and operating data in a computer, and can be a simple data type (such as an integer, a floating point number, a string) or a complex data structure (such as an array, a linked list, a tree, a graph, etc.). The data object usually exists in the memory of the computer and can be directly accessed and operated in the program. The data file can be a file that permanently stores data on the external storage medium (such as a hard disk, an SSD, a magnetic tape, etc.) of the computer. The data file usually exists in the form of a file in the file system of the computer, and can be accessed and processed through the operation of the file system. The data file can contain one or more data objects, and metadata (such as a file header, an index, etc.) for describing and organizing the data objects.

[0037] The object number of each object corresponds to each to-be-stored object. Exemplarily, the object number can be a unique number allocated by using a hash algorithm.

[0038] The above-mentioned obtaining of the at least one to-be-stored object and the object number of each to-be-stored object can be directly obtaining the at least one to-be-stored object and the object number of each to-be-stored object of the to-be-stored data file; or, for obtaining the at least one to-be-stored object, each to-be-stored object is allocated a unique object number based on a preset encoding rule, and each object number is used to indicate each to-be-stored object. In addition, the obtaining of the at least one to-be-stored object can also be that the obtained to-be-stored data file is divided according to a preset first capacity threshold to obtain the at least one to-be-stored object of the to-be-stored data file.

[0039] In S102, the above-mentioned metadata can be used to represent the storage location of each to-be-stored object, that is, to uniquely locate the to-be-stored object.

[0040] The above-mentioned storing of the at least one to-be-stored object and the object number of each to-be-stored object in the object storage server and obtaining of the metadata of each to-be-stored object can be directly storing the at least one to-be-stored object and the object number of each to-be-stored object in the object storage server and obtaining the metadata of each to-be-stored object in the case that the object storage server has only one object storage server. Alternatively, in the case that the object storage server includes at least one object storage sub-server and each object storage sub-server is configured with an object storage sub-server number, the object storage sub-server number mapped by each to-be-stored object is determined based on the object number of the at least one to-be-stored object and a preset data distribution rule; and each to-be-stored object and the object number of each to-be-stored object are stored in the object storage sub-server with the object storage sub-server number.

[0041] In S103, the above submitting application can be used to apply to submit each metadata to the object storage server.

[0042] The above generating the submitting application of each metadata and sending each submitting application to the submitting queue can exemplarily be that the cluster server generates the submitting application of each metadata and sends each submitting application to the submitting queue in the cluster server. Alternatively, the cluster server can include a control node server and at least one worker node server, the control node server includes the submitting queue, each worker node server generates the submitting application of each metadata; each worker node server sends each submitting application to the submitting queue of the control node server based on a preset submitting strategy, and the submitting strategy includes a write sparse type strategy and a write intensive type strategy.

[0043] The write sparse type strategy indicates that each worker node server directly sends each submitting application to the submitting queue of the control node server. The write intensive type strategy indicates that each worker node server submits each submitting application to each submitting sub-queue, each submitting sub-queue is located in each worker node server; at least one worker node server sends a query signal to the control node server one by one according to a preset time period; the control node server receives the query signal and checks the queue length of the submitting queue; in the case that the queue length is less than a preset length threshold, the target worker node server sends the submitting application of at least one of the submitting sub-queues to the submitting queue of the control node server, and the target worker node server is the worker node server that sends the query signal in the at least one worker node server.

[0044] In S104, the above submitting operation is used to submit each metadata to the object storage server.

[0045] The above executing order based on the submitting queue indication, in response to each submitting application, executes the submitting operation of each metadata, which can exemplarily be that each submitting application is responded in series according to the executing order indicated by the submitting queue, so that the submitting operation of each metadata is executed one by one.

[0046] In some embodiments, the above S101 can specifically include:

[0047] Obtaining a to-be-stored data file;

[0048] Splitting the to-be-stored data file according to a preset first capacity threshold to obtain at least one to-be-stored object of the to-be-stored data file;

[0049] Based on a preset encoding rule, each to-be-stored object is assigned a unique object number, and each object number is used to indicate each to-be-stored object.

[0050] The first capacity threshold may exemplarily be a size of an object storage interface in the Iceberg data lake.

[0051] The preset encoding rule may exemplarily be a hash function. The preset encoding rule is not limited to the hash function, but can also be other unique encoding rules, such as a hash function and a message digest function, which are not limited herein.

[0052] In the embodiment, the data file to be stored is divided into at least one object to be stored according to the preset first capacity threshold, and each object to be stored is assigned a unique object number, which can meet the requirements of structured, semi-structured and unstructured data storage, and realize unified access and multi-protocol support of the data file.

[0053] In some embodiments, the object storage server can specifically include at least one object storage sub-server, and each object storage sub-server is configured with an object storage sub-server number.

[0054] The S102 can specifically include:

[0055] Based on the object number of each object to be stored and a preset data distribution rule, the object storage sub-server number mapped by each object to be stored is determined.

[0056] Each object to be stored and the object number of each object to be stored are stored in the object storage sub-server with the object storage sub-server number.

[0057] The preset data distribution rule may exemplarily be a controlled replication under scalable hashing (Crush) algorithm, which is widely used in a Ceph distributed storage system. In the present application, the preset data distribution rule is not limited to the Crush algorithm, but can also be other data distribution rules that can realize uniform distribution of data, which are not limited herein.

[0058] In the embodiment, in order to improve the storage capacity of the object storage server, the object storage server can include at least one object storage sub-server. Further, based on the object number of each object to be stored and a preset data distribution rule, the object storage sub-server number mapped by each object to be stored is determined, so that each object to be stored and the object number of each object to be stored are uniformly distributed and stored in the object storage sub-server with the object storage sub-server number.

[0059] In some embodiments, the cluster server can include a control node server and at least one worker node server, and the control node server can include a submission queue.

[0060] The S103 can specifically include:

[0061] Each work node server generates a submission application of each metadata;

[0062] Each work node server sends each submission application to a submission queue of the control node server based on a preset submission strategy,

[0063] The submission strategy includes a write-sparse strategy and a write-intensive strategy.

[0064] The submission application can be used to apply for submitting the metadata of each to-be-stored object.

[0065] The submission strategy can include a write-sparse strategy and a write-intensive strategy. The write-sparse strategy indicates that each work node server directly sends each submission application to the submission queue of the control node server. The write-intensive strategy indicates that each work node server submits each submission application to each submission sub-queue, each submission sub-queue is located in each work node server; at least one work node server sends a query signal to the control node server one by one according to a preset time period; the control node server receives the query signal and checks the queue length of the submission queue; in the case that the queue length is less than a preset length threshold, the target work node server sends the submission application of at least one of the submission sub-queues to the submission queue of the control node server, and the target work node server is the work node server that sends the query signal in the at least one work node server.

[0066] In this embodiment, the cluster server can include a control node server and at least one work node server, each work node server sends a submission application of each metadata to a submission queue of the control node server based on a preset submission strategy, so that multiple processes running on multiple node servers in the cluster server can perform submission application and submission operation at any time.

[0067] In some embodiments, each work node server sends each submission application to a submission queue of the control node server based on a preset submission strategy, which can specifically include:

[0068] In the case that the submission strategy is a write-sparse strategy, each work node server directly sends each submission application to the submission queue of the control node server.

[0069] In the embodiment, the application operation density is sparse in the same time period, the resource occupation ratio is not high, and therefore the FIFO (First In First Out) strategy is adopted for the application on each work node server. The application generated by the thread on the work node server is directly submitted to the tail of the queue of the control node server, and then sequentially executed according to the submission queue of the control node server, thereby improving the efficiency of the metadata submission.

[0070] In some embodiments, each work node server sends each application to the submission queue of the control node server based on the preset submission strategy, which can specifically include:

[0071] In the case of the write-intensive strategy, each work node server submits each application to each submission subqueue, and each submission subqueue is located in each work node server.

[0072] At least one work node server sends a query signal to the control node server in a preset time period;

[0073] The control node server receives the query signal and checks the queue length of the submission queue.

[0074] In the case where the queue length is less than the preset length threshold, the target work node server sends at least one application in the submission subqueue to the submission queue of the control node server, and the target work node server is the work node server that sends the query signal in the at least one work node server.

[0075] In the embodiment, the concurrency is high in the same time period, which occupies a large amount of cache resources, and the node resources of the control node server are valuable, so the application generated by the process on each work node server can be temporarily stored in the submission subqueue of the work node server, and is not directly sent to the control node server, thereby reducing the occupation of the node resources of the control node server.

[0076] As an implementation manner of the application, in order to solve the problem of the additional upload of new data of the data file to be stored, after S104, the method can further include:

[0077] Obtaining an additional upload file of the data file to be stored;

[0078] Uploading the additional upload file to the object storage server.

[0079] The additional upload file can be new data of the data file to be stored, and the additional upload is to add new data at the end of the existing file without affecting the original data.

[0080] The uploading of the additional upload file into the object storage server can be performed by a file sorter to identify large files and small files of the additional upload file. For the small files, the same type of files is merged based on a K-Means clustering algorithm, and the merged size is not greater than a threshold set by the sorter. For the large files, a segmented upload strategy is used. When the cache data reaches the size of a segment, a segment is created and initialized, and an upload number is obtained. Each segment is uploaded according to the upload number. When the amount of stream I / O data is very large, the local cache forms multiple segments for parallel uploading. After all the segments are uploaded, a complete operation is performed to form a complete additional upload file according to the upload number UpLoadID.

[0081] In the embodiment, the additional upload file of the data file to be stored can be obtained, and the additional upload file is uploaded into the object storage server without affecting the original data file.

[0082] As another implementation of the present application, in order to predict the size of the additional upload file in the object storage, which cannot solve the problem of additional upload, before the uploading of the additional upload file into the object storage server, the method further includes:

[0083] obtaining a file capacity of the additional upload file;

[0084] The uploading of the additional upload file into the object storage server includes:

[0085] In a case where the file capacity is greater than a preset second capacity threshold, the additional upload file is segmented according to the second capacity threshold to obtain at least one segment and an upload number of each segment.

[0086] The at least one segment is segmented and uploaded into the object storage server according to the upload number.

[0087] The at least one segment in the object storage server is merged according to the upload number to obtain the additional upload file.

[0088] The file capacity of the additional upload file can be obtained by a file sorter. The file sorter is a tool or program used to classify and organize files according to certain rules. It can judge and filter files according to their properties, contents, names, etc., and then place them in corresponding directories or locations. The file sorter can help users better manage and organize files, and improve the efficiency of file searching and use. Common file sorters can classify files according to their extensions, dates, sizes, etc., and automatically move them to designated folders or directories.

[0089] The second capacity threshold can be a value set by the user according to actual needs, can be the same as the first capacity threshold, and of course can not be used.

[0090] In this embodiment, by segmenting and uploading the additional upload file to the object storage server according to the second capacity threshold, the size of the additional upload file does not need to be known in advance, and the scenario of uploading the additional upload file to the object storage can be realized.

[0091] In some embodiments, in order to solve the influence of mass small file uploading on performance, the uploading of the additional upload file to the object storage server can further include:

[0092] In the case where the file capacity is less than the second capacity threshold, the file tag information of the additional upload file is obtained;

[0093] The additional upload file is merged with similar files to obtain a target upload file, the file capacity of the target upload file is less than or equal to the second capacity threshold, and the similarity between the file tag information of the similar files and the file tag of the additional upload file is less than or equal to a preset similarity threshold.

[0094] The target upload file is uploaded to the object storage server.

[0095] The file tag information can be, for example, the file extension, date, size, storage format, file source, and the like.

[0096] The similarity between the file tag information of the similar files and the file tag of the additional upload file is less than or equal to a preset similarity threshold.

[0097] The merging of the additional upload file with the similar files to obtain the target upload file, and the file capacity of the target upload file being less than or equal to the second capacity threshold, can be, for example, based on a K-means clustering algorithm, and files with similar file tags are merged, and the merging size is not more than a threshold set by a sorter.

[0098] In this embodiment, the additional upload file with a file capacity less than the second capacity threshold is merged with similar files to obtain a target upload file and is uploaded, which can reduce the memory occupation of the cluster server and reduce the influence on the performance of the cluster server.

[0099] In order to facilitate the understanding of the data object storage method in the embodiments of the present application, the application process of the data object storage method is described as follows:

[0100] As shown in Figure 2 The data lake system architecture provided by the embodiments of the present application includes:

[0101] Unified table mapping: The unified mapping module integrates the implementation of the table specifications and storage paths of Iceberg object storage, Hadoop, Hive, and Jdbc. The unified mapping module is called by the computing engine during initialization to load the metadata of the table. Each engine defines its own mapping specification and interface, and supports plug-in deployment.

[0102] Object storage interface: Based on the object storage connector, a unified storage interface service of object storage is provided for the upper layer, supporting object (S3 / S3a, Swift, Atmos) and file (Hdfs, NFS) protocols.

[0103] Object storage OSS (i.e., the above-mentioned object storage server): Object Storage Service, abbreviated as OSS, is a large-scale distributed storage service for the Internet, including at least one storage server. OSS takes object as the basic storage unit and is accessed through a unique object storage address (URL). Each object consists of data, metadata, and a unique identifier (Key). OSS supports massive object storage and has high reliability and scalability. In the object storage service (OSS), HDD and SSD are different types of storage media, representing Hard Disk Drive and Solid State Drive, respectively. HDD (Hard Disk Drive) is a traditional storage medium that uses mechanical components (including rotating disks and moving heads) to read and write data. HDD has a large storage capacity and relatively low cost, suitable for scenarios that require high storage space and relatively low read / write speed. SSD (Solid State Drive) is a new type of storage medium that uses flash memory chips to store data without mechanical components. SSD has a smaller size, lower energy consumption, and faster read / write speed, suitable for scenarios that require high storage performance, such as large-scale data processing and high-concurrency access. In OSS, HDD and SSD are usually used for different storage types and service levels. For example, HDD is commonly used for low-frequency access storage types, suitable for storing less frequently accessed cold data; while SSD is commonly used for high-frequency access storage types, suitable for storing frequently accessed hot data. In summary, HDD and SSD, as different storage media, can be selected and configured according to specific storage requirements and performance requirements to achieve better storage results and performance optimization.

[0104] Based on the above data lake system architecture, the embodiment of the present application further provides a data object storage method, and the storage process is as follows:

[0105] 1) Call the object storage interface to split the data file D into D1, D2, D3, etc. data objects according to the interface size limit.

[0106] 2) Use Hash function to assign unique OID for D1, D2, D3, etc. data objects.

[0107] 3) Based on OID and Crush algorithm, determine the OSD number of D1, D2, D3, etc. data objects mapped to the object storage service OSS, call the object storage interface, and store the data objects into the object storage service OSS.

[0108] 4) Generate metadata of each data object, and complete the object storage process of the data file through the concurrent submission controller.

[0109] 5) Also through the additional upload processor, solve the additional upload and upload of mass small files, and through the concurrent submission controller, submit the metadata of the additional upload file again.

[0110] The above process also needs to submit metadata when the data object is written to the Iceberg Table. The metadata can uniquely locate the user's stored data object. In order to avoid the inconsistency of the data stored in the data lake bottom layer leading to the error of subsequent operation results, it is necessary to ensure the consistency of the data writing process. The key to ensuring the consistency of data writing is to ensure the consistency of metadata concurrent submission. Using a distributed object storage system as the underlying storage of the iceberg data lake, multiple processes running on multiple machines in the cluster can perform transaction operations at any time. Therefore, when multiple processes in the cluster simultaneously perform commit operations on the same Iceberg Table, high concurrency will cause conflicts, that is, the Iceberg Table successfully submitted and overwritten by the last process lacks the metadata of other processes, which means the loss of this part of data.

[0111] In order to solve the concurrent transaction control and realize linear transaction submission, a concurrent submission controller is constructed to guarantee the consistency of linear transaction submission and data concurrent submission. As shown in Figure 3 The concurrent submission controller is used to create a commit application queue on each machine, and then each worker node (i.e. the above-mentioned worker node server) sends its own commit queue (i.e. the above-mentioned submission sub-queue) to the master node (i.e. the above-mentioned control node server), and the master node generates the commit transaction queue of the entire cluster (i.e. the above-mentioned submission queue), and takes out one commit request from the head of the queue each time, and notifies the corresponding process to realize the commit operation (i.e. the submission operation).

[0112] In the execution process, there are two key macro values: commit_queue_max_size (i.e. the preset length threshold described above) and commit_strategy (i.e. the submission strategy described above), which can be set adaptively according to the calculation mode, and manual intervention is also allowed to set the parameters.

[0113] commit_strategy: The submission strategy is divided into a write-intensive strategy and a write-sparse strategy. Different master node queue modes and algorithms are selected according to different submission strategies to realize the algorithm of the master node selecting each worker node.

[0114] commit_queue_max_size: In the case of a write-intensive strategy, the maximum length of the master node submission queue.

[0115] <1> Write-sparse strategy

[0116] Since the commit operation density is sparse in the same time period, the resource occupation ratio is not high, so the FIFO (First In First Out) strategy is adopted for the commit operation on each machine. The commit application generated by the thread on the node is directly submitted to the tail of the master node queue. Then the cluster executes according to the commit queue of the master node.

[0117] <2> Write-intensive strategy

[0118] In the same time period, the concurrency is high, which will occupy a large amount of cache resources. Since the master node resources are valuable, the commit application generated by the process on each machine is temporarily stored in the commit queue of the node, and is not directly sent to the master node.

[0119] The master node commit queue maintains the queue length according to commit_queue_max_size under this strategy. In order to avoid the "starvation" of each process, the time slice rotation method is adopted. N nodes send a heartbeat signal (i.e. the query signal described above) to the master node every 1 / N millisecond time slice to maintain "heartbeat" and query whether the commit application can be submitted. If the length of the master node commit queue at this moment is less than commit_queue_max_size, the master node pulls a commit application from the worker node maintaining "heartbeat" in the time slice to join the tail of the queue. For the multi-process commit joining the commit queue of the node, the FIFO (First In First Out) strategy is still adopted.

[0120] For example Figure 4As shown, the execution steps of metadata submission are as follows:

[0121] <1> commit application submission steps:

[0122] a. The process generates a commit application (i.e., a submission application).

[0123] b. The commit application is submitted to the worker node where the process is located.

[0124] c. Determine the current task commit_strategy. If it is a write sparse strategy, the current worker node is directly submitted to the master node. If it is a write intensive strategy, the time slice rotation method is used, and every N nodes keep a "heartbeat" to the master node every 1 / N millisecond.

[0125] d. The master node receives the heartbeat signal and checks whether the current commit queue length is greater than commit_queue_max_size. If it is less than the macro value, a commit application is pulled from the worker node and added to the end of the commit queue.

[0126] <2> commit operation execution steps:

[0127] e. The master node pops a commit application from the head of the commit queue.

[0128] f. According to the information in the element structure, the process is notified to obtain the right to execute the commit operation.

[0129] g. After the current commit operation is executed, return to step e to continue execution.

[0130] Using a distributed object storage system as the data lake storage base not only needs to solve the problem of additional upload, but also needs to solve the impact of mass small file upload on performance. In order to solve the problem that object storage does not support additional upload, an additional upload controller is constructed, and the specific steps are as follows:

[0131] 1) Through the file sorter, identify large files and small files for additional upload.

[0132] 2) For small files, based on the K-Means clustering algorithm, merge files of the same type, and the merged size does not exceed the threshold set by the sorter.

[0133] 3) For large files, based on the segmented upload strategy, when the cache data reaches the segment block size, create a segment and initialize it to get the upload number.

[0134] 4) Each chunk is uploaded in segments according to the upload number. When the amount of streaming I / O data is very large, the local cache forms multiple chunks and uploads in parallel. When all the chunks are uploaded, the complete operation is performed, and the complete upload file is formed according to the upload number UpLoadID, as shown in Figure 5

[0135] Based on the data object storage method provided in the above embodiments, the application also provides a specific implementation of a data object storage device. Please refer to the following embodiments.

[0136] As shown in Figure 6 The data object storage device provided in the embodiments of the application is applied to a cluster server, and the cluster server is connected with an object storage server. The device can include the following modules: a first acquisition module 601, a storage module 602, a generation module 603, and an execution module 604.

[0137] The first acquisition module 601 is configured to acquire at least one to-be-stored object and object numbers of the to-be-stored objects, the object numbers correspond to the to-be-stored objects one by one, and the at least one to-be-stored object is a data object obtained according to a to-be-stored data file.

[0138] The storage module 602 is configured to store the at least one to-be-stored object and the object numbers of the to-be-stored objects in the object storage server, and obtain metadata of the to-be-stored objects, the metadata are used to represent storage positions of the to-be-stored objects.

[0139] The generation module 603 is configured to generate submission applications of the metadata, and send the submission applications to a submission queue, the submission queue indicating an execution order of the submission applications.

[0140] The execution module 604 is configured to execute, based on the execution order indicated by the submission queue, a submission operation of the metadata in response to the submission applications, the submission operation being used to submit the metadata to the object storage server.

[0141] ​The storage device of the data object in the embodiment of the application is applied to a cluster server, the cluster server is connected with an object storage server, at least one to-be-stored object and object numbers of the to-be-stored objects are stored into the object storage server first, each object number corresponds to each to-be-stored object one by one, the at least one to-be-stored object is a data object obtained according to a to-be-stored data file, and metadata of each to-be-stored object is obtained, each metadata is used to represent a storage position of each to-be-stored object; then, each metadata submission application is generated, and each submission application is sent to a submission queue, the submission queue indicates an execution order of each submission application; finally, based on the execution order indicated by the submission queue, each metadata submission operation is executed in response to each submission application, and each submission operation is used to submit each metadata to the object storage server. In this way, in the embodiment of the application, when at least one to-be-stored object is stored, the submission queue of each metadata submission application is obtained according to the submission application of the metadata of each to-be-stored object, and the submission operation of each metadata is executed according to the execution order indicated by the submission queue, so that each metadata is submitted to the object storage server, thereby ensuring the consistency of metadata writing and avoiding the loss of data written first.

[0142] In some embodiments, the first obtaining module 601 can specifically include:

[0143] The first obtaining unit is configured to obtain a to-be-stored data file.

[0144] The splitting unit is configured to split the to-be-stored data file according to a preset first capacity threshold to obtain at least one to-be-stored object of the to-be-stored data file.

[0145] The allocation unit is configured to allocate a unique object number to each to-be-stored object based on a preset encoding rule, and each object number is used to indicate each to-be-stored object.

[0146] In some embodiments, the object storage server can specifically include at least one object storage subserver, and each object storage subserver is configured with an object storage subserver number.

[0147] The storage module 602 can specifically include:

[0148] The determination unit is configured to determine an object storage subserver number mapped by each to-be-stored object based on the object number of the at least one to-be-stored object and a preset data distribution rule.

[0149] The storage unit is configured to store each to-be-stored object and the object number of each to-be-stored object into the object storage subserver with the object storage subserver number.

[0150] In some embodiments, the cluster server can include a control node server and at least one worker node server, the control node server can include a submission queue;

[0151] The generation module 603 can specifically include:

[0152] The generation unit is configured to generate, by each worker node server, a submission application of each metadata;

[0153] The sending unit is configured to send, by each worker node server, each submission application to the submission queue of the control node server based on a preset submission strategy,

[0154] The submission strategy includes a write sparse type strategy and a write intensive type strategy.

[0155] In some embodiments, the sending unit can specifically include:

[0156] The first sending sub-unit is configured to send, by each worker node server, each submission application to the submission queue of the control node server directly when the submission strategy is the write sparse type strategy.

[0157] In some embodiments, the sending unit can specifically include:

[0158] The first submission sub-unit is configured to submit, by each worker node server, each submission application to each submission sub-queue when the submission strategy is the write intensive type strategy, each submission sub-queue is located in each worker node server;

[0159] The second sending sub-unit is configured to send, by at least one worker node server, a query signal to the control node server one by one according to a preset time period.

[0160] The receiving sub-unit is configured to receive, by the control node server, the query signal and check a queue length of the submission queue.

[0161] The third sending sub-unit is configured to send, by a target worker node server, a submission application of at least one of the submission sub-queues to the submission queue of the control node server when the queue length is less than a preset length threshold, the target worker node server is a worker node server that sends the query signal in the at least one worker node server.

[0162] As an implementation manner of the present application, in order to solve the problem of uploading new data of a data file to be stored, the apparatus 600 can further include:

[0163] The second acquisition module is configured to acquire an uploaded file of the data file to be stored;

[0164] The uploading module is configured to upload the uploaded file to an object storage server.

[0165] As another implementation manner of the present application, in order to solve the problem of unpredictable size of the additional upload file in the object storage, the device 600 can further include:

[0166] The third obtaining module is configured to obtain the file capacity of the additional upload file.

[0167] The uploading module can specifically include:

[0168] The segmenting unit is configured to segment the additional upload file according to the second capacity threshold to obtain at least one segment and an upload number of each segment, when the file capacity is greater than the preset second capacity threshold.

[0169] The first uploading unit is configured to upload the at least one segment to the object storage server according to the upload number of each segment.

[0170] The first merging unit is configured to merge the at least one segment in the object storage server according to the upload number of each segment to obtain the additional upload file.

[0171] In some embodiments, in order to solve the influence of mass small file uploading on performance, the uploading module can further include:

[0172] The second obtaining unit is configured to obtain file tag information of the additional upload file, when the file capacity is less than the second capacity threshold.

[0173] The second merging unit is configured to merge the additional upload file and a similar file to obtain a target upload file, the file capacity of the target upload file is less than or equal to the second capacity threshold, and the similar file is a file whose file tag information has a similarity less than or equal to a preset similarity threshold with the file tag of the additional upload file.

[0174] The second uploading unit is configured to upload the target upload file to the object storage server.

[0175] Figure 7 A hardware structure schematic diagram of an electronic device provided by an embodiment of the present application is shown.

[0176] The electronic device can include a processor 701 and a memory 702 storing computer program instructions.

[0177] Specifically, the processor 701 can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured as one or more integrated circuits that implement one or more embodiments of the present application.

[0178] The memory 702 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 702 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 702 can include removable or non-removable (or fixed) media. Where appropriate, the memory 702 can be internal or external to the integrated gateway disaster recovery appliance. In particular embodiments, the memory 702 is non-volatile, solid-state memory.

[0179] In particular embodiments, the memory 702 can include read-only memory (ROM), random access memory (RAM), a disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to

[0180] The processor 701 implements the data object storage method of any of the above embodiments by reading and executing computer program instructions stored in the memory 702.

[0181] In one example, the electronic device can further include a communication interface 703 and a bus 710. Wherein, as shown, the processor 701, the memory 702, the communication interface 703 are connected through the bus 710 and complete the communication between each other. Figure 7

[0182] The communication interface 703 is mainly used to realize the communication between each module, device, unit and / or equipment in the embodiments of the application.

[0183] ​Bus 710 includes a hardware, software, or both that couples components of electronic device to each other. As an example and not by way of limitation, bus can include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these. Where appropriate, bus 710 can include one or more buses. Although this application describes and shows a particular bus, this application contemplates any suitable bus or interconnect.

[0184] The electronic device can perform the data object storage method in the embodiments of the present application, thereby realizing the data object storage method and apparatus described in combination Figure 1 and Figure 6 with the above embodiments.

[0185] In addition, in combination with the data object storage method in the above embodiments, the embodiments of the present application can provide a computer readable storage medium to realize. The computer readable storage medium has computer program instructions stored thereon; the computer program instructions are executed by a processor to realize any one of the data object storage methods in the above embodiments.

[0186] It needs to be clear that the present application is not limited to the specific configurations and processes described above and shown in the drawings. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after understanding the spirit of the present application.

[0187] The functions noted in the description of the structural block diagrams above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium, or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transport information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0188] It is also important to note that the examples mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the steps mentioned above, that is, the steps can be performed in the order mentioned in the examples, or in an order different from the examples, or several steps can be performed simultaneously.

[0189] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0190] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A data object storage method characterized by comprising: The application is applied to a cluster server connected with an object storage server, the cluster server comprises a control node server and at least one working node server, the control node server comprises a submission queue; the method comprises: Obtaining at least one to-be-stored object and object numbers of each to-be-stored object, each object number corresponding to each to-be-stored object, the at least one to-be-stored object being a data object obtained according to a to-be-stored data file; Storing the at least one to-be-stored object and the object numbers of each to-be-stored object into the object storage server, and obtaining metadata of each to-be-stored object, each metadata being used for representing a storage position of each to-be-stored object; Generating submission applications of each metadata, and sending each submission application to the submission queue, the submission queue indicating an execution order of each submission application; Based on the execution order indicated by the submission queue, performing a submission operation of each metadata in response to each submission application, each submission operation being used for submitting each metadata to the object storage server; The method comprises: Each working node server generates a submission application of each metadata; each working node server sends each submission application to the submission queue of the control node server based on a preset submission strategy, the submission strategy comprising a write sparse type strategy and a write intensive type strategy.

2. The method of claim 1, wherein, The method comprises: Obtaining the to-be-stored data file; Splitting the to-be-stored data file according to a preset first capacity threshold to obtain at least one to-be-stored object of the to-be-stored data file; Based on a preset encoding rule, assigning a unique object number to each to-be-stored object, each object number being used for indicating each to-be-stored object.

3. The method of claim 1, wherein, The object storage server comprises at least one object storage sub-server, each object storage sub-server being configured with an object storage sub-server number; The method comprises: Based on the object numbers of the at least one to-be-stored object and a preset data distribution rule, determining object storage sub-server numbers mapped by each to-be-stored object; Storing each to-be-stored object and the object numbers of each to-be-stored object into the object storage sub-servers with the object storage sub-server numbers.

4. The method of claim 1, wherein, The method comprises: In a case where the submission strategy is the write sparse type strategy, each working node server directly sends each submission application to the submission queue of the control node server.

5. The method of claim 1, wherein, The method comprises: In a case that the commit strategy is the write-intensive strategy, each of the work node servers submits each of the commit applications to each of the commit sub-queues, and each of the commit sub-queues is located at each of the work node servers; The at least one work node server sends a query signal to the control node server in a preset time period; The control node server receives the query signal and checks the queue length of the commit queue; In a case that the queue length is less than a preset length threshold, a target work node server sends the commit application of at least one of the commit sub-queues to the commit queue of the control node server, and the target work node server is the work node server that sends the query signal among the at least one work node server.

6. The method of claim 1, wherein, In a case that the commit queue indicates the execution order, the commit operation of each of the metadata is performed in response to each of the commit applications, and the method further comprises: obtaining an additional upload file of the to-be-stored data file; uploading the additional upload file to the object storage server.

7. The method of claim 6, wherein, Before the additional upload file is uploaded to the object storage server, the method further comprises: obtaining a file capacity of the additional upload file; The uploading of the additional upload file to the object storage server comprises: in a case that the file capacity is greater than a preset second capacity threshold, segmenting the additional upload file according to the second capacity threshold to obtain at least one segment and an upload number of each of the segments; segmenting and uploading the at least one segment to the object storage server according to the upload number of each of the segments; merging the at least one segment in the object storage server according to the upload number of each of the segments to obtain the additional upload file.

8. The method of claim 7, wherein, The uploading of the additional upload file to the object storage server further comprises: in a case that the file capacity is less than the second capacity threshold, obtaining file tag information of the additional upload file; merging the additional upload file with a similar file to obtain a target upload file, the file capacity of the target upload file is less than or equal to the second capacity threshold, and the file tag information of the similar file is similar to the file tag of the additional upload file with a similarity less than or equal to a preset similarity threshold; uploading the target upload file to the object storage server.

9. A storage device for data objects, characterized by The method is applied to a cluster server, the cluster server is connected with an object storage server, the cluster server comprises a control node server and at least one work node server, the control node server comprises a commit queue, and the method comprises: a first obtaining module, configured to obtain at least one to-be-stored object and an object number of each of the to-be-stored objects, each of the object numbers corresponds to each to-be-stored object in a one-to-one manner, and the at least one to-be-stored object is a data object obtained according to a to-be-stored data file; A storage module is configured to store the at least one object to be stored and an object number of each of the at least one object to be stored into the object storage server, and obtain metadata of each of the at least one object to be stored, each of the metadata being used to represent a storage location of each of the at least one object to be stored; A generation module is configured to generate a submission application of each of the metadata, and send each of the submission application to the submission queue, the submission queue indicating an execution order of each of the submission application; An execution module is configured to execute a submission operation of each of the metadata in response to each of the submission application based on the execution order indicated by the submission queue, each of the submission operation being used to submit each of the metadata into the object storage server. The generation module is further configured to generate the submission application of each of the metadata by each of the worker node servers, and send each of the submission application to the submission queue of the control node server based on a preset submission strategy, the submission strategy including a write sparse type strategy and a write intensive type strategy.

10. An electronic device, comprising: The device comprises a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the data object storage method in any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the data object storage method in any one of claims 1-8.

Images