Data transfer methods, devices, and computer program products for object storage

Abstract

Embodiments of this disclosure relate to a data transmission method, apparatus, and computer program product for object storage. In the data transmission method provided by embodiments of this disclosure, a set of objects obtained from an object layer is stored in a storage space for storing an object stream, as a first part of the object stream; and in response to an event occurring on a client relating to at least one of the objects in the set, information about the event is stored in the storage space as a second part of the object stream, the information about the event including at least one of the following: an identifier of at least one object, the type of the event, and metadata of the event. In this manner, streaming transmission for object storage can be achieved.

Description

Technical Field

[0001] Embodiments of this disclosure relate to data transmission, and more specifically, to methods, apparatus, and computer program products for data transmission to object storage. Background Technology

[0002] Continuous data pipelines or streaming processing for structured data are becoming increasingly popular. Databases are used to store structured data. Modification Data Capture (CDC) protocols can be used to identify and track data modifications made to tables in a database, making it feasible to transform stored structured data into streaming data for streaming processing. For unstructured data, in addition to object storage, there are streaming storage systems (such as Pravega) that inherently support streaming computation.

[0003] Streaming data is a continuous, massive stream of data generated at high volume and speed from various data sources. Common examples of streaming data include IoT sensor data, server and security logs, real-time advertising, and clickstream data from applications and websites. Summary of the Invention

[0004] In a first aspect of this disclosure, a data transfer method is provided. The method includes storing a set of objects obtained from an object layer in a storage space for storing an object stream, as a first part of the object stream. The method further includes storing information about the event in the storage space as a second part of the object stream in response to an event occurring on a client that relates to at least one of the objects in the set. The information about the event includes at least one of the following: an identifier for at least one object, the type of the event, and metadata about the event.

[0005] In a second aspect of this disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor. The actions include storing a set of objects obtained from an object layer in a storage space for storing an object stream, as a first part of the object stream. The actions also include storing information about the event in the storage space, as a second part of the object stream, in response to an event occurring on a client relating to at least one of the objects in the set. The information about the event includes at least one of the following: an identifier for at least one object, the type of the event, and metadata about the event.

[0006] In a third aspect of this disclosure, a computer program product is provided, which is tangibly stored on a computer-readable medium and includes machine-executable instructions that, when executed, cause a machine to perform the method according to the first aspect.

[0007] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0008] The above and other objects, features, and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts. In the drawings:

[0009] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure may be implemented is shown;

[0010] Figure 2 A flowchart illustrating an example method for data transmission according to an embodiment of the present disclosure is shown;

[0011] Figure 3 A schematic diagram illustrating the acquisition of object and event information as an object stream according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A flowchart illustrating an example method for data transmission according to some embodiments of this disclosure is shown; and

[0013] Figure 5 A block diagram of an example device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0014] The principles of this disclosure will now be described with reference to several exemplary embodiments illustrated in the accompanying drawings. While preferred embodiments of this disclosure are shown in the drawings, it should be understood that these embodiments are described only to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0015] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0016] Besides databases and streaming storage, object storage is also a commonly used storage method for storing unstructured data. The Simplified Storage Service (S3) protocol is a standard protocol for object storage. Many object storage systems are known to be compatible with the S3 protocol, such as Elastic Cloud Storage (ECS) systems.

[0017] However, in many traditional S3 object stores, querying or computing objects can only be done in batch mode, not streaming mode. In traditional S3 object storage systems, subscribers only begin to receive notifications from the system after registering with it. Modifications to objects in the system before the subscriber's registration, as well as versions of those objects, are not transmitted to the subscriber; the subscriber only receives the latest version of the object data.

[0018] Embodiments of this disclosure propose a data transfer method that obtains a set of objects from an object layer and stores the set of objects in a storage space for storing an object stream as part of the object stream. In response to an event occurring on a client regarding at least one object, information about the event is stored in the aforementioned storage space as another part of the object stream. The information about the event may include at least one of the identifier of the at least one object, the type of the event, and the metadata of the event. According to the embodiments described herein, a continuous data pipeline for object storage can be implemented using object streams.

[0019] The following is for reference Figures 1 to 5 The present disclosure is provided to illustrate the basic principles and several exemplary implementations. It should be understood that these exemplary embodiments are given only to enable those skilled in the art to better understand and implement the embodiments of the present disclosure, and are not intended to limit the scope of the disclosure in any way.

[0020] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure may be implemented is shown. (See diagram for reference.) Figure 1 As shown, environment 100 includes server 110, client 120, and server 130 (servers 130 and 110 can be the same entity or two separate entities). Client 120 uploads object data to server 110, and server 110 converts the object data into streaming data before sending it to server 130. Server 130 provides streaming data storage and streaming processing. Client 120 can be a user device or workstation. Examples of client 120 include, but are not limited to, cellular mobile phones, tablets, and laptops. Server 110 can be any object storage system, such as ECS. Server 130 can be a streaming storage system, such as Pravega.

[0021] Server 110 stores data from client 120, and simultaneously generates a data stream for the stored data, sending it to server 130 for streaming data storage. Server 110 can also utilize an object storage architecture compatible with the S3 protocol to store data as objects. Server 130 stores the data as a stream.

[0022] Data generated by client 120 is sent as an object to server 110. Upon receiving the data from client 120, server 110 first stores the data, then checks for any registered streaming data requests. If so, it converts the stored object data into streaming data and sends it to a registered streaming storage system, such as server 130. This will be discussed later. Figure 2 and Figure 3 Provide a detailed description.

[0023] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. For example, embodiments of this disclosure can also be applied to environments different from environment 100. Additionally, although Figure 1 Only two clients are shown, but it is not limited to this; there may be more or fewer clients.

[0024] Figure 2 A flowchart illustrating an example method 200 for data transmission according to an embodiment of the present disclosure is shown. Method 200 may be, for example, by... Figure 1 The server 110 shown is used to execute this. It should be understood that method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of this disclosure is not limited in this respect. The following is in conjunction with... Figure 1 and the following Figure 3 Let me describe method 200 in detail.

[0025] In box 210, server 110 stores a set of objects obtained from the object layer in storage space used to store the object stream, as the first part of the object stream. As previously described, when server 110 detects a registered streaming data request, it allocates storage space for storing the object stream. The objects from the object layer are then stored in that storage space as part of the object stream.

[0026] The following is for reference. Figure 3 The exemplary embodiments of this disclosure are described in detail. Figure 3 A schematic diagram illustrating the acquisition of object and event information from the object layer as an object stream, according to some embodiments of this disclosure, is shown. Figure 3In the illustrated embodiment, in the object layer, after objects 1 and 2 are created, server 130 registers a streaming data request with server 110. Server 110 then begins converting the objects into an object stream. At this point, objects 1 and 2 are first retrieved from the object layer and stored in the storage space used to store the object stream.

[0027] In some embodiments, client 120 may be a user's mobile phone, the object may be an order created by the user in an online trading platform application on the mobile phone, and server 130 may be a device used by the merchant in the transaction. For example, if a merchant wants to perform data analysis on previously generated orders and orders to be generated in the future, server 130 chooses to subscribe to the order streaming demand. Once the subscription is selected, server 110 retrieves all orders previously stored in the object layer and stores the retrieved orders in the storage space used to store the object stream.

[0028] return Figure 2 Then, in box 220, in response to an event occurring at client 120 relating to at least one of the aforementioned objects, information about the event is stored in the aforementioned storage space as a second part of the object stream. The information about the event may include: the identifier of the at least one object, the type of the event, and the event's metadata. The event's metadata indicates the metadata about the object generated by the event. The type of the event may indicate that at least one object has been modified, which will be discussed later. Figure 4 and Figure 3 Provide a detailed description. The type of event can also indicate that at least one object has been deleted.

[0029] In such Figure 3 In the illustrated embodiment, if the previously created object 2 is deleted from the object layer (e.g., because a user deleted an order from a mobile trading platform), then server 110 will record the deletion event in storage as part of the object stream. Specifically, server 110 will obtain the time when object 2 was deleted from client 120 as metadata for the event, and obtain the identifier of object 2. Then, it will store this metadata and the identifier of object 2 in storage as part of the object stream.

[0030] In this way, an object stream is created for object storage, which enables data to be synchronized between two devices via streaming.

[0031] In some embodiments, the event that occurs may not be related to the aforementioned set of objects; for example, a new object may be created in the object layer. Figure 3In the illustrated embodiment, after server 110 begins creating an object stream, a new object 3 is created in the object layer (e.g., due to a user placing a new order in a mobile trading platform application). Accordingly, server 110 retrieves the newly created object 3 and stores it in storage as part of the object stream. Furthermore, server 110 stores the identifier and event type (indicating the creation of a new object) of object 3 in storage as part of the information about the creation event. Additionally, server 110 obtains metadata such as the creation time of object 3 and the size of the created object 3 from client 120, using this metadata as another part of the information about the creation event.

[0032] Figure 4 A flowchart of an example method 400 for data transmission according to some embodiments of the present disclosure is shown. Method 400 can be considered as an example implementation of block 220 in method 200. It should be understood that method 400 may also include additional actions not shown and / or actions shown may be omitted, and the scope of the present disclosure is not limited in this respect. The following is in conjunction with... Figure 1 and Figure 3 Let me describe method 400 in detail.

[0033] In box 410, if the type of the event indicates that at least one object has been modified, then in box 420, the event's metadata is stored in the storage space, which indicates when the at least one object was modified and the modification version number.

[0034] For example, in Figure 3 In the illustrated embodiment, object 1 is modified on client 120, for example, a user changes the quantity of goods in an order from 100 to 50 on a mobile trading platform. The modified object 1 is then stored in the object layer. Simultaneously, because server 130 has already registered the streaming data request for client 120 with server 110 (i.e., server 110 has allocated storage space for storing the object stream for client 120), this modification event is recorded in the object stream. Specifically, when object 1 is modified, server 110 obtains the time of modification of object 1 from client 120 as the event's metadata and stores it in the storage space. Server 110 also stores a modification version number. The default value for the modification version number can be 0, and it increments by 1 when the object is modified. The modification version number increments as the number of modifications to the object increases. Of course, server 110 also stores the identifier of object 1 in the storage space.

[0035] In this way, events that occur after server 130 registers its streaming data request for client 120 with server 110 can be reflected in the object stream. Even if client 120 encounters an error that causes incorrect records in the object layer, the operation history of client 120 can still be reproduced using this object stream.

[0036] In some embodiments, server 110 may also store the specific content of the event as part of the object stream in the storage space. The specific content is the data payload in the object request. For example, in the embodiment where object 1 has been modified, the modified object 1 was created using 2KB client data, so the entire 2KB client data will be stored in the storage space as the value of the specific content.

[0037] If the object size is large, the above approach is obviously unsuitable because it would result in a very large object stream, which is detrimental to transmission. Therefore, in some embodiments, the specific content of the event can also be stored in other locations. The server 110 then stores an index in the storage space indicating the storage location of the specific content, which is then used as information about the event to be constructed into the object stream.

[0038] In embodiments of this disclosure, server 110 may transmit an object stream to server 130 in response to a request from server 130.

[0039] In some embodiments, server 110 can also determine the length of time between the current time and the time when server 130 client 120-2 last requested the transmission of the object stream. If this length of time exceeds a threshold length, then server 110 deletes the object stream from storage. The threshold length can be 1 day, 2 days, 3 days, 1 week, etc. The threshold length can be set by the user or changed by the user according to their needs.

[0040] This approach reduces waste of server resources and lowers storage costs.

[0041] In some embodiments, server 110 may also stop storing newly created objects or information about events in the storage space based on the size of the object stream exceeding a threshold size.

[0042] The embodiments of this disclosure have been described above with reference to the accompanying drawings. It should be understood that the information about events described above is merely illustrative. Based on this disclosure, those skilled in the art will be able to conceive of incorporating other information about events into the object stream.

[0043] Figure 5 A schematic block diagram of an example device 500 that can be used to implement embodiments of the present disclosure is shown. For example, such as Figure 1The server 110 shown can be implemented by device 500. For example... Figure 5 As shown, device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 502 or loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0044] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0045] The various processes and handling described above, such as method 200, can be executed by processing unit 501. For example, in some embodiments, method 200 can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by CPU 501, one or more actions of method 200 described above can be performed.

[0046] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0047] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0048] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0049] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0050] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0051] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0052] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0053] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0054] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A data transmission method, comprising: A set of objects obtained from the object layer is stored in the storage space used to store the object stream, as the first part of the object stream; In response to an event occurring on the client relating to at least one of the set of objects, information about the event is stored in the storage space as a second part of the object stream, the information about the event including at least one of the following: an identifier of the at least one object, the type of the event, and the metadata of the event; as well as In response to determining that the size of the object stream exceeds a threshold size, the storage of information about the event in the storage space is stopped.

2. The method according to claim 1, further comprising: If it is determined that a new object is created in the object layer, the new object obtained from the object layer is stored in the storage space as a third part of the object stream; as well as In response to determining that the size of the object stream exceeds the threshold size, the storage of the new object in the storage space is stopped.

3. The method of claim 1, wherein the information regarding the event further comprises: The specific content of the event, or An index indicating the storage location of the specific content of the event.

4. The method according to any one of claims 1-3, wherein storing the information about the event in the storage space comprises: If the type of the event indicates that at least one object has been deleted, the metadata of the event, which indicates the time when the at least one object was deleted, is stored in the storage space.

5. The method according to any one of claims 1-3, wherein storing the information about the event in the storage space comprises: If the type of the event indicates that at least one object has been modified, the metadata of the event is stored in the storage space, the metadata indicating the time when the at least one object was modified and the modification version number, wherein the modification version number increments with the number of modifications to the at least one object.

6. The method according to claim 1, further comprising: Determine the length of time between the current time and the last time the object stream was requested to be transmitted; If the time length exceeds the threshold length, the object stream is deleted from the storage space.

7. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, the instructions causing the device to perform actions when executed by the processor, the actions including: A set of objects obtained from the object layer is stored in the storage space used to store the object stream, as the first part of the object stream; In response to an event occurring on the client relating to at least one of the group of objects, information about the event is stored in the storage space as a second part of the object stream. The information about the event includes at least one of the following: an identifier of the at least one object, the type of the event, and metadata of the event; and In response to determining that the size of the object stream exceeds a threshold size, the storage of information about the event in the storage space is stopped.

8. The device according to claim 7, wherein the action further includes: If it is determined that a new object is created in the object layer, the new object obtained from the object layer is stored in the storage space as a third part of the object stream; as well as In response to determining that the size of the object stream exceeds the threshold size, the storage of the new object in the storage space is stopped.

9. The device of claim 7, wherein the information regarding the event further comprises: The specific content of the event, or An index indicating the storage location of the specific content of the event.

10. The device according to any one of claims 7-9, wherein storing the information about the event in the storage space comprises: If the type of the event indicates that at least one object has been deleted, the metadata of the event, which indicates the time when the at least one object was deleted, is stored in the storage space.

11. The device according to any one of claims 7-9, wherein storing the information about the event in the storage space comprises: If the type of the event indicates that at least one object has been modified, the metadata of the event is stored in the storage space, the metadata indicating the time when the at least one object was modified and the modification version number, wherein the modification version number increments with the number of modifications to the at least one object.

12. The device according to claim 7, wherein the action further includes: Determine the length of time between the current time and the last time the object stream was requested to be transmitted; If the time length exceeds the threshold length, the object stream is deleted from the storage space.

13. A computer program product tangibly stored on a computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 6.

Images