Synchronous storage methods, systems, terminals, and storage media for unstructured data

Abstract

This invention relates to the field of object storage technology, specifically providing a method, system, terminal, and storage medium for the synchronous storage of unstructured data. The method includes: acquiring source information of newly added unstructured data and saving the source information to a source list; creating a storage information table, which includes object storage information items to be recorded; generating configuration information based on the source list and basic information of the unstructured data, and saving the configuration information to a task configuration list, whereby the basic information includes name, time parameters, and timing strategies; setting a scheduler to generate a storage task to save the unstructured data to object storage based on the configuration list; and calling an executor to execute the storage task, acquiring the structured data corresponding to the object storage information items, and saving the structured data. This invention achieves unified storage and management of unstructured data, facilitating presentation based on actual business needs.

Description

Technical Field

[0001] This invention belongs to the field of object storage technology, specifically relating to a method, system, terminal, and storage medium for synchronous storage of unstructured data. Background Technology

[0002] Object storage, also known as object-based storage, is a general term used to describe methods for solving and processing discrete units called objects. Like files, objects contain data, but unlike files, objects do not have a hierarchical structure within a single layer. Each object resides at the same level in a flat address space called a storage pool; one object does not belong to a lower level of another. Both files and objects have metadata associated with the data they contain, but objects are characterized by extended metadata. Each object is assigned a unique identifier, allowing a server or end user to retrieve the object without needing to know the physical address of the data. This approach is helpful for automating and simplifying data storage in cloud computing environments.

[0003] As structured data from various business systems is continuously collected and aggregated, numerous links to unstructured data, such as images, videos, and files, exist within the structured data tables. When presenting data based on actual business needs, some links can access the unstructured data normally, while others require downloading the unstructured data, which significantly limits the possibilities for business presentations. Summary of the Invention

[0004] To address the limitation in data display caused by inconsistent link addresses in existing technologies, this invention provides a method, system, terminal, and storage medium for synchronous storage of unstructured data, thereby resolving the aforementioned technical problems.

[0005] In a first aspect, the present invention provides a method for synchronous storage of unstructured data, comprising:

[0006] Obtain the source information of the newly added unstructured data and save the source information to the source list;

[0007] Create a storage information table, which includes object storage information items that need to be recorded;

[0008] Configuration information is generated based on the source list and unstructured data, and the configuration information is saved to the task configuration list. The basic information includes name, time parameters, and timing strategy.

[0009] The scheduler is configured to generate a storage task that saves unstructured data to object storage based on the configuration list;

[0010] The executor is invoked to execute the storage task, and the structured data corresponding to the object storage information item is obtained and saved.

[0011] In one optional implementation, the source information of the newly added unstructured data is obtained, and the source information is saved to the source list, including:

[0012] Obtain the source information of newly added unstructured data, including id, resource address, addition time, user information, device information, and type information;

[0013] Create a source list and import the source information into the source list;

[0014] Assign a list name to the source list and save the source list to the first path.

[0015] In an optional implementation, configuration information is generated based on the source list and unstructured data, and the configuration information is saved to the task configuration list. The basic information includes name, time parameters, and timing strategy, including:

[0016] The list name and basic information of the source list of unstructured data are integrated into configuration information, and the time parameters include creation time and update time.

[0017] In an optional implementation, an executor is invoked to execute the storage task, and the structured data corresponding to the object storage information item is obtained and the structured data is saved, including:

[0018] The executor reads the configuration information of the target unstructured data from the configuration list, wherein the target unstructured data is a task object storing the task;

[0019] The resource address is parsed from the configuration information, the target unstructured data is obtained from the resource address, and the target unstructured data is uploaded to the object storage platform.

[0020] Based on the configuration information, the storage information table of the target unstructured data is retrieved, the parameter value corresponding to the object storage information item in the storage information table is obtained from the object storage platform, and the parameter value is imported into the storage information table.

[0021] In a second aspect, the present invention provides a synchronous storage system for unstructured data, comprising:

[0022] The first creation module is used to obtain the source information of newly added unstructured data and save the source information to the source list;

[0023] The second creation module is used to create a storage information table, which includes object storage information items that need to be recorded.

[0024] The third creation module is used to generate configuration information based on the source list and the basic information of unstructured data, and save the configuration information to the task configuration list. The basic information includes name, time parameters and timing strategy.

[0025] The scheduling module is used to configure the scheduler to generate storage tasks that save unstructured data to object storage based on the configuration list;

[0026] The execution module is used to call the executor to execute the storage task, obtain the structured data corresponding to the object storage information item, and save the structured data.

[0027] In an optional implementation, the first creation module includes:

[0028] The source acquisition unit is used to acquire the source information of newly added unstructured data, the source information including id, resource address, addition time, user information, device information and type information;

[0029] A list creation unit is used to create a source list and import the source information into the source list;

[0030] A list naming unit is used to assign a list name to the source list and save the source list to a first path.

[0031] In an optional implementation, the third creation module includes:

[0032] The configuration unit is used to integrate the list name and basic information of the source list of unstructured data into configuration information, wherein the time parameters include creation time and update time.

[0033] In an optional implementation, the execution module includes:

[0034] A configuration reading unit is used for the executor to read configuration information of target unstructured data from a configuration list, wherein the target unstructured data is a task object storing the task;

[0035] The data storage unit is used to parse the resource address from the configuration information, obtain the target unstructured data from the resource address, and upload the target unstructured data to the object storage platform.

[0036] The parameter recording unit is used to retrieve the storage information table of the target unstructured data based on the configuration information, obtain the parameter value corresponding to the object storage information item in the storage information table from the object storage platform, and import the parameter value into the storage information table.

[0037] Thirdly, a terminal is provided, including:

[0038] Processor, memory, among which,

[0039] This memory is used to store computer programs.

[0040] The processor is used to retrieve and run the computer program from memory, causing the terminal to perform the terminal method described above.

[0041] Fourthly, a computer storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the methods described in the above aspects.

[0042] The beneficial effects of this invention are that the method, system, terminal, and storage medium for synchronous storage of unstructured data provided by this invention, through setting configuration files, periodically synchronize and store unstructured data, and record the storage parameters of the unstructured data in the object storage platform. Based on the storage parameters, the unstructured data can be directly retrieved from the object storage platform. This invention achieves unified storage and management of unstructured data through periodic synchronization of unstructured data with OSS object storage tasks, facilitating demonstration based on actual business needs.

[0043] Furthermore, the design principle of this invention is reliable, the structure is simple, and it has a very wide range of application prospects. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.

[0046] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0048] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0050] The method for synchronous storage of unstructured data provided in this embodiment of the invention is executed by a computer device, and correspondingly, the system for synchronous storage of unstructured data runs in the computer device.

[0051] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The executing entity can be a synchronous storage system for unstructured data. Depending on different requirements, the order of the steps in this flowchart can be changed, and some can be omitted.

[0052] like Figure 1 As shown, the method includes:

[0053] Step 110: Obtain the source information of the newly added unstructured data and save the source information to the source list;

[0054] Step 120: Create a storage information table, which includes object storage information items that need to be recorded;

[0055] Step 130: Generate configuration information based on the source list and unstructured data, and save the configuration information to the task configuration list. The basic information includes name, time parameters, and timing strategy.

[0056] Step 140: Configure the scheduler to generate a storage task that saves unstructured data to object storage based on the configuration list;

[0057] Step 150: Invoke the executor to execute the storage task, obtain the structured data corresponding to the object storage information item, and save the structured data.

[0058] To facilitate understanding of the present invention, the following description further illustrates the method for synchronous storage of unstructured data provided by the present invention, based on the principle of the method for synchronous storage of unstructured data and in conjunction with the process of synchronous storage of unstructured data in the embodiments.

[0059] Specifically, the method for synchronously storing unstructured data includes:

[0060] S1. Obtain the source information of the newly added unstructured data and save the source information to the source list.

[0061] Obtain source information for newly added unstructured data, including ID, resource address, addition time, user information, device information, and type information; create a source list and import the source information into the source list; assign a list name to the source list and save the source list to a first path.

[0062] Specifically, a source list is created, and the source table name and field information are stored in the scheduled task source information table to support the executor in accurately obtaining data information from the source table when executing tasks.

[0063] S2. Create a storage information table, which includes the object storage information items that need to be recorded.

[0064] Create a storage information table and store the storage table name and field information in the scheduled task storage information table to support the executor in saving the generated OSS object storage access link address to the database when executing the task.

[0065] S3. Generate configuration information based on the source list and unstructured data, and save the configuration information to the task configuration list. The basic information includes name, time parameters, and timing strategy.

[0066] The list name and basic information of the source list of unstructured data are integrated into configuration information, and the time parameters include creation time and update time.

[0067] Specifically, the unstructured data link address table that needs to be synchronized is configured into the scheduled task configuration table, providing data support for the executor to retrieve the source table, link fields and storage table information required by the current task when executing the task.

[0068] S4. Configure the scheduler to generate a storage task that saves unstructured data to object storage based on the configuration list.

[0069] The scheduler queries the scheduled task configuration information table for tasks to be executed soon. Specifically, the scheduler monitors the configuration information in the configuration list, calculates the upload execution time based on the timing policy and update time, and generates a storage task for that configuration information when the system time reaches the upload execution time.

[0070] S5. Call the executor to execute the storage task, obtain the structured data corresponding to the object storage information item, and save the structured data.

[0071] The executor reads the configuration information of the target unstructured data from the configuration list. The target unstructured data is the task object of the storage task. It parses the resource address from the configuration information, obtains the target unstructured data from the resource address, and uploads the target unstructured data to the object storage platform. Based on the configuration information, it retrieves the storage information table of the target unstructured data, obtains the parameter value corresponding to the object storage information item in the storage information table from the object storage platform, and imports the parameter value into the storage information table.

[0072] In some embodiments, the unstructured data synchronization storage system 200 may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the unstructured data synchronization storage system 200 may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) Functionality for synchronous storage of unstructured data.

[0073] In this embodiment, the unstructured data synchronization storage system 200 can be divided into multiple functional modules according to its functions, such as... Figure 2 As shown. The functional modules may include: a first creation module 210, a second creation module 220, a third creation module 230, a scheduling module 240, and an execution module 250. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and are stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0074] The first creation module is used to obtain the source information of newly added unstructured data and save the source information to the source list;

[0075] The second creation module is used to create a storage information table, which includes object storage information items that need to be recorded.

[0076] The third creation module is used to generate configuration information based on the source list and the basic information of unstructured data, and save the configuration information to the task configuration list. The basic information includes name, time parameters and timing strategy.

[0077] The scheduling module is used to configure the scheduler to generate storage tasks that save unstructured data to object storage based on the configuration list;

[0078] The execution module is used to call the executor to execute the storage task, obtain the structured data corresponding to the object storage information item, and save the structured data.

[0079] Optionally, as an embodiment of the present invention, the first creation module includes:

[0080] The source acquisition unit is used to acquire the source information of newly added unstructured data, the source information including id, resource address, addition time, user information, device information and type information;

[0081] A list creation unit is used to create a source list and import the source information into the source list;

[0082] A list naming unit is used to assign a list name to the source list and save the source list to a first path.

[0083] Optionally, as an embodiment of the present invention, the third creation module includes:

[0084] The configuration unit is used to integrate the list name and basic information of the source list of unstructured data into configuration information, wherein the time parameters include creation time and update time.

[0085] Optionally, as an embodiment of the present invention, the execution module includes:

[0086] A configuration reading unit is used for the executor to read configuration information of target unstructured data from a configuration list, wherein the target unstructured data is a task object storing the task;

[0087] The data storage unit is used to parse the resource address from the configuration information, obtain the target unstructured data from the resource address, and upload the target unstructured data to the object storage platform.

[0088] The parameter recording unit is used to retrieve the storage information table of the target unstructured data based on the configuration information, obtain the parameter value corresponding to the object storage information item in the storage information table from the object storage platform, and import the parameter value into the storage information table.

[0089] Figure 3This is a schematic diagram of the structure of a terminal 300 provided in an embodiment of the present invention. The terminal 300 can be used to execute the synchronous storage method for unstructured data provided in the embodiment of the present invention.

[0090] The terminal 300 may include a processor 310, a memory 320, and a communication module 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0091] The memory 320 can be used to store the execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile memory terminal or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the terminal 300 is able to perform some or all of the steps in the above method embodiments.

[0092] The processor 310 serves as the control center of the storage terminal, connecting various parts of the electronic terminal via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic terminal and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0093] The communication module 330 is used to establish a communication channel, enabling the storage terminal to communicate with other terminals. It receives user data sent by other terminals or sends user data to other terminals.

[0094] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0095] Therefore, this invention, by setting a configuration file, periodically synchronizes and stores unstructured data, and records the storage parameters of the unstructured data in the object storage platform. Based on these storage parameters, the unstructured data can be directly retrieved from the object storage platform. This invention achieves unified storage and management of unstructured data through periodic synchronization of unstructured data with OSS object storage tasks, facilitating demonstrations based on actual business needs. The technical effects achieved in this embodiment can be found in the description above, and will not be repeated here.

[0096] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0097] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0098] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0099] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0101] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the invention should also be covered within the protection scope of the invention. Therefore, the protection scope of the invention should be determined by the scope of the claims.

Claims

1. A method for synchronously storing unstructured data, characterized in that, The method comprises the following steps: acquiring source information of newly added unstructured data and saving the source information to a source list; creating a storage information table, the storage information table comprising object storage information items that need to be recorded; generating configuration information based on the source list and basic information of the unstructured data, and saving the configuration information to a task configuration list, the basic information comprising a name, a time parameter, and a timing strategy; setting a scheduler to generate a storage task for saving the unstructured data to an object storage based on the configuration list; calling an executor to execute the storage task and acquiring structured data corresponding to the object storage information items, and saving the structured data; calling an executor to execute the storage task and acquiring structured data corresponding to the object storage information items, and saving the structured data, comprising: the executor reading configuration information of target unstructured data from the configuration list, the target unstructured data being a task object of the storage task; parsing a resource address from the configuration information, acquiring the target unstructured data from the resource address, and uploading the target unstructured data to an object storage platform; based on the configuration information, calling a storage information table of the target unstructured data, acquiring parameter values corresponding to object storage information items of the storage information table from the object storage platform, and importing the parameter values into the storage information table.

2. The method of claim 1, wherein, acquiring source information of newly added unstructured data and saving the source information to a source list, comprising: acquiring source information of newly added unstructured data, the source information comprising an id, a resource address, an addition time, user information, device information, and type information; creating a source list and importing the source information into the source list; allocating a list name to the source list and saving the source list to a first path.

3. The method of claim 2, wherein, generating configuration information based on the source list and basic information of the unstructured data, and saving the configuration information to a task configuration list, the basic information comprising a name, a time parameter, and a timing strategy, comprising: integrating the list name and the basic information of the source list of the unstructured data into configuration information, the time parameter comprising a creation time and an update time.

4. A system for synchronously storing unstructured data, characterized by The method comprises the following steps: a first creation module for acquiring source information of newly added unstructured data and saving the source information to a source list; a second creation module for creating a storage information table, the storage information table comprising object storage information items that need to be recorded; a third creation module for generating configuration information based on the source list and basic information of the unstructured data, and saving the configuration information to a task configuration list, the basic information comprising a name, a time parameter, and a timing strategy; a scheduling module for setting a scheduler to generate a storage task for saving the unstructured data to an object storage based on the configuration list; an execution module for calling an executor to execute the storage task and acquiring structured data corresponding to the object storage information items, and saving the structured data; the execution module comprising: A configuration reading unit is configured to read configuration information of target unstructured data from a configuration list, the target unstructured data being a task object of a storage task; A data storage unit is configured to parse a resource address from the configuration information, acquire the target unstructured data from the resource address, and upload the target unstructured data to an object storage platform; A parameter recording unit is configured to call a storage information table of the target unstructured data based on the configuration information, acquire a parameter value corresponding to an object storage information item of the storage information table from the object storage platform, and import the parameter value into the storage information table.

5. The system of claim 4, wherein, The first creating module comprises: A source acquisition unit is configured to acquire source information of the added unstructured data, the source information including an id, a resource address, an adding time, user information, device information, and type information; A list creating unit is configured to create a source list and import the source information into the source list; A list naming unit is configured to assign a list name to the source list and save the source list to a first path.

6. The system of claim 5, wherein, The third creating module comprises: A configuration unit is configured to integrate a list name and basic information of the source list of the unstructured data into configuration information, the time parameter including a creation time and an update time.

7. A terminal, characterized by comprising: The memory is configured to store a synchronization storage program of the unstructured data. The processor is configured to implement the steps of the synchronization storage method of the unstructured data when executing the synchronization storage program of the unstructured data. The readable storage medium has stored thereon a synchronization storage program of the unstructured data, and the synchronization storage program of the unstructured data implements the steps of the synchronization storage method of the unstructured data when executed by the processor.

8. A computer readable storage medium storing a computer program, characterized in that, ​

Images