Method, apparatus, device and computer readable medium for writing data
By establishing a correspondence between the database and cloud storage, and a file identifier, and by using plugins to achieve data segmentation and uploading, the problem of cumbersome write operations is solved, write efficiency is improved, and the storage solution is optimized.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAN TONGXING HENGYAO INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2022-07-05
- Publication Date
- 2026-04-14
AI Technical Summary
In existing technologies, the write operation steps when a database archives data to cloud object storage are cumbersome and inefficient, making it impossible to efficiently complete the data dumping and querying.
Establish the storage mapping between external tables in the database and storage spaces in OSS (Object Storage Service) on the cloud, as well as the mapping between local tables in the database and preset file identifiers. Implement data segmentation and uploading through plugins to simplify the writing process.
By simplifying write operations, data writing efficiency is improved, storage space usage is reduced, and a solution for separating hot and cold data storage is provided.
Smart Images
Figure CN115168376B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, device, and computer-readable medium for writing data. Background Technology
[0002] Databases have limited storage capacity, and as data accumulates, the challenge of dumping massive amounts of data often arises. Cloud-based Object Storage Service (OSS) can provide a storage location for such massive amounts of data. To dump and archive historical data, the data must first be exported to the local machine, then uploaded to OSS, and finally deleted to free up storage space. Similarly, when querying dumped data, the OSS data must first be downloaded to the local machine, and then restored to the database for querying.
[0003] In the process of realizing this invention, the inventors discovered that the prior art has at least the following problems: when the business party has a write operation on the archived data, the archived data can only be exported to the local machine, and then the write operation is performed to upload it to the cloud OSS, which is a cumbersome process with low writing efficiency. Summary of the Invention
[0004] In view of this, embodiments of the present invention provide a method, apparatus, device, and computer-readable medium for writing data, which can simplify the writing operation process and improve writing efficiency.
[0005] To achieve the above objectives, according to one aspect of the present invention, a method for writing data is provided, comprising:
[0006] Establish the storage mapping between external tables in the database and storage space in OSS (Object Storage Service) on the cloud, as well as the mapping between local tables in the database and preset file identifiers.
[0007] In response to a data write command, data is scanned in the local table of the database, and the scanned data is divided into sub-files according to the preset file identifier;
[0008] The sub-files are written into the external table of the database based on the identifier correspondence and uploaded to the storage space that stores the correspondence.
[0009] The sub-files include comma-separated value CSV files.
[0010] The method further includes:
[0011] In response to a data read command, the file with the preset file identifier is retrieved from an external table in the database based on the identifier correspondence.
[0012] Load the file with the preset identifier into the local table of the database.
[0013] The step of retrieving the file with the preset file identifier from the external table of the database in response to a data read command, based on the identifier correspondence, includes:
[0014] In response to a data read command, the data read range is obtained from the data read command;
[0015] Based on the identification correspondence, the file with the preset file identification is obtained from the data reading range of the external table of the database.
[0016] The response to the data read command includes:
[0017] Create a plugin for the database, which is used to respond to the data read command.
[0018] The response to the data write command includes:
[0019] Create a plugin for the database, which is used to respond to the data write command.
[0020] The database includes a PostgreSQL database.
[0021] According to a second aspect of the present invention, an apparatus for writing data is provided, comprising:
[0022] The module is used to establish the storage mapping between external tables in the database and storage spaces in the cloud object storage OSS, as well as the mapping between local tables in the database and preset file identifiers.
[0023] The response module is used to scan data in the local table of the database in response to a data write command, and to divide the scanned data into sub-files according to the preset file identifier;
[0024] The transmission module is used to write the sub-file into an external table of the database based on the identifier correspondence and upload it to the storage space storing the correspondence.
[0025] According to a third aspect of the present invention, an electronic device for writing data is provided, comprising:
[0026] One or more processors;
[0027] Storage device for storing one or more programs.
[0028] When the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.
[0029] According to a fourth aspect of the present invention, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described above.
[0030] One embodiment of the above invention has the following advantages or beneficial effects: It establishes a storage correspondence between the external tables of the database and the storage space in the cloud object storage (OSS), and an identifier correspondence between the local tables of the database and preset file identifiers; in response to a data write command, it scans data in the local tables of the database and divides the scanned data into sub-files according to the preset file identifiers; it writes the sub-files into the external tables of the database based on the identifier correspondence and uploads them to the storage space of the storage correspondence. The external and local tables of the database simplify the write operation process and improve write efficiency.
[0031] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description
[0032] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:
[0033] Figure 1 This is a schematic diagram of the main flow of the method for writing data according to an embodiment of the present invention;
[0034] Figure 2 This is a schematic diagram illustrating the establishment of a correspondence according to an embodiment of the present invention;
[0035] Figure 3 This is a schematic diagram of the data reading process according to an embodiment of the present invention;
[0036] Figure 4 This is a schematic diagram of the process for obtaining a file with a preset file identifier according to an embodiment of the present invention;
[0037] Figure 5 This is a schematic diagram illustrating the interaction between the database and OSS according to an embodiment of the present invention;
[0038] Figure 6 This is a schematic diagram of the main structure of the device for writing data according to an embodiment of the present invention;
[0039] Figure 7 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;
[0040] Figure 8 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation
[0041] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0042] Currently, PostgreSQL's Foreign Data Wrappers (FDW) use SQL to manage external tables, and by implementing the corresponding hook functions, it is possible to query and write external data.
[0043] While it's possible to archive and dump data, leveraging the stable, portable, and massive storage capabilities of object storage, the process is complex, and there are certain issues with querying the dumped data. The main drawbacks are as follows:
[0044] The data dumping process is complex, often requiring exporting PostgreSQL database data locally, then uploading it to OSS, and finally releasing the database storage space. Furthermore, the existing open-source s3_fdw plugin does not support writing to OSS.
[0045] In summary, the existing process of writing data is cumbersome.
[0046] To address the technical problem of cumbersome data writing steps, the following technical solutions from the embodiments of the present invention can be adopted.
[0047] See Figure 1 , Figure 1 This is a schematic diagram of the main flow of a data writing method according to an embodiment of the present invention. Data is written to OSS by storing and identifying correspondences. Figure 1 As shown, the specific steps include:
[0048] S101. Establish the storage correspondence between the external tables of the database and the storage space in OSS, as well as the identifier correspondence between the local tables of the database and the preset file identifiers.
[0049] In this embodiment of the invention, in order to write data into OSS, a mapping relationship needs to be established. That is, a storage mapping relationship needs to be established between the external tables of the database and the storage space in OSS, and an identifier mapping relationship needs to be established between the local tables of the database and the preset file identifiers. As an example, the storage space in OSS includes OSS Buckets.
[0050] In one embodiment of the present invention, the database includes a PostgreSQL database. That is, the technical solutions in this embodiment of the present invention can be applied to a PostgreSQL database.
[0051] The following example uses a PostgreSQL database as a case study.
[0052] The external data wrapper for a PostgreSQL database consists of a set of functions called by the PostgreSQL core service. The external data wrapper is responsible for retrieving data from a remote data source and returning it to the PostgreSQL executor.
[0053] The plugin in this embodiment of the invention is specifically the s3_fdw plugin. It should be noted that s3_fdw is the plugin name, and this name can be changed. Only the plugin name s3_fdw is used as an example for illustration. Furthermore, the same plugin name does not necessarily mean that the plugins have the same function.
[0054] The s3_fdw plugin is based on an external data wrapper and implements corresponding handler and validator functions in C. The handler function returns a function pointer structure pointing to callback functions, which are invoked by the planner, executor, and various maintenance commands. The validator function is responsible for verifying the validity of the process using the external data wrapper, thereby enabling PostgreSQL to read and write data to OSS based on the s3 protocol.
[0055] See Figure 2 , Figure 2 This is a schematic diagram illustrating the establishment of a correspondence according to an embodiment of the present invention. Figure 2 The process includes four steps: creating a plugin, creating a service, creating a user mapping, and creating associations. These are explained below:
[0056] Create a plugin
[0057] The PostgreSQL database loads the corresponding plugin based on the SQL, which is "CREATE EXTENSIONs3_fdw;".
[0058] Create service
[0059] Create an s3_fdw server, which corresponds to the OSS connection domain name and the corresponding OSS bucket name. The corresponding SQL is "CREATE SERVER s3_fdw_server FOREIGN DATA WRAPPER s3_fdw options(host'xxxxxxxx',bucket'xxxxxxxx');".
[0060] The purpose of creating the service is to obtain the address of the OSS service in order to associate the external tables of the PostgreSQL database with the OSS service.
[0061] Create user mapping
[0062] This establishes a mapping between the currently logged-in user in the PostgreSQL database and the user on OSS. The corresponding SQL is "CREATE USER MAPPING FOR CURRENT_USER SERVER s3_fdw_server OPTIONS(access_key_id'xxxxxxxx',secret_access_key'xxxxxxxx');". Here, access_key_id and secret_access_key correspond to the OSS user access information, similar to a username and password.
[0063] The purpose of creating user mappings is to execute SQL through database accounts to use the OSS service. Specifically, the account used to log in to the database and the account used to upload data to OSS are different. By establishing an account mapping relationship, SQL can be executed through the database account, thereby using the OSS service.
[0064] Create relationships
[0065] To establish a relationship between an external table in a PostgreSQL database and a file on OSS, the corresponding SQL is: "CREATE FOREIGN TABLE oss(id integer,first_name character varying,last_name character varying)SERVER s3_fdw_server OPTIONS(dir's3_fdw_dir / ',format'csv');".
[0066] Here, dir is a virtual file path on OSS. Under the host and bucket corresponding to s3_fdw_server, all files with the prefix "s3_fdw_dir / " can be associated with the external table of this PostgreSQL database.
[0067] During the creation of relationships, a storage mapping is established between the external tables in the database and the storage space in OSS to store data in OSS. The default file identifier is s3_fdw_dir / . An identifier mapping is also established between the external tables in the database and the default file identifier to write data to the external tables in the database.
[0068] S102. In response to the data write command, scan the data in the local table of the database and divide the scanned data into sub-files according to the preset file identifier.
[0069] In this embodiment of the invention, in order to write a file to OSS, it is necessary to receive a data write command input by the user. As an example, the data write command is input into the database client via an SQL statement. That is, an SQL statement is input into the database client.
[0070] To ensure the database fully understands SQL statements, an abstract syntax tree (AST) can be constructed using metadata from external tables. This AST is then optimized using predefined rules to obtain an optimized query plan tree. Furthermore, a predefined file identifier can be derived from the optimized query plan tree. For example, the predefined file identifier is 001.
[0071] In response to a data write command, the system can scan the local tables of the database according to a preset file identifier. The purpose of scanning the database is to determine the uploaded file from the local tables. The uploaded file includes the data being written.
[0072] In embodiments of the present invention, to facilitate writing data to OSS, scanned files can be divided into sub-files. As an example, scanned data is divided into multiple sub-files according to a preset file identifier, and the size of each sub-file is a preset size. That is, the scanned file is divided into multiple sub-files according to a preset size. As an example, the preset size is 1GB. As another example, scanned data can be divided according to a preset size in the data write instruction.
[0073] Multiple sub-files can be uploaded to OSS separately. The purpose of splitting the scanned data into multiple sub-files is to prevent individual files from becoming too large and inconvenient to download. Sub-files are relatively small in size, making it easier for users to search for specific files and improving search efficiency.
[0074] In this embodiment of the invention, the sub-file includes a comma-separated value (CSV) file. The CSV file can be read by multiple servers. The fact that the file is saved in CSV format means that any destination can read the data and extract information from it.
[0075] In the actual implementation, the default file identifier is s3_fdw. The default capacity is s3_fdw.put_file_size in the data write command. Using the default capacity, the scanned data is split into multiple CSV files.
[0076] In one embodiment of the present invention, the technical solution of the present invention can be implemented by a plugin, thereby simplifying the data writing steps. That is, a database plugin is created, which is used to respond to data write commands.
[0077] S103. Write the sub-files into the external table of the database based on the identifier correspondence and upload them to the storage space that stores the correspondence.
[0078] External tables in a database act as an intermediary between the database and OSS. Through external tables, sub-files of local tables in the database can be written to the OSS storage space. As an example, executing the SQL statement "INSERT INTO oss SELECT * FROM local" archives and dumps local table data to OSS.
[0079] Specifically, the system scans local table data using SQL commands, splits it into sub-files of a certain size, and writes the data to an external table in the database based on the identifier mapping. Thus, the sub-files are written to the external table according to the identifier mapping. As an example, triggering the handler function of the s3_fdw plugin writes the data scanned from the PostgreSQL external table to the external table in the database.
[0080] It's understandable that external tables in a database can synchronize information with OSS. External tables establish a connection between the database's local data and OSS.
[0081] Then, the data in the local tables of the database is uploaded to the storage space that stores the corresponding relationships, using the external tables of the database as the transmission medium. The storage space is located in OSS, and the data is then written to OSS.
[0082] Specifically, during the write process, because the external table in the database has a virtual file path prefix, s3_fdw generates filenames with the prefix "s3_fdw_dir / ", writes the data in CSV format, and uploads it to the storage space that stores the corresponding relationship so that the written data can be queried and matched later. Thus, the data writing process can be completed with a simple SQL operation.
[0083] In one embodiment of the present invention, data can be read after it has been written to OSS. Figure 1 In this embodiment, based on the data written, data reading, i.e., data querying, is implemented. See also... Figure 3 , Figure 3 This is a schematic diagram of the data reading process according to an embodiment of the present invention. Specifically, it includes the following steps:
[0084] S301. In response to the data read command, retrieve the file with the preset file identifier from the external table of the database according to the identifier correspondence.
[0085] Currently, when a business needs to query written data, all corresponding data must be restored to the database each time before the query can be performed, which is a cumbersome process. The technical solution in the following embodiments allows loading pre-identified files into the local database without restoring all data locally.
[0086] In this embodiment of the invention, in order to read files from OSS, it is necessary to receive data read instructions input by the user. As an example, the data read instruction is input into the database client via an SQL statement. That is, an SQL statement is entered into the database client.
[0087] To ensure the database fully understands SQL statements, an abstract syntax tree (AST) can be constructed using metadata from external tables. This AST is then optimized using predefined rules to obtain an optimized query plan tree. Finally, a predefined file identifier can be derived from the optimized query plan tree.
[0088] In one embodiment of the present invention, the technical solution of the present invention can be implemented by a plugin, thereby simplifying the data reading steps. That is, a database plugin is created, which is used to respond to data reading commands.
[0089] In response to a data read command, the file is retrieved from the external table of the database according to a preset file identifier and its mapping. Specifically, since the external table of the database is already associated with the file on OSS, by querying the external table of the database and executing the SQL "SELECT * FROM oss", the corresponding handler function of the s3_fdw plugin is triggered. The s3_fdw plugin will then filter and match the files on OSS to obtain files with the prefix "s3_fdw_dir / ".
[0090] See Figure 4 , Figure 4 This is a schematic diagram illustrating the process of obtaining a file with a preset file identifier according to an embodiment of the present invention. Specifically, it includes the following steps:
[0091] S401, In response to a data read command, obtain the data read range from the data read command.
[0092] To improve data reading speed, files can be retrieved based on a data read range. Specifically, in response to a data read command, the data read range is obtained from the data read command.
[0093] As an example, in response to a data read command, the s3_fdw plugin provides the parameter "s3_fdw.read_files". By setting this parameter, you can specify the range of files to read on OSS, narrow down the OSS file matching range, and quickly obtain the data read response.
[0094] S402. Based on the identifier correspondence, retrieve the file with the preset file identifier from the data reading range of the external table in the database.
[0095] Based on the identifier mapping, the file with the preset file identifier is retrieved from the data read range of the external table in the database, rather than retrieving the file with the preset file identifier from the external table of the database.
[0096] exist Figure 4 In this embodiment, a data reading range is used to narrow down the matching range of the acquired file, thereby improving the speed of data reading.
[0097] S302. Load the file with the preset identifier into the local table of the database.
[0098] Load files with preset identifiers into a local table in the database to enable reading and querying of data in OSS.
[0099] exist Figure 3 In one embodiment, in response to a data read instruction, a file is loaded into a local table in the database based on an identifier correspondence.
[0100] In the embodiments of the present invention described above, a storage mapping relationship is established between the external tables of the database and the storage space in the cloud object storage (OSS), and an identifier mapping relationship is established between the local tables of the database and preset file identifiers. In response to a data write command, data is scanned in the local tables of the database, and the scanned data is divided into sub-files according to the preset file identifiers. The sub-files are written into the external tables of the database based on the identifier mapping relationship and uploaded to the storage space of the storage mapping relationship. By using the external and local tables of the database, the write operation process can be simplified, and write efficiency can be improved.
[0101] See Figure 5 , Figure 5 This is a schematic diagram illustrating the interaction between the database and OSS according to an embodiment of the present invention. Figure 5The database can be directly uploaded to OSS via a plugin that allows data to be written using SQL. The plugin exports the data in CSV format, meaning any destination service can read the data and extract the desired information.
[0102] Figure 5 The database also allows for quick querying of data dumped to OSS via plugins. It also supports persistent storage of data from OSS to the local database.
[0103] The technical solution adopted in this invention greatly improves the data flow capability of the database, reduces storage space occupation, and provides a new solution for the storage separation of hot and cold data.
[0104] See Figure 6 , Figure 6 This is a schematic diagram of the main structure of a data writing device according to an embodiment of the present invention. The data writing device can implement a data writing method, such as... Figure 6 As shown, the data writing device 600 specifically includes:
[0105] Module 601 is used to establish the storage correspondence between the external tables of the database and the storage space in the cloud object storage OSS, as well as the identifier correspondence between the local tables of the database and the preset file identifiers.
[0106] The response module 602 is used to scan data in the local table of the database in response to a data write command, and to divide the scanned data into sub-files according to the preset file identifier;
[0107] The transmission module 603 is used to write the sub-file into an external table of the database based on the identifier correspondence and upload it to the storage space storing the correspondence.
[0108] In one embodiment of the invention, the sub-file includes a comma-separated value CSV file.
[0109] In one embodiment of the present invention, the transmission module 603 is further configured to, in response to a data reading instruction, obtain the file with the preset file identifier from an external table of the database according to the identifier correspondence;
[0110] Load the file with the preset identifier into the local table of the database.
[0111] In one embodiment of the present invention, the transmission module 603 is specifically used to obtain a data reading range from the data reading instruction in response to the data reading instruction;
[0112] Based on the identification correspondence, the file with the preset file identification is obtained from the data reading range of the external table of the database.
[0113] In one embodiment of the present invention, the response module 602 is further configured to create a plugin for the database, the plugin being configured to respond to the data read instruction.
[0114] In one embodiment of the present invention, the database module 601 is specifically used to create a plugin for the database, the plugin being used to respond to the data write instruction.
[0115] In one embodiment of the present invention, the database includes a PostgreSQL database.
[0116] Figure 7 An exemplary system architecture 700 is shown, in which a method or apparatus for writing data can be applied according to embodiments of the present invention.
[0117] like Figure 7 As shown, system architecture 700 may include terminal devices 701, 702, and 703, a network 704, and a server 705. Network 704 serves as the medium for providing communication links between terminal devices 701, 702, and 703 and server 705. Network 704 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.
[0118] Users can use terminal devices 701, 702, and 703 to interact with server 705 via network 704 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 701, 702, and 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).
[0119] Terminal devices 701, 702, and 703 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.
[0120] Server 705 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 701, 702, and 703 (for example only). The backend management server can analyze and process data such as received product information query requests, and feed back the processing results (such as target push information, product information - for example only) to the terminal devices.
[0121] It should be noted that the data writing method provided in the embodiments of the present invention is generally executed by server 705, and correspondingly, the data writing device is generally disposed in server 705.
[0122] It should be understood that Figure 7 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.
[0123] The following is for reference. Figure 8 It shows a schematic diagram of the structure of a computer system 800 suitable for implementing a terminal device of the present invention. Figure 8 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0124] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the system 800. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0125] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.
[0126] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs the functions defined above in the system of this invention.
[0127] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.
[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0129] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a creation module, a response module, and a transmission module. The names of these modules do not necessarily limit the module itself; for example, the creation module can also be described as "used to establish the storage correspondence between external tables of the database and storage spaces in the cloud object storage OSS, and the identifier correspondence between local tables of the database and preset file identifiers."
[0130] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, cause the device to include:
[0131] Establish the storage mapping between external tables in the database and storage space in OSS (Object Storage Service) on the cloud, as well as the mapping between local tables in the database and preset file identifiers.
[0132] In response to a data write command, data is scanned in the local table of the database, and the scanned data is divided into sub-files according to the preset file identifier;
[0133] The sub-files are written into the external table of the database based on the identifier correspondence and uploaded to the storage space that stores the correspondence.
[0134] According to the technical solution of this invention, a storage correspondence is established between the external tables of the database and the storage space in the cloud object storage OSS, as well as an identifier correspondence between the local tables of the database and preset file identifiers. In response to a data write command, data is scanned in the local tables of the database, and the scanned data is divided into sub-files according to the preset file identifiers. The sub-files are written into the external tables of the database based on the identifier correspondence and uploaded to the storage space of the storage correspondence. The use of external and local tables in the database simplifies the write operation process and improves write efficiency.
[0135] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A method for writing data, characterized in that, include: Establish the storage correspondence between the external tables of the database and the storage space in the cloud object storage OSS, as well as the identifier correspondence between the external tables of the database and the preset file identifiers; In response to a data write command, data is scanned in the local table of the database, and the scanned data is divided into sub-files according to the preset file identifier; The sub-files are written into the external table of the database based on the identifier correspondence and uploaded to the storage space storing the correspondence. The method further includes: In response to a data read command, the data read range is obtained from the data read command; Based on the identification correspondence, the file with the preset file identifier is obtained from the data reading range of the external table of the database; Load the file with the preset identifier into the local table of the database.
2. The method for writing data according to claim 1, characterized in that, The sub-files include comma-separated value CSV files.
3. The method for writing data according to claim 1, characterized in that, The response to the data read command includes: Create a plugin for the database, which is used to respond to the data read command.
4. The method for writing data according to claim 1, characterized in that, The response to the data write command includes: Create a plugin for the database, which is used to respond to the data write command.
5. The method for writing data according to claim 1, characterized in that, The database includes a PostgreSQL database.
6. A device for writing data, characterized in that, include: The module is used to establish the storage correspondence between the external tables of the database and the storage space in the cloud object storage OSS, as well as the identifier correspondence between the external tables of the database and the preset file identifiers. The response module is used to scan data in the local table of the database in response to a data write command, and to divide the scanned data into sub-files according to the preset file identifier; The transmission module is used to write the sub-file into an external table of the database based on the identifier correspondence and upload it to the storage space storing the correspondence; In addition, in response to a data read instruction, a data read range is obtained from the data read instruction; and based on the identifier correspondence, a file with the preset file identifier is obtained from the data read range of the external table of the database; Load the file with the preset identifier into the local table of the database.
7. An electronic device for writing data, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.
8. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-5.