Object acquisition method and device, equipment and storage medium
By starting the data acquisition process in the first database, obtaining and storing objects from the second database to shared memory, the problem of low performance of a single process is solved, multi-process interaction and multi-table connection query are realized, and data stored in cloud object are improved Access performance and security.
Patent Information
- Application Number
- CN202310534258.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-12
- Publication Date
- 2024-11-12
AI Technical Summary
During data query and processing, due to poor network access and data processing performance of single process, the data performance in querying target database is poor, and the mapping of multiple files corresponding to one local database table is not possible, further reducing the Performance.
A method for obtaining an object is proposed, by starting a data acquisition process in the first database, obtaining target objects from the second database and storing them into shared memory, realizing multi-process interaction to improve data access efficiency, and establishing external tables through preset mapping information The relationship with objects supports mapping of multiple objects corresponding to a table.
It improves the data access performance and efficiency in the second database, realizes accessing data while downloading and processing, and supports multi-table connection query, improving the data access accuracy and security of object storage.
Smart Images

Figure CN118939689A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer technology, and specifically relates to an object acquisition method, device, equipment and storage medium. Background Art
[0002] Cloud Object Storage (COS) is a distributed storage service that can store massive amounts of data. Users can store and process large amounts of enterprise data in batches at any time through the Internet. It has the characteristics of high scalability, low cost, reliability and security.
[0003] The related technology extracts the virtual connection address from the database instance access request, obtains the physical connection address that has a mapping relationship with the virtual connection address, and accesses the target database instance through the database plug-in that matches the database type according to the physical connection address. However, the related technology uses a single process for network access and has poor data processing performance, which reduces the performance of querying data in the target database; and a file in the object storage in the related technology can only correspond to a table in a local database, and it is impossible to achieve the mapping of multiple files in the object storage to a table in a local database, thereby further reducing the performance of querying data in the target database. Summary of the invention
[0004] In order to solve the above technical problems, the present application provides an object acquisition method, device, equipment and storage medium.
[0005] On the one hand, the present application proposes an object acquisition method, the method comprising:
[0006] In response to an object acquisition request triggered by a target account, acquiring a target external table corresponding to the object acquisition request from a first database;
[0007] Querying the target external table to determine at least one target object in the second database corresponding to the target external table based on preset mapping information; the preset mapping information represents a mapping relationship between the external table and the object in the second database;
[0008] Initiate a data acquisition process in the first database, so that the data acquisition process acquires at least one target object from the second database; and enable the data acquisition process to store at least one target object in a shared memory in the first database;
[0009] At least one of the target objects is obtained from the shared memory, and at least one of the target objects is sent to the target account.
[0010] On the other hand, the present application proposes an object acquisition device, the device comprising:
[0011] A response module, configured to respond to an object acquisition request triggered by a target account and acquire a target external table corresponding to the object acquisition request from the first database;
[0012] A query module, configured to query the target external table to determine at least one target object in the second database corresponding to the target external table based on preset mapping information; the preset mapping information represents a mapping relationship between the external table and the object in the second database;
[0013] a start module, configured to start a data acquisition process in the first database, so that the data acquisition process acquires at least one of the target objects from the second database; and to enable the data acquisition process to store at least one of the target objects in a shared memory in the first database;
[0014] The acquisition and sending module is used to acquire at least one of the target objects from the shared memory and send at least one of the target objects to the target account.
[0015] In an optional embodiment, the device further comprises:
[0016] a target external service acquisition module, used to acquire a target external service corresponding to the target external table; the target external service stores at least one target storage location of the target object in the second database, and target login information of the target account in the second database;
[0017] A login location acquisition module, used to acquire the target login information and the target storage location from the target external service;
[0018] The target login information is used to log into the second database, and the target storage location is used to obtain at least one target object.
[0019] In an optional embodiment, the startup module includes:
[0020] A wrapper plug-in acquisition unit, used to acquire an external data wrapper plug-in corresponding to the first database; the external data wrapper plug-in is used to access the second database in the first database;
[0021] The first startup unit is used to start the data acquisition process in the first database, so that the data acquisition process uses the external data wrapper plug-in, logs in to the second database through the login information, and obtains at least one target object from the target storage location in the second database.
[0022] In an optional embodiment, the first starting unit is further used to start the data acquisition process in the first database, so that the data acquisition process sends the object acquisition request to the second database based on the external data wrapper plug-in; the object acquisition request carries the target login information and the target storage location; and the data acquisition process receives at least one target object sent by the second database;
[0023] Among them, at least one of the target objects is obtained from the target storage location when the second database responds to the object acquisition request and determines that the identity information of the target account satisfies a preset condition.
[0024] In an optional embodiment, when the target external table is a table connected by multiple tables, the query module and the start module include:
[0025] a target external table query unit, configured to query the target external table to determine a current object in the second database corresponding to the target external table based on preset mapping information;
[0026] A second starting unit, configured to start the data acquisition process in the first database, so that the data acquisition process acquires the current object from the second database;
[0027] a repeating unit, configured to repeat the querying of the target external table, to enable the data acquisition process to acquire the current object from the second database, until the number of repetitions matches the size of the target external table;
[0028] A determination unit is used to determine the current object obtained after the repetitive process is completed as at least one of the target objects.
[0029] In an optional embodiment, the device further comprises:
[0030] An external service determination module, used to determine an external service; the external service stores a storage location of an object in the second database and login information of an account in the second database; the storage location includes the target storage location, and the login information includes the target login information;
[0031] an external table generation module, configured to generate, in the first database, an external table corresponding to at least one object in the second database based on the external service; and to establish the preset mapping information representing the mapping relationship between the external table and the object in the second database;
[0032] Accordingly, the response module includes:
[0033] The external table acquisition unit is used to acquire the target external table corresponding to the object identification information from the external table in the first database.
[0034] In an optional embodiment, the device further comprises:
[0035] A calling module, used to call a preset encryption plug-in corresponding to the first database;
[0036] An encryption module, used to encrypt the login information based on the preset encryption plug-in to obtain an encrypted login information result;
[0037] A storage module, used for storing the login information encryption result in an encryption information storage table;
[0038] The login information encryption result stored in the encryption information storage table is used to obtain at least one of the target objects from the second database.
[0039] In an optional embodiment, the login location acquisition module includes:
[0040] The encryption result obtaining unit is used to obtain the login information encryption result from the encryption information storage table.
[0041] The decryption unit is used to call the preset encryption plug-in to decrypt the login information to obtain the decrypted target login information.
[0042] In an optional embodiment, the acquisition and sending module includes:
[0043] Function acquisition module, used to obtain preset hook functions;
[0044] an acquisition conversion module, configured to acquire at least one of the target objects from the shared memory through the preset hook function, and convert the format of the at least one of the target objects into a data format corresponding to the first database;
[0045] The object sending module is used to send the target object after format conversion to the target account.
[0046] On the other hand, the present application proposes an electronic device for object acquisition, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the object acquisition method as described above.
[0047] On the other hand, the present application proposes a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the object acquisition method as described above.
[0048] On the other hand, the present application proposes a computer program product, which implements the object acquisition method as described above when the computer program is executed by a processor.
[0049] The object acquisition method, apparatus, device and storage medium proposed in the embodiments of the present application obtain a target external corresponding to the object acquisition request from a first database in response to an object acquisition request triggered by a target account, query the target external table to determine at least one target object corresponding to the target external table in a second database based on preset mapping information, start a data acquisition process in the first database so that the data acquisition process obtains at least one target object from the second database; and enables the data acquisition process to store at least one target object in a shared memory in the first database, obtain at least one target object from the shared memory, and send at least one target object to the target account. , a new parallel framework is implemented, which can additionally start a data acquisition process, through which the target object is acquired from the second database, and the target object is stored in the shared memory of the first database. The original process acquires the target object from the shared memory, thereby realizing downloading and processing the target object, and further realizing access to the data stored in the second database through multi-process interaction, thereby improving the access performance and access efficiency of the data in the second database; in addition, since the target external table corresponds to at least one target object in the second database, the mapping of multiple objects in the object storage to a table in the first database is realized, thereby further improving the access performance and access efficiency of the data in the second database. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0051] Figure 1 The figure is a schematic diagram of an implementation environment of an object acquisition method according to an exemplary embodiment.
[0052] Figure 2 is a flow chart of an object acquisition method according to an exemplary embodiment. Figure 1 .
[0053] Figure 3 is a flow chart of an object acquisition method according to an exemplary embodiment. Figure 2 .
[0054] Figure 4 is a flow chart of an object acquisition method according to an exemplary embodiment. Figure 3 .
[0055] Figure 5 The present invention is a schematic diagram of a process of encrypting login information according to an exemplary embodiment.
[0056] Figure 6 The figure is a block diagram of an object acquisition device according to an exemplary embodiment.
[0057] Figure 7 A hardware structure block diagram of a server provided according to an exemplary embodiment. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.
[0060] In order to make the purpose, technical solution and advantages disclosed in the embodiments of the present application more clearly understood, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the present application and are not used to limit the embodiments of the present application.
[0061] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.
[0062] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing. Cloud technology includes cloud basic technology and cloud applications. Furthermore, cloud basic technology can further include cloud computing, cloud storage, database, big data, etc. Cloud applications can further include medical cloud, cloud Internet of Things, cloud security, public cloud, private cloud, etc.
[0063] In short, a database can be seen as an electronic filing cabinet - a place where electronic files are stored, and users can add, query, update, delete, and other operations on the data in the files. The so-called "database" is a collection of data that is stored together in a certain way, can be shared with multiple users, has as little redundancy as possible, and is independent of the application.
[0064] A database management system (DBMS) is a computer software system designed for managing databases. It generally has basic functions such as storage, retrieval, security, and backup. Database management systems can be classified according to the database model it supports, such as relational, Extensible Markup Language (XML); or according to the type of computer it supports, such as server clusters, mobile phones; or according to the query language used, such as Structured Query Language (SQL); or according to the performance focus, such as maximum scale, maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMS can cross categories, for example, supporting multiple query languages at the same time.
[0065] Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.
[0066] Public Cloud usually refers to a cloud provided by a third-party provider for users to use. Public Cloud is generally available through the Internet and may be free or low-cost. The core attribute of Public Cloud is shared resource services. There are many instances of this cloud, which can provide services throughout the open public network today.
[0067] It should be noted that in the specific implementation of this application, related data such as user information (for example, target account, login information, etc.) is involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0068] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of an object acquisition method according to an exemplary embodiment. Figure 1 As shown, the implementation environment may include at least a client 01 and a server 02. The client 01 and the server 02 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.
[0069] Specifically, the server 02 can be used to receive an object acquisition request triggered by the target account 01 of the client, acquire the target object, and send the target object to the target account. Optionally, the server 02 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0070] Specifically, the client 01 can be used to send an object acquisition request to the server 02 and receive the target object sent by the server. Optionally, the client 01 can include but is not limited to a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc.
[0071] It should be noted that Figure 1This is just an example. In other scenarios, other implementation environments may also be included.
[0072] In order to facilitate understanding of the technical solutions and technical effects of the embodiments of the present application, the technical terms used in the embodiments of the present application are first explained:
[0073] CDB: A ready-to-use, stable, reliable, and elastically scalable online database service. Based on the different database engines provided, it can be divided into CDB For PostgreSQL, CDB For Mysql, CDB For SQL Server, etc. Among them, PostgreSQL is an object-relational database management system (ORDBMS). Mysql and SQL Server both refer to a relational database management system.
[0074] COS: Cloud Object Storage (COS) is a distributed storage service that can store massive amounts of data for enterprises and individual developers. Users can store and process large amounts of enterprise data in batches at any time through the Internet. It has the characteristics of high scalability, low cost, reliability and security.
[0075] FDW: A foreign data wrapper plug-in for the PostgreSQL database, which can be used to access the content of other database instances or services, mainly including mysql_fdw, redis_fdw, www_fdw, etc. Among them, mysql_fdw refers to the foreign data wrapper plug-in for accessing Mysql, and www_fdw refers to the foreign data wrapper plug-in for accessing the World Wide Web (www).
[0076] Bucket: Bucket is used to store objects in COS. Multiple objects can be stored in a bucket. Bucket name is composed of user-defined string and system-generated numeric string linked by hyphens to ensure that the bucket is globally unique.
[0077] Shm_mq: Shared memory message queue is a data structure in the PostgreSQL kernel. It stores data that need to be synchronized in shared memory, completes resource control of critical sections through locks, and ultimately realizes data sharing among multiple processes.
[0078] Parallel framework: A framework implemented in the PostgreSQL kernel that allows the kernel to schedule, process, and synchronize data for parallel algorithms.
[0079] Figure 2 is a flow chart of an object acquisition method according to an exemplary embodiment. Figure 1 This method can be used for Figure 1 In the implementation environment of the embodiment. This specification provides the method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on routine or non-creative work. The order of steps listed in the embodiment is only one way of executing the steps among many steps, and does not represent the only execution order. When the actual system or server product is executed, it can be executed in the order of the method shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include:
[0080] S101. In response to an object acquisition request triggered by a target account, acquire a target external table corresponding to the object acquisition request from a first database.
[0081] Specifically, the target account may be an account for which objects in the second database need to be queried.
[0082] Optionally, the first database may be considered as a local database, which may be any type of database, for example, a relational database. Exemplarily, when the first database is a relational database, it may be PostgreSQL.
[0083] Optionally, the target external table is a table in the first database, and there is a mapping relationship between the target external table and the object in the second database. It should be noted that the target external table includes structural information of the object in the second database, but does not include data of the object in the second database.
[0084] Optionally, the second database may be an external database. Exemplarily, it may be a remote file server, such as a cloud object storage (COS).
[0085] When the first database is PostgreSQL and the second database is COS, the object acquisition method provided in the embodiment of the present application is equivalent to a solution of using PostgreSQL in a public cloud to access files in COS.
[0086] S103. Query the target external table to determine at least one target object in the second database corresponding to the target external table based on preset mapping information; the preset mapping information represents a mapping relationship between the external table and the object in the second database.
[0087] Optionally, preset mapping information of a mapping relationship between a target external table and an object in the second database may be pre-established, and during the query process, at least one target object in the second database corresponding to the target external table may be determined based on the preset mapping information.
[0088] It should be noted that a target external table in the first database can correspond to an object in the second database, or to at least two objects in the second pair of databases, thereby supporting the mapping of multiple objects in the object storage to an external table in the first database, thereby improving the access performance and efficiency of the data in the object storage.
[0089] It should be noted that the objects in the second database may refer to: files, data, etc.
[0090] S105. Start a data acquisition process in the first database, so that the data acquisition process acquires at least one target object from the second database; and so that the data acquisition process stores the at least one target object in the shared memory in the first database.
[0091] Optionally, this embodiment can implement a new parallel framework, which can also pull up an additional data acquisition process in the first database in addition to the existing process, and obtain at least one target object from the second database through the additional data acquisition process. And store the at least one target object in the dynamic shared memory (Shm) in the first database through the additional data acquisition process. The dynamic shared memory can be registered in the kernel of the first database when the parallel framework is initialized.
[0092] Exemplarily, the data acquisition process may be a bgworker, and the data acquisition request may be a data download request. For example, when the second database is COS and the first database is PostgreSQL, the parallel framework may also pull up an additional bgworker in the first database in addition to the existing process, and download at least one target object from COS through the additionally pulled up bgworker. The additionally pulled up bgworker also stores at least one target object in the dynamic shared memory (Shm) in the first database.
[0093] S107. Obtain at least one target object from the shared memory, and send at least one target object to the target account.
[0094] Continuously, the original process and the additional process can share the dynamic shared memory. The process of these two processes sharing the dynamic shared memory can be abstracted as a single producer and single consumer problem.
[0095] Among them, single producer and single consumer means that there is one producer and one consumer. The producer controls the operation of entering the message queue by writing the index, and the consumer controls the operation of exiting the message queue by reading the index. Among them, the message queue can be a shared memory message queue (Shm_mq). The producer keeps putting products into the message queue; the consumer takes products from the message queue; the capacity of the message queue is limited and can only accommodate a certain number of products; if the producer produces products too fast, it needs to wait for the consumer to take the product, and the message queue is not empty before it can continue to put new products into the product library; on the contrary, if the consumer takes the product too fast, it may face the situation that there is no product available in the message queue. At this time, it needs to wait for the producer to put a product before the consumer can continue to work.
[0096] Since the original process and the additionally started process can share the dynamic shared memory, the original process can obtain the downloaded target object from the dynamic shared memory and send the target object to the target account.
[0097] In the embodiment of the present application, a data acquisition process is started in the first database so that the data acquisition process acquires at least one target object from the second database; and the data acquisition process is enabled to store at least one target object in the shared memory in the first database, acquire at least one target object from the shared memory, and send at least one target object to the target account, thereby realizing a new parallel framework. The parallel framework can additionally start a data acquisition process, acquire the target object from the second database through the data acquisition process, and store the target object in the shared memory of the first database. The original process acquires the target object from the shared memory, thereby realizing downloading and processing of the target object, and further realizing access to the data stored in the object storage through multi-process interaction, thereby improving the access performance and access efficiency of the data in the object storage; in addition, since the target external table corresponds to at least one target object in the second database, the mapping of multiple objects in the object storage to a table in the first database is realized, thereby further improving the access performance and access efficiency of the data in the second database.
[0098] Figure 3 is a flow chart of an object acquisition method according to an exemplary embodiment. Figure 2 ,like Figure 3 As shown, in an optional embodiment, before responding to the object acquisition request triggered by the target account in the above step S101, the above method may further include:
[0099] S001. Determine an external service; the external service stores a storage location of an object in the second database and login information of an account in the second database; the storage location includes a target storage location, and the login information includes target login information.
[0100] S003. Based on the external service, generate an external table corresponding to at least one object in the second database in the first database; and establish preset mapping information representing the mapping relationship between the external table and the at least one object.
[0101] In this embodiment, an external service may be created in advance, and the domain name (host) of the second database, the storage location of objects in the second database, login information of each account in the second database, etc. may be stored in the external service.
[0102] Exemplarily, taking the second database as COS as an example, the storage location may be a Bucket in COS, and the login information is used to log in to the second database, which may include but is not limited to a user name (id) and a password (key).
[0103] After the external service is created, you can create an external table under the external service, that is, a table in the second database that needs to be accessed from the first database. The external table refers to data stored in the second database, not in the first database. When creating an external table, you need to establish a mapping relationship with the object in the second database. An external table can correspond to one or at least two objects in the second database, that is, an external table can store the structural information of one or at least two objects.
[0104] The following describes the process of creating external services and external tables, taking PostgreSQL as the first database and COS as the second database as an example:
[0105] 1) Create cos_fdw and pg_crypto plug-ins. The creation statements are as follows:
[0106] CREATE EXTENSION pgcrypto: Create a pg_crypto plug-in, where the pg_crypto plug-in refers to the encryption extension plug-in. The pg_crypto plug-in can encrypt the user's key information (for example, the user name and password of the object storage) to protect the user's privacy.
[0107] CREATE EXTENSION cos_fdw: Creates the cos_fdw plugin (foreign data wrapper plugin for accessing COS).
[0108] 2) Create ForeignServer: Create an external service. The creation statement is as follows:
[0109] CREATE SERVER COS_server FOREIGN DATA WRAPPER cos_fdwOPTIONS(host'xxxxxx.COS.ap-nanjing.myqcloud.com',bucket'xxxxxxxx',id'xxxxxxxx',key'xxxxxxxxxxx').
[0110] Among them, host'xxxxxx.COS.ap-nanjing.myqcloud.com' refers to the domain name of COS, bucket'xxxxxxxx' refers to the bucket where objects are stored in COS, id'xxxxxxxx' refers to the login user name of the account in COS, and key'xxxxxxxxxx' refers to the login password of the account in COS.
[0111] After creating the ForeignServer, you can use the statement "SELECT * FROM pg_foreign_server" to query the currently established ForeignServer and the host, bucket, id, key and other information corresponding to different servers.
[0112] 3) Create an external table. The creation statement can be as follows:
[0113] CREATE FOREIGN TABLE test_csv(word1 text OPTIONS(force_not_null'true'),word2 text OPTIONS(force_not_null'off'))SERVER COS_server OPTIONS(filepath' / test.csv',format'csv',null'NULL).
[0114] Here, csv refers to the file stored in COS, and test_csv refers to the table in PostgreSQL. It can be seen that the external table in PostgreSQL corresponds to the object (for example, file) in COS.
[0115] It should be noted that the external table is established under the corresponding external service, and there is a corresponding relationship between the external table and the corresponding external service.
[0116] Accordingly, continue as Figure 3As shown, the object acquisition request carries object identification information, and in the step S101, the step of acquiring the target external table corresponding to the object acquisition request from the first database may include:
[0117] A target external table corresponding to the object identification information is obtained from the external table in the first database.
[0118] Optionally, the object acquisition request in this embodiment can carry object identification information of the object that the target account wants to access. Since there is a mapping relationship between the external table and the object in the second database, the external table can store the structural information of the object in the second database. Therefore, according to the object identification information of the object to be accessed, the external table storing the structural information of the object can be searched from the created external tables to obtain the target external table corresponding to the object identification information.
[0119] By establishing an external service, this embodiment can accurately obtain the login information of the target account in the second database, and accurately locate the storage location of the object to be accessed in the second database, that is, accurately bind it with the storage location in the second database; by establishing an external table under the external service, it can accurately locate the specific object to be accessed, that is, accurately bind it with the object in the second database. In this way, in the object acquisition process, the access performance, access efficiency and access accuracy of the object in the second database can be effectively improved through the pre-bound storage location and object.
[0120] In an alternative embodiment, continue as Figure 3 As shown, after determining the external service in the above step S001, the above method may further include: S002. Encrypting the login information. Specifically, the encryption of the login information in S002 may include:
[0121] S0021. Call the preset encryption plug-in corresponding to the first database.
[0122] S0023. Encrypt the login information based on the preset encryption plug-in to obtain the login information encryption result.
[0123] S0025. Store the login information encryption result to the encryption information storage table.
[0124] The login information encryption result stored in the encryption information storage table is used to obtain at least one target object from the second database.
[0125] Since the login information in the Foreign Server (for example, id and key) is sensitive information, the preset encryption plug-in corresponding to the first database can be called to encrypt the login information so that the plain text cannot be seen during the query, thereby effectively protecting the account information and improving the security of object acquisition.
[0126] In the encryption process of this embodiment, the key can be obtained from the environment variable, the preset encryption plug-in (for example, pgcrypto) corresponding to the first database is called, and the login information is encrypted using the encryption algorithm and key provided by pgcrypto to obtain the login information encryption result. It should be noted that each instance has a key, which is provided by the management and control. Among them, PostgreSQL and COS can be considered as an instance.
[0127] For example, the encryption algorithm provided by pgcrypto can use symmetric encryption. For example, the encryption algorithm is pgp_sym_encrypt in pgcrypto, without any parameters; the length of the id before encryption is 36, and the length of the key is 32. The encrypted ciphertext starts with "\\xc" and will not be re-encrypted after being exported and imported.
[0128] In this embodiment, after obtaining the encrypted information storage table, the encrypted information storage table can be stored in an encrypted information storage table (for example, pg_foreign_server), and the login information encryption result stored in the encrypted information storage table is used to log in to the second database and obtain at least one target object from the second database.
[0129] It should be noted that, in addition to calling the preset encryption plug-in to encrypt the login information, the login information can also be encrypted directly in the first database using a symmetric encryption algorithm, a digital signature algorithm, etc., without calling the preset encryption plug-in.
[0130] It should be noted that when using the above "SELECT * FROM pg_foreign_server" to query, it is necessary to ensure that the displayed login information (id and key) are encrypted.
[0131] In an optional embodiment, in the above step S103, since a mapping relationship between the external table and one or at least two objects in the second database has been established, after obtaining the target external table, the target external table can be queried to determine one or at least two target objects in the second database corresponding to the target external table based on preset mapping information, that is, to locate one or at least two target objects in the second database corresponding to the target external table.
[0132] In an alternative embodiment, continue as Figure 3 As shown, before starting the data acquisition process in the first database in the above step S105, the above method may further include: S104. Acquire login information and storage location. Specifically, the above step S104 may include:
[0133] S1041. Obtain a target external service corresponding to the target external table; the target external service stores at least one target object in the target storage location in the second database, and target login information of the target account in the second database;
[0134] S1043. Obtain target login information and target storage location from the target external service;
[0135] The target login information is used to log into the second database, and the target storage location is used to obtain at least one target object.
[0136] In this embodiment, before starting the data acquisition process in the first database, a target external service corresponding to the target external table can also be acquired from a plurality of pre-established external services. The target external service stores a target storage location of at least one target object in the second database and target login information of a target account in the second database. The target login information and target storage location are acquired from the target external service.
[0137] For example, the second database is COS, the storage location can be a bucket in COS, and the login information can include but is not limited to a user name (id) and a password (key). Since there is a corresponding relationship between the external table and the external service, the target external service corresponding to the target external table can be obtained from multiple pre-established external services. The target external service stores at least one bucket of the target object in the second database, as well as the user name (id) and password (key) of the target account in the second database. The user name (id) and password (key) are obtained from the target external service.
[0138] In this way, the target external service corresponding to the target external table can be accurately obtained from the pre-created external service, thereby accurately obtaining the target login information of the target account and the target storage location of at least one target object that the target object wants to access in the second database, thereby improving the access performance, access efficiency and access accuracy of the objects in the second database.
[0139] In an optional embodiment, in the above S1043, the above acquiring the target login information from the target external service may include:
[0140] Get the login information encryption result from the encryption information storage table.
[0141] Call the preset encryption plug-in to decrypt the login information and obtain the decrypted target login information.
[0142] Since the login information encryption result is stored in the encryption information storage table in advance, the login information encryption result can be obtained from the encryption information storage table first, and then the login information is decrypted using the preset encryption plug-in and the corresponding secret key to obtain the decrypted target login information. For example, the decrypted id and key are obtained.
[0143] It should be noted that the preset encryption plug-in used in the decryption process is the same as the preset encryption plug-in used in the encryption process, for example, both are pgcrypto. The decryption algorithm used in the decryption process matches the encryption algorithm used in the encryption process.
[0144] In this way, before starting the data acquisition process, the preset encryption plug-in can be called to decrypt the login information to obtain the decrypted target login information, so that the decrypted target login information can be used to log in to the second database, effectively protecting the account information and improving the security of object acquisition.
[0145] In an alternative embodiment, continue as Figure 3 As shown, in the above step S105, the data acquisition process is started in the first database so that the data acquisition process acquires at least one target object from the second database, including:
[0146] S1051. Obtain an external data wrapper plug-in corresponding to the first database; the external data wrapper plug-in is used to access the second database from the first database.
[0147] S1053. Start a data acquisition process in the first database, so that the data acquisition process uses the external data wrapper plug-in to log in to the second database through the login information, and obtain at least one target object from the target storage location in the second database.
[0148] In this embodiment, in the process of obtaining at least one target object from the second database, the existing process can obtain a pre-created external data wrapper plug-in (i.e., a cos_fdw plug-in), and additionally pull up a data acquisition process in the first database through a parallel framework, and through the additionally pulled up data acquisition process, use the external data wrapper plug-in to log in to the second database through the login information, and obtain at least one target object from the target storage location in the second database.
[0149] For example, taking the data acquisition process as bgworker, the data acquisition request as a data download request, the second database as COS, the first database as PostgreSQL, and the target storage location as a target storage bucket as an example, the above steps S1051-S1053 are explained: the existing process first obtains the cos_fdw plug-in corresponding to the first database. In addition to the existing process, the parallel framework can also pull up an additional bgworker in the first database. The additionally pulled up bgworker uses the login information to log in to COS through the external data wrapper plug-in, locates to the target storage bucket, and downloads at least one target object from the target storage bucket.
[0150] As a result, a new parallel framework can be implemented in the kernel of the first database (e.g., PostgreSQL), and objects stored in the second database (e.g., COS) can be accessed through multi-process interaction, so that the downloaded objects can be processed while they are being downloaded, thereby improving the access performance and efficiency of the objects in the second database.
[0151] In an optional embodiment, in the above step S1053, starting the data acquisition process in the first database so that the data acquisition process uses the external data wrapper plug-in to log in to the second database through the login information and obtain at least one target object from the target storage location in the second database may include:
[0152] Initiate a data acquisition process in the first database, so that the data acquisition process sends an object acquisition request to the second database based on the external data wrapper plug-in; the object acquisition request carries target login information and target storage location; and so that the data acquisition process receives at least one target object sent by the second database;
[0153] Wherein, at least one target object is obtained from a target storage location when the second database responds to an object acquisition request and determines that identity information of a target account satisfies a preset condition.
[0154] In this embodiment, the parallel framework can start a data acquisition process in the first database, and use the external data wrapper plug-in through the data acquisition process to send an object acquisition request to the second database, and the object acquisition request carries the target login information and the target storage location. The second database responds to the object acquisition request, verifies the identity information of the target account according to the target login information, and obtains the at least one target object from the target storage location when it is determined that the identity information of the target account meets the preset conditions, and sends the at least one target object to the data acquisition process.
[0155] It should be noted that the preset condition can be set according to actual business needs and is not specifically limited here. For example, the identity information of the target account may be legal, or the matching degree between the identity information of the target account and the preset identity information may be less than a predetermined threshold.
[0156] Taking the data acquisition process as bgworker, the data acquisition request as a data download request, the second database as COS, the first database as PostgreSQL, and the target storage location as a target storage bucket as an example, the above steps are explained: in addition to the existing process, the parallel framework can also pull up an additional bgworker in the first database. The additionally pulled up bgworker uses an external data wrapper plug-in to send an object acquisition request carrying the target login information and the target storage bucket to COS. COS responds to the object acquisition request and verifies whether the identity information of the target account is legal based on the target login information. If so, COS downloads at least one target object from the target storage bucket and sends the at least one target object to the data acquisition process.
[0157] As a result, a new parallel framework can be implemented in the kernel of the first database (for example, PostgreSQL), and data stored in the second database (for example, COS) can be accessed through multi-process interaction, thereby improving the access performance and efficiency of objects in the second database; in addition, before downloading the target object in the second database, the identity information of the target account can be verified first, and the target object will be downloaded only if the verification is passed, thereby improving the security of downloading the target object.
[0158] In an optional embodiment, when the target external table is a table connected by multiple tables, in the above steps S103-S105, the above querying the target external table to determine at least one target object corresponding to the target external table in the second database based on preset mapping information; starting a data acquisition process in the first database so that the data acquisition process acquires at least one target object from the second database may include:
[0159] S201. Query the target external table to determine the current object in the second database corresponding to the target external table based on preset mapping information.
[0160] S203. Start a data acquisition process in the first database, so that the data acquisition process acquires the current object from the second database.
[0161] S205. Repeat querying the target external table, so that the data acquisition process acquires the operation of the current object from the second database, until the number of repetitions matches the size of the target external table.
[0162] S207. Determine the current object obtained after the repetitive process is completed as at least one target object.
[0163] In this embodiment, it can be determined whether the target external table needs to be re-scanned. If it does not need to be re-scanned, the target external table is scanned once, and all objects corresponding to the target external table are downloaded from the second database, that is, the above steps S103-S105 are executed. If it is necessary to re-scan, the data acquisition process can be notified through the conditional variable, and the file transfer tool (CommandLine Uniform Resource Locator, curl) is used to request resources from the second database again.
[0164] In this embodiment, there may be multiple ways to determine whether the target external table needs to be rescanned, which are not specifically limited here. In one embodiment, the optimizer in the first database may determine whether the target external table is an inner table in a multi-table connection. If so, it is determined that the target external table needs to be rescanned. If not, it is determined that the target external table does not need to be rescanned. In another embodiment, the table structure of the target external table may be analyzed. If the table structure analysis result indicates that the target external table is an inner table in a multi-table connection, it is determined that the target external table needs to be rescanned. If not, it is determined that the target external table does not need to be rescanned.
[0165] Exemplarily, the scanning process may be performed based on the rows of the target external table. Accordingly, in one embodiment, the number of repetitions matches the size of the target external table, which may mean that the number of repetitions is equal to the number of rows of the target external table. In another embodiment, the number of repetitions matches the size of the target external table, which may mean that the difference between the number of repetitions and the number of rows of the target external table is less than a preset threshold.
[0166] Assuming that the target external table has N rows, the repeated scanning process shown in steps S201 to S207 may include:
[0167] It is determined whether the target external table is an inner table in a multi-table join. If not, the above steps S103-S105 are executed.
[0168] If so, query the first row of the target external table to determine the current object corresponding to the target external table in the second database based on the preset mapping information, start the data acquisition process in the first database, and after verifying that the identity information of the target account meets the preset conditions, enable the data acquisition process to obtain the current object from the second database. Then query the second row of the target external table to determine the current object corresponding to the target external table in the second database based on the preset mapping information, start the data acquisition process in the first database, and enable the data acquisition process to obtain the current object from the second database. And so on, until the Nth row of the target external table is queried, determine the current object corresponding to the target external table in the second database based on the preset mapping information, start the data acquisition process in the first database, and enable the data acquisition process to obtain the current object from the second database. Finally, the current object obtained at the end of the query process is used as at least one target object. It should be noted that in the repeated scanning process, the identity information will only be verified during the first scan, and the identity information does not need to be verified in subsequent scans.
[0169] Among them, multi-table connection refers to connecting multiple tables according to the set connection conditions. The multi-table connection can include left connection, right connection, outer connection, inner connection, cross connection, etc.
[0170] Left join: Returns all records in the left table and records in the right table whose join fields are equal.
[0171] Right join: Returns all records in the right table and records in the left table whose join fields are equal.
[0172] Outer join: Returns all records from the left and right tables, and combines records with equivalent join fields into one.
[0173] Inner join: Returns records where the join fields in the left and right tables are equal.
[0174] Cross join: If the join condition is not specified, the Cartesian product of the left and right tables is returned, and the number of rows returned is the number of rows in the left table multiplied by the number of rows in the right table. If the join condition is specified, the records with equal join fields in the left and right tables are returned (the same effect as inner join).
[0175] Therefore, when the target external table is a table connected to multiple tables, multi-table connection queries can be supported through multi-process interactive access, and multiple objects in the second database can be supported as multiple tables for connection queries; in addition, when the target external table is a table connected to multiple tables, objects can be downloaded from the second database by repeatedly scanning multiple target external tables, which can improve the access performance, access efficiency and access accuracy of the objects in the second database.
[0176] In an optional embodiment, in the above step S107, the above step of obtaining at least one target object from the shared memory and sending at least one target object to the target account may include:
[0177] Get the default hook function.
[0178] At least one target object is obtained from the shared memory through a preset hook function, and the format of the at least one target object is converted into a data format corresponding to the first database.
[0179] Send the converted target object to the target account.
[0180] The parallel framework in this embodiment also implements a preset hook function, through which at least one target object is read from the shared memory and the format of the target object is converted into the data format corresponding to the first database. In one embodiment, the target object after the format conversion can be imported into the target external table, and the target external table with the imported target object can be sent to the target account. In another embodiment, the target object after the format conversion can be directly sent to the target account.
[0181] Exemplarily, the preset hook function may be a COPY hook function.
[0182] Since the preset hook function can be attached to the target hanging point at a certain moment, the preset hook function can accurately access the objects stored in the second database through multi-process interaction, thereby improving the access performance, access efficiency and access accuracy of the objects in the second database; in addition, since the data format in each database is inconsistent, the format of the target object read from the shared memory is converted into the format of the native database (i.e., the first database), the display and reading convenience of the target object can be improved.
[0183] In the following, taking the first database as PostgreSQL and the second database as COS as an example, the object acquisition method provided in the embodiment of the present application is described, that is, the method of using PostgreSQL in the public cloud to access files in COS is described:
[0184] Figure 4 is a flow chart of an object acquisition method according to an exemplary embodiment. Figure 3 ,like Figure 4 As shown, the object acquisition method includes:
[0185] S301. Create cos_fdw.
[0186] S303. Create external services.
[0187] S305. Create a target external table.
[0188] S307. In response to the object acquisition request triggered by the target account, initialize the parallel framework and register the dynamic shared memory in PostgreSQL.
[0189] S309. Start bgworker and launch an additional data acquisition process to access the target object in COS.
[0190] S3011.bgworker downloads the target object from COS and then stores the downloaded target object into the dynamic shared memory.
[0191] S3013. The existing process obtains at least one target object from the shared memory.
[0192] S3015. Determine whether the target external table needs to be rescanned.
[0193] S3017. If not, send an exit signal to bgworker, exit the object download process, and send the downloaded at least one target object to the target account.
[0194] S3019. If yes, send a repeat scan signal to bgworker through the condition variable, repeat the above steps S3011-S3013 until the last row in the target external table is scanned, exit the object download process, and send at least one downloaded target object to the target account.
[0195] Figure 5 is a schematic diagram of a process for encrypting login information according to an exemplary embodiment. Figure 5 As shown, encrypting the login information may include:
[0196] S401. Create cos_fdw.
[0197] S403. Create external services.
[0198] S405. Load the key from the environment variable.
[0199] S407. Call a preset encryption plug-in (eg, pgcrypto) corresponding to the first database.
[0200] S409. Encrypt the login information based on pgcrypto and the key to obtain the login information encryption result.
[0201] S4011. Store the login information encryption result to the encryption information storage table (for example, pg_foreign_server).
[0202] S4013. Query the target external table to determine at least one target object in the second database corresponding to the target external table based on preset mapping information.
[0203] S4015. Obtain the login information encryption result from the encryption information storage table, call the preset encryption plug-in to decrypt the login information, and obtain the decrypted target login information.
[0204] S4017. Start a data acquisition process in the first database, so that the data acquisition process uses the external data wrapper plug-in to log in to the second database through the login information, and obtain at least one target object from the target storage location in the second database.
[0205] The data acquisition method provided by the embodiment of the present invention has the following beneficial effects:
[0206] 1) Implement a new parallel framework to access the data stored in the second database through multi-process interaction, effectively improving the access performance, access efficiency and access accuracy of objects in the second database.
[0207] 2) Support multi-table connection query, and support multiple object storage files as multiple tables for connection query, further improving the access performance, access efficiency and access accuracy of objects in the second database.
[0208] 3) A many-to-one mapping can be performed between objects in the second database and tables in the first database. Multiple objects in the second database can correspond to the same table in the first database.
[0209] 4) No pipeline files will be left after the database crashes, thus avoiding the phenomenon that the pipeline files cannot be cleaned up and cause data file confusion.
[0210] 5) Calling the preset encryption plug-in corresponding to the first database to encrypt the login information, thereby effectively protecting the account information and improving the security of object acquisition and system security.
[0211] Figure 6 is a block diagram of an object acquisition device according to an exemplary embodiment. Figure 6 As shown, the object acquisition device includes:
[0212] A response module 501 is used to respond to an object acquisition request triggered by a target account and acquire a target external table corresponding to the object acquisition request from a first database;
[0213] A query module 503 is used to query the target external table to determine at least one target object in the second database corresponding to the target external table based on preset mapping information; the preset mapping information represents a mapping relationship between the external table and the object in the second database;
[0214] A starting module 505 is used to start a data acquisition process in the first database, so that the data acquisition process acquires at least one target object from the second database; and to enable the data acquisition process to store at least one target object in a shared memory in the first database;
[0215] The acquisition and sending module 507 is used to acquire at least one of the target objects from the shared memory and send at least one of the target objects to the target account.
[0216] In an optional embodiment, the device further comprises:
[0217] a target external service acquisition module, used to acquire a target external service corresponding to the target external table; the target external service stores at least one target storage location of the target object in the second database, and target login information of the target account in the second database;
[0218] A login location acquisition module, used to acquire the target login information and the target storage location from the target external service;
[0219] The target login information is used to log into the second database, and the target storage location is used to obtain at least one target object.
[0220] In an optional embodiment, the startup module includes:
[0221] A wrapper plug-in acquisition unit, used to acquire an external data wrapper plug-in corresponding to the first database; the external data wrapper plug-in is used to access the second database in the first database;
[0222] The first startup unit is used to start the data acquisition process in the first database, so that the data acquisition process uses the external data wrapper plug-in, logs in to the second database through the login information, and obtains at least one target object from the target storage location in the second database.
[0223] In an optional embodiment, the first starting unit is further used to start the data acquisition process in the first database, so that the data acquisition process sends the object acquisition request to the second database based on the external data wrapper plug-in; the object acquisition request carries the target login information and the target storage location; and the data acquisition process receives at least one target object sent by the second database;
[0224] Among them, at least one of the target objects is obtained from the target storage location when the second database responds to the object acquisition request and determines that the identity information of the target account satisfies a preset condition.
[0225] In an optional embodiment, when the target external table is a table connected by multiple tables, the query module and the start module include:
[0226] a target external table query unit, configured to query the target external table to determine a current object in the second database corresponding to the target external table based on preset mapping information;
[0227] A second starting unit, configured to start the data acquisition process in the first database, so that the data acquisition process acquires the current object from the second database;
[0228] a repeating unit, configured to repeat the querying of the target external table, to enable the data acquisition process to acquire the current object from the second database, until the number of repetitions matches the size of the target external table;
[0229] A determination unit is used to determine the current object obtained after the repetitive process is completed as at least one of the target objects.
[0230] In an optional embodiment, the device further comprises:
[0231] An external service determination module, used to determine an external service; the external service stores a storage location of an object in the second database and login information of an account in the second database; the storage location includes the target storage location, and the login information includes the target login information;
[0232] an external table generation module, configured to generate, in the first database, an external table corresponding to at least one object in the second database based on the external service; and to establish the preset mapping information representing the mapping relationship between the external table and the object in the second database;
[0233] Accordingly, the response module includes:
[0234] The external table acquisition unit is used to acquire the target external table corresponding to the object identification information from the external table in the first database.
[0235] In an optional embodiment, the device further comprises:
[0236] A calling module, used to call a preset encryption plug-in corresponding to the first database;
[0237] An encryption module, used to encrypt the login information based on the preset encryption plug-in to obtain an encrypted login information result;
[0238] A storage module, used for storing the login information encryption result in an encryption information storage table;
[0239] The login information encryption result stored in the encryption information storage table is used to obtain at least one of the target objects from the second database.
[0240] In an optional embodiment, the login location acquisition module includes:
[0241] The encryption result obtaining unit is used to obtain the login information encryption result from the encryption information storage table.
[0242] The decryption unit is used to call the preset encryption plug-in to decrypt the login information to obtain the decrypted target login information.
[0243] In an optional embodiment, the acquisition and sending module includes:
[0244] Function acquisition module, used to obtain preset hook functions;
[0245] an acquisition conversion module, configured to acquire at least one of the target objects from the shared memory through the preset hook function, and convert the format of the at least one of the target objects into a data format corresponding to the first database;
[0246] The object sending module is used to send the target object after format conversion to the target account.
[0247] It should be noted that the device embodiment provided in the embodiments of the present application and the above-mentioned method embodiment are based on the same inventive concept.
[0248] An embodiment of the present application also provides an electronic device for object acquisition, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement an object acquisition method provided in any of the above embodiments.
[0249] An embodiment of the present application also provides a computer-readable storage medium, which can be set in a terminal to store at least one instruction or at least one program for implementing a method in a method embodiment, and the at least one instruction or at least one program is loaded and executed by a processor to implement the object acquisition method provided in the above method embodiment.
[0250] Optionally, in the embodiment of this specification, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the above storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.
[0251] The memory of the embodiment of this specification can be used to store software programs and modules, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, applications required for functions, etc.; the data storage area may store data created according to the use of the device, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.
[0252] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the object acquisition method provided in the above method embodiment.
[0253] The object acquisition method embodiment provided in the embodiment of the present application can be executed in a terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 7 FIG. 1 is a hardware structure diagram of a server provided according to an exemplary embodiment. Figure 7As shown, the server 600 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 610 (the central processing unit 610 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 630 for storing data, and one or more storage media 620 (such as one or more mass storage devices) for storing application programs 623 or data 622. Among them, the memory 630 and the storage medium 620 can be short-term storage or permanent storage. The program stored in the storage medium 620 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processing unit 610 can be configured to communicate with the storage medium 620 and execute a series of instruction operations in the storage medium 620 on the server 600. The server 600 may also include one or more power supplies 660, one or more wired or wireless network interfaces 650, one or more input and output interfaces 640, and / or one or more operating systems 621, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0254] The input / output interface 640 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the server 600. In one example, the input / output interface 640 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 640 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0255] It can be understood by those skilled in the art that Figure 7 The structure shown is for illustration only and does not limit the structure of the above electronic device. Figure 7 More or fewer components as shown, or with Figure 7 Different configurations are shown.
[0256] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of this specification are described. Other embodiments are within the scope of the attached claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0257] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and server embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0258] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0259] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A method for acquiring an object, characterized in that: The method comprises: In response to an object acquisition request triggered by a target account, acquiring a target external table corresponding to the object acquisition request from a first database; Querying the target external table to determine at least one target object in the second database corresponding to the target external table based on preset mapping information; the preset mapping information represents a mapping relationship between the external table and the object in the second database; Initiate a data acquisition process in the first database, so that the data acquisition process acquires at least one target object from the second database; and enable the data acquisition process to store at least one target object in a shared memory in the first database; At least one of the target objects is obtained from the shared memory, and at least one of the target objects is sent to the target account.
2. The object acquisition method according to claim 1, characterized in that: Before starting the data acquisition process in the first database, the method further includes: Acquire a target external service corresponding to the target external table; the target external service stores at least one target storage location of the target object in the second database and target login information of the target account in the second database; Obtaining the target login information and the target storage location from the target external service; The target login information is used to log into the second database, and the target storage location is used to obtain at least one target object.
3. The object acquisition method according to claim 2, characterized in that: The starting of the data acquisition process in the first database so that the data acquisition process acquires at least one target object from the second database includes: Acquire an external data wrapper plug-in corresponding to the first database; the external data wrapper plug-in is used to access the second database in the first database; The data acquisition process is started in the first database, so that the data acquisition process uses the external data wrapper plug-in, logs into the second database through the login information, and acquires at least one target object from the target storage location in the second database.
4. The object acquisition method according to claim 3, characterized in that: The starting the data acquisition process in the first database so that the data acquisition process uses the external data wrapper plug-in to log in to the second database through the login information and acquire at least one target object from the storage location in the second database includes: Initiate the data acquisition process in the first database, so that the data acquisition process sends the object acquisition request to the second database based on the external data wrapper plug-in; the object acquisition request carries the target login information and the target storage location; and the data acquisition process receives at least one target object sent by the second database; Among them, at least one of the target objects is obtained from the target storage location when the second database responds to the object acquisition request and determines that the identity information of the target account satisfies a preset condition.
5. The object acquisition method according to any one of claims 1 to 4, characterized in that: In the case where the target external table is a table connected by multiple tables, querying the target external table to determine at least one target object corresponding to the target external table in the second database based on preset mapping information; Starting a data acquisition process in the first database so that the data acquisition process acquires at least one target object from the second database includes: Querying the target external table to determine a current object in the second database corresponding to the target external table based on preset mapping information; Starting the data acquisition process in the first database so that the data acquisition process acquires the current object from the second database; Repeating the querying of the target external table, to enable the data acquisition process to acquire the current object from the second database, until the number of repetitions matches the size of the target external table; The current object obtained after the repetitive process is completed is determined as at least one of the target objects.
6. The object acquisition method according to claim 2, characterized in that: Before responding to the object acquisition request triggered by the target account, the method further includes: Determine an external service; the external service stores a storage location of an object in the second database and login information of an account in the second database; the storage location includes the target storage location, and the login information includes the target login information; Based on the external service, generating an external table corresponding to at least one object in the second database in the first database; and establishing the preset mapping information representing the mapping relationship between the external table and the object in the second database; The object acquisition request carries object identification information, and acquiring a target external table corresponding to the object acquisition request from the first database includes: The target external table corresponding to the object identification information is obtained from the external table in the first database.
7. The object acquisition method according to claim 6, characterized in that: After determining the external service, the method further includes: Calling a preset encryption plug-in corresponding to the first database; Encrypting the login information based on the preset encryption plug-in to obtain a login information encryption result; Storing the login information encryption result in an encryption information storage table; The login information encryption result stored in the encryption information storage table is used to obtain at least one of the target objects from the second database.
8. The object acquisition method according to claim 7, characterized in that: The acquiring the target login information from the target external service includes: Obtain the login information encryption result from the encryption information storage table; The preset encryption plug-in is called to decrypt the login information to obtain the decrypted target login information.
9. The object acquisition method according to any one of claims 1 to 4, characterized in that: The acquiring at least one target object from the shared memory and sending at least one target object to the target account includes: Get the preset hook function; Acquire at least one of the target objects from the shared memory through the preset hook function, and convert the format of at least one of the target objects into a data format corresponding to the first database; Send the target object after format conversion to the target account.
10. An object acquisition device, characterized in that: The device comprises A response module, configured to respond to an object acquisition request triggered by a target account and acquire a target external table corresponding to the object acquisition request from the first database; A query module, configured to query the target external table to determine at least one target object in the second database corresponding to the target external table based on preset mapping information; The preset mapping information represents a mapping relationship between an external table and an object in the second database; a start module, configured to start a data acquisition process in the first database, so that the data acquisition process acquires at least one target object from the second database; and to enable the data acquisition process to store at least one target object in a shared memory in the first database; The acquisition and sending module is used to acquire at least one of the target objects from the shared memory and send at least one of the target objects to the target account.
11. An electronic device for object acquisition, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded by the processor and executes the object acquisition method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the object acquisition method according to any one of claims 1 to 9.
Citation Information
Patent Citations
Method of accessing shared memory and apparatus thereof
CN103607428A
Storage data access method and device, and storage medium
CN108846129A
Method and system for quickly exporting batch data from database based on multi-concurrency technology
CN110362617A
Database operation method and fully encrypted database
CN114722115A
Apparatus and method for transparent access of foreign databases in a heterogeneous database system
US6226649B1
