Access method, device, equipment, system and medium of distributed data node

Abstract

This application discloses a method, apparatus, device, system, and medium for accessing distributed data nodes, belonging to the field of data processing. The method includes: determining the target data node accessed by the database operation statement based on a database operation statement transmitted by a client through an established front-end connection; obtaining an idle back-end connection from a preset connection pool as the target connection, the connection pool including idle back-end connections, the target connection undergoing check and synchronization processing to ensure normal use; sending the database operation statement to the target data node using the target connection; and, if the database operation statement has not initiated a database transaction or has already closed a database transaction, adding the target connection as an idle back-end connection to the connection pool. Embodiments of this application can reduce network resource waste.

Description

Technical Field

[0001] This application belongs to the field of data processing, and in particular relates to a method, apparatus, device, system and medium for accessing distributed data nodes. Background Technology

[0002] A distributed database architecture consists of a proxy layer and data nodes. The proxy layer establishes a front-end connection with the client, receives database operation statements from the front-end connection, establishes a back-end connection with the data nodes, routes the database operation statements to the corresponding data nodes, receives the responses from the data nodes, processes the responses, and sends them back to the client. The front-end connection between the proxy layer and the client and the back-end connection between the proxy layer and the data nodes are bound together; that is, for a front-end connection, the proxy layer will establish one or more back-end connections and maintain them bound to that front-end connection until the front-end connection is closed.

[0003] However, since the front-end connection and the back-end connection remain bound throughout the entire lifecycle of the front-end connection between the proxy layer and the client, the binding will not be released even if the front-end connection and the back-end connection are idle. Therefore, a large number of back-end connections need to be established, resulting in a waste of network resources. Summary of the Invention

[0004] This application provides a method, apparatus, device, system, and medium for accessing distributed data nodes, which can reduce network resource waste.

[0005] In a first aspect, embodiments of this application provide a method for accessing distributed data nodes, comprising: determining the target data node accessed by the database operation statement based on the database operation statement transmitted by the client through an established front-end connection; obtaining an idle back-end connection from a preset connection pool as the target connection, the connection pool including idle back-end connections, the target connection undergoing a check and synchronization process to ensure normal use of the target connection; sending the database operation statement to the target data node using the target connection; and, if the database operation statement has not started a database transaction or has closed a database transaction, placing the target connection as an idle back-end connection into the connection pool.

[0006] Secondly, embodiments of this application provide a distributed data node access device, comprising: a receiving module, configured to determine the target data node accessed by the database operation statement based on the database operation statement transmitted by the client through an established front-end connection; a connection processing module, configured to obtain an idle backend connection from a preset connection pool as the target connection, the connection pool including idle backend connections, the target connection undergoing inspection and synchronization processing to ensure normal use of the target connection; a sending module, configured to send the database operation statement to the target data node using the target connection; the connection processing module is further configured to place the target connection as an idle backend connection into the connection pool when the database operation statement has not started a database transaction or has closed a database transaction.

[0007] Thirdly, embodiments of this application provide a proxy device, including: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the distributed data node access method of the first aspect.

[0008] Fourthly, embodiments of this application provide a distributed data node access system, including: a client; a proxy device (as described in the third aspect), the proxy device being communicatively connected to the client; and a data node, being communicatively connected to the proxy device.

[0009] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the distributed data node access method of the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the distributed data node access method of the first aspect.

[0011] This application provides a method, apparatus, device, system, and medium for accessing distributed data nodes. Upon receiving a database operation statement transmitted by a user through a front-end connection, the target data node for the database operation can be parsed. An idle backend connection is obtained from a preset connection pool including idle backend connections as the target connection for sending the database operation statement to the target data node, and the database operation statement is sent through this target connection. If the database operation statement has not initiated a database transaction or has already closed a database transaction, the target connection is restored to an idle state and placed in the connection pool as an idle backend connection, allowing idle backend connections in the connection pool to be reused by different clients and different front-end connections. By reusing idle backend connections in the connection pool, the binding between front-end and back-end connections can be decoupled, reducing the coupling between them, fully utilizing idle backend connections, reducing network resource waste, and improving network resource utilization efficiency. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram illustrating an example of an application architecture for accessing distributed data nodes provided in an embodiment of this application;

[0014] Figure 2 A flowchart illustrating a method for accessing distributed data nodes according to an embodiment of this application;

[0015] Figure 3 A flowchart illustrating a method for accessing distributed data nodes according to another embodiment of this application;

[0016] Figure 4 A schematic diagram of an example architecture of a two-level connection pool provided in an embodiment of this application;

[0017] Figure 5 A schematic diagram illustrating an example of a first linked list provided in an embodiment of this application;

[0018] Figure 6 A schematic diagram illustrating an example of a second linked list provided in an embodiment of this application;

[0019] Figure 7 A flowchart illustrating an example of a client accessing a data node through a proxy layer, as provided in embodiments of this application;

[0020] Figure 8 A schematic diagram of the structure of a distributed data node access device provided in an embodiment of this application;

[0021] Figure 9 This is a schematic diagram of the structure of an agent device provided in an embodiment of this application. Detailed Implementation

[0022] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples. It should be noted that the acquisition, storage, use, and processing of information and data in the embodiments of this application are all authorized by users or relevant organizations and comply with the relevant provisions of national laws and regulations.

[0023] A distributed database architecture consists of a proxy layer and data nodes. The proxy layer establishes a front-end connection with the client, receives database operation statements from the front-end connection, establishes a back-end connection with the data nodes, routes the database operation statements to the corresponding data nodes, receives responses from the data nodes, processes the responses, and sends them back to the client. The front-end connection between the proxy layer and the client, and the back-end connection between the proxy layer and the data nodes, are bound together. That is, for a front-end connection, the proxy layer establishes one or more back-end connections and maintains this binding until the front-end connection is closed. However, because the front-end and back-end connections remain bound throughout the entire lifecycle of the front-end connection between the proxy layer and the client—even when the front-end and back-end connections are idle—the binding is not released. This necessitates establishing a large number of back-end connections, resulting in wasted network resources. In scenarios where there are performance requirements for distributed data nodes and a limit on the number of back-end connections, the number of front-end connections may also be limited, leading to lower efficiency in accessing distributed data nodes.

[0024] This application provides a method, apparatus, device, system, medium, and program product for accessing distributed data nodes. It establishes a connection pool including idle backend connections, which are held in the connection pool but not bound. When a database operation statement is received from a client via a frontend connection, an idle backend connection can be obtained from the connection pool and used to send the database operation statement to the data node for database operation. If the database operation statement has not started a database transaction or has already closed a database transaction, the backend connection that sent the database operation statement can be put back into the connection pool for use in subsequent database operation statement transmissions, thus reusing the idle backend connections in the connection pool. By reusing idle backend connections in the connection pool, the binding between frontend and backend connections can be decoupled, reducing the coupling between them, fully utilizing idle backend connections, reducing network resource waste, and improving network resource utilization efficiency. Furthermore, since idle backend connections are fully utilized, a limited number of backend connections can support more frontend connections, thereby reducing the limitation on the number of frontend connections, improving the maximum performance of the distributed data node cluster, and increasing the efficiency of accessing distributed data nodes.

[0025] To facilitate understanding, a brief introduction will be given here of the application architecture of the distributed data node access method, device, equipment, system, medium and program product provided in this application. Figure 1 A schematic diagram illustrating an example of an application architecture for accessing distributed data nodes provided in an embodiment of this application, as shown below. Figure 1 As shown, client 11 can establish a front-end connection with proxy layer 12. Client 11 can establish a front-end connection with the proxy layer through, but is not limited to, Java Database Connectivity (JDBC) and Common Application Programming Interface (CAPI).

[0026] The number of proxy layers can be one or more. Figure 1 The diagram shows three proxy layers 12. Each proxy layer 12 can be configured with a front-end connection module 121, which can be used to establish a front-end connection with the client 11. Proxy layers 12 can also be configured with a connection pool 14, which includes idle back-end connections. Proxy layers 12 can utilize these idle back-end connections to transmit database operation statements when needed. It should be noted that when an idle back-end connection is used by the proxy layer to transmit database operation statements, it is an occupied back-end connection and is not in an idle state until the database transaction is closed, at which point it becomes an idle back-end connection again.

[0027] Data nodes can be in a distributed data node architecture, where each data node can be viewed as a database on its own, or as a component of a database. Figure 1 Five data nodes 15 are shown. The proxy layer 12 can obtain an idle backend connection from the connection pool 14 and use the idle backend connection to transmit the database operation statements received from the client 11 to the corresponding data node 15. In this embodiment, the data node 15 can be a multi-replica node structure. For example, the data node 15 may include a primary data node, a backup data node, and a slave data node to improve data security and reliability.

[0028] The following describes the access methods, devices, equipment, systems, media, and program products for distributed data nodes provided in this application.

[0029] The first aspect of this application provides a method for accessing distributed data nodes, which can be applied to a proxy layer. That is, the method for accessing distributed data nodes can be executed by the proxy layer, which can be specifically implemented as a distributed data node access device, a proxy device, etc., and is not limited thereto. Figure 2 A flowchart of a distributed data node access method provided in an embodiment of this application is shown below. Figure 2 As shown, the method for accessing the distributed data node may include steps S201 to S204.

[0030] In step S201, the target data node accessed by the database operation statement is determined based on the database operation statement transmitted by the client through the established front-end connection.

[0031] After the proxy layer establishes a front-end connection with the client, the client can send database operation statements to the proxy layer through the front-end connection. Database operation statements are used to perform data operations on data nodes. In some examples, database operation statements may include Structured Query Language (SQL), which may include, but is not limited to, query statements such as select statements, commit statements such as commit statements, insert statements such as insert statements, and update statements such as update statements.

[0032] The proxy layer can parse database operation statements to obtain the database sharding results, which indicate the data nodes accessed by the database operation statements. The target data nodes include the data nodes accessed by the database operation statements.

[0033] In step S202, an idle backend connection is obtained from a preset connection pool as the target connection.

[0034] The connection pool includes idle backend connections. More specifically, the connection pool only includes idle backend connections; that is, any backend connection in the connection pool is an idle backend connection, and all backend connections obtainable from the connection pool are idle backend connections. Idle backend connections include backend connections in an idle state, and these backend connections are connections between the proxy layer and the data nodes. Idle backend connections do not execute the transmission of client database operation statements and are in a waiting state, ready to be requested and used at any time. Idle backend connections in the connection pool can be reused. For example, at time t1, backend connection a1 is obtained from the connection pool; this backend connection a1 is an idle backend connection, and it is used to transmit database operation statement A1. At time t2, the database transaction corresponding to database operation statement A1 has been closed, and backend connection a1 is again in an idle state and returned to the connection pool. At time t3, the idle backend connection a1 can be obtained again from the connection pool, and it is used to transmit database operation statement A2. Idle backend connections in the connection pool can be reused by multiple database operation statements from multiple clients.

[0035] When a target connection is needed, the connection pool can pop an idle backend connection as the target connection. The target connection includes the backend connection used to send database operation statements. To ensure the target connection can reliably transmit database operation data, it undergoes a check and synchronization process to guarantee its normal use. This check and synchronization process includes a liveness check and a synchronization process. The liveness check verifies the liveness of the target connection; an idle backend connection serving as the target connection is considered live. If the cluster topology of the target connection changes, it is considered inactive and needs to be replaced. The synchronization process synchronizes the attributes of the frontend connection and the idle backend connection serving as the target connection. These attributes may include, but are not limited to, database setting variables. If the liveness check determines that the selected backend connection is live and its attributes are synchronized with the frontend connection, then the selected backend connection can be used as the target connection.

[0036] In some examples, the liveness check process may include: checking the cluster topology of the target connection; if the cluster topology of the target connection changes, releasing the target connection, re-establishing the backend connection with the target data node, and identifying the re-established backend connection as the new target connection.

[0037] Changes in the cluster topology of the target connection may include a replica switch of the target data node associated with the target connection and / or a disconnection of the target connection. The data nodes may have a multi-replica structure, such as primary, standby, and secondary data nodes. For various reasons, the data node providing data may switch between these primary, standby, and secondary data nodes. For example, if the backend connection to the target connection was initially associated with the primary data node of the target data node, but subsequently switched from a primary to a secondary data node (i.e., the data service is currently provided by the secondary data node), it can be determined that the cluster topology of the target connection has changed, and the target connection is unusable. This unused target connection should be released, and a new backend connection to the target data node should be established. This newly established backend connection should then be used as the target connection for sending database operation statements. For example, if a communication problem causes the target data node associated with the target connection to temporarily go offline, resulting in a broken target connection, and the target data node subsequently comes back online but the target connection does not reconnect to that data node, it can be determined that the topology of the target connection has changed, and the target connection is unusable. This useless target connection should be released, and a new backend connection to the target data node should be established. This newly established backend connection should be used as the target connection for sending database operation statements. Liveness checks can be performed by pinging the target data node or sending query statements (select statements) to the target data node.

[0038] In some examples, the liveness check process may include: periodically checking the cluster topology of idle backend connections in the connection pool; if the cluster topology of an idle backend connection changes, releasing the idle backend connection. For details regarding changes in the cluster topology of idle backend connections, please refer to the relevant explanation of changes in the cluster topology of the target connection in the examples above, which will not be repeated here. By periodically checking the cluster topology of idle backend connections in the connection pool in advance, idle backend connections with changed cluster topologies can be released in a timely manner, ensuring that all idle backend connections in the connection pool are usable. This avoids situations where an idle backend connection is discovered to be unusable only after it has been popped from the connection pool due to a changed cluster topology, further shortening the time required to access data nodes and improving the efficiency of data node access.

[0039] The liveness detection process may also include other processing methods, and all methods capable of detecting the liveness of backend connections are within the protection scope of the embodiments of this application.

[0040] In some examples, synchronization may include, but is not limited to: setting database configuration variables for the target connection to be consistent with those for the frontend connection. To ensure database operation statements are transmitted to the target data node via both the frontend and backend connections, the database environments of the frontend and backend connections must be consistent. This database environment can be adjusted using database configuration variables (i.e., database set variables). By setting the database configuration variables for the target connection to be consistent with those for the frontend connection, the database environments of the frontend and target connections can be kept aligned, preventing errors.

[0041] In step S203, database operation statements are sent to the target data node using the target connection.

[0042] The proxy layer uses the target connection to send database operation statements to the target data node. These statements perform corresponding data operations on the target data node. The target data node then transmits its response to the database operation statements to the proxy layer via the target connection. The proxy layer then transmits the response to the client via the front-end connection. In some examples, the proxy layer can filter, merge, and package the response information before sending it back to the client via the front-end connection.

[0043] In step S204, if the database operation statement has not started a database transaction or has closed a database transaction, the target connection is placed into the connection pool as an idle backend connection.

[0044] A database transaction is a sequence of database operations that access and potentially manipulate data. A database transaction consists of the database operations performed between the start and end of the transaction. For some simple database operations, it's possible to choose not to initiate a database transaction; in this case, the database operation statements can be considered as not initiating a database transaction. A database operation statement is the statement corresponding to the last database operation in the database operation sequence. After this statement is executed, the database operation sequence is complete, and the database transaction ends. In this case, the database operation statement can be considered as closing the database transaction. When a database operation statement neither initiates nor closes a database transaction, the target connection does not need to remain bound to the frontend connection and can be unbound, i.e., the target connection is restored to an idle state. This restored idle target connection is placed as an idle backend connection in the connection pool for reuse in subsequent processes, achieving the reuse of idle backend connections in the connection pool.

[0045] If the database operation statement does not close the database transaction, the target connection is kept bound to the frontend connection and is not put into the connection pool.

[0046] In this embodiment, upon receiving a database operation statement transmitted by a user through a frontend connection, the target data node for the database operation can be parsed. An idle backend connection is obtained from a pre-defined connection pool containing idle backend connections and used as the target connection to send the database operation statement to the target data node. The database operation statement is then sent through this target connection. If the database operation statement has not initiated a database transaction or has already closed a database transaction, the target connection is restored to an idle state and placed in the connection pool as an idle backend connection. This allows idle backend connections in the connection pool to be reused by different clients and different frontend connections. By reusing idle backend connections in the connection pool, the binding between frontend and backend connections can be decoupled, reducing their coupling, fully utilizing idle backend connections, reducing network resource waste, and improving network resource utilization efficiency. Furthermore, since idle backend connections are fully utilized, a limited number of backend connections can support more frontend connections, thereby reducing the limitation on the number of frontend connections, improving the maximum performance of the distributed data node cluster, and increasing the efficiency of accessing distributed data nodes.

[0047] In some embodiments, accessing distributed data nodes requires threads, which can form processes. Correspondingly, the connection pool can include thread pools and process pools. A thread pool includes idle backend connections; further, any backend connection in a thread pool is an idle backend connection. Each thread pool corresponds to one thread, and the idle backend connections in a thread pool are associated with the thread corresponding to that thread pool. Idle backend connections in a thread pool can only be invoked by the thread corresponding to that thread pool. A process pool also includes idle backend connections; further, any backend connection in a process pool is an idle backend connection. Idle backend connections in a process pool can be invoked by any thread in the process corresponding to that process pool. Idle backend connections can be provided through a multi-level connection pooling approach using thread pools and process pools. Figure 3 A flowchart illustrating a method for accessing distributed data nodes according to another embodiment of this application. Figure 3 and Figure 2 The difference is that, Figure 2 Step S202 can be further refined as follows: Figure 3 Steps S2021 to S2023 in the process.

[0048] In step S2021, it is determined whether there is an idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement.

[0049] The thread pool corresponding to a given thread can be determined based on the thread used by the database operation statement. Any backend connection in the thread pool is an idle backend connection; if the thread pool has backend connections, it means it has idle backend connections; if the thread pool does not have backend connections, it means it does not have idle backend connections.

[0050] In step S2022, if there is an idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement, an idle backend connection is obtained from the thread pool and determined as the target connection.

[0051] If the thread pool corresponding to the thread has an idle backend connection, an idle backend connection can be directly popped from the thread pool as the target connection to transmit database operation statements. It should be noted that the popped idle backend connection needs to undergo a check and synchronization process. After confirming that it is active and its attributes are synchronized, the idle backend connection is used as the target connection.

[0052] Since thread pools correspond to specific threads, idle backend connections in a thread pool are only available to the thread corresponding to that thread pool. Therefore, thread pools do not require a locking mechanism. Frontend connections occupying a thread will preferentially access the thread pool corresponding to that thread through a proxy layer, without requiring a lock waiting step. That is, the lock waiting step can be skipped from the thread pool to obtain an idle backend connection as the target connection. Obtaining idle backend connections from the thread pool is faster, further improving the efficiency of accessing data nodes.

[0053] In step S2023, if there is no idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement, an idle backend connection is obtained from the process pool corresponding to the process to which the thread occupied by the database operation statement belongs and determined as the target connection.

[0054] If the thread pool corresponding to a thread does not have an idle backend connection, an idle backend connection is popped from the process pool of the process to which the thread belongs as the target connection. It should be noted that the popped idle backend connection needs to undergo a check and synchronization process to ensure that it is live and its attributes are synchronized before it is used as the target connection.

[0055] All threads within a process can obtain idle backend connections from the process pool. The process pool can be considered a global connection pool, requiring a locking mechanism. Obtaining an idle backend connection from the process pool necessitates a lock-waiting step before acquiring the connection.

[0056] In some examples, a switching timeout can be preset. This timeout, represented as `idle_exchange_timeout`, includes a threshold of the idle time required for an idle backend connection to be transferred from the thread pool to the process pool. The specific timeout can be set based on the scenario, requirements, and experience. If the idle time of an idle backend connection in the thread pool exceeds the preset switching timeout, the connection is moved to the process pool. The idle time of an idle backend connection is the duration it remains idle. If the idle time of an idle backend connection in the thread pool exceeds the switching timeout, it indicates that the threads in that thread pool have relatively low workload (i.e., low activity level). Idle backend connections in this thread pool that exceed the switching timeout can be demoted and moved to the process pool. This allows other, more active threads to obtain and use idle backend connections from the process pool when their own thread pools lack sufficient available connections, thus achieving dynamic allocation of backend connection resources. The switching timeout allows for the scaling down of the thread pool under low workload conditions. It should be noted that after the target connection transmits the database operation statement and its corresponding response, if the database operation statement has not started a database transaction or has already closed a database transaction, the target connection is preferentially placed into the thread pool corresponding to the thread used by the database operation statement. The thread pool can be expanded by placing the target connection, after the session is completed and the response to the corresponding data operation of the database operation statement is received, into the thread pool.

[0057] In some examples, a release timeout can be preset. This timeout, represented as `idle_free_timeout`, includes a threshold of the idle time required for an idle backend connection to be released. The specific timeout can be set based on the scenario, requirements, and experience. If the idle time of an idle backend connection in the process pool exceeds the preset release timeout, the idle backend connection is released. The release timeout can be longer than the switching timeout. If the idle time of an idle backend connection in the process pool exceeds the release timeout, it indicates that the process's workload is low, meaning the idle backend connection is likely to remain unused for a long time. Therefore, idle backend connections in the process pool whose idle time exceeds the release timeout can be released, achieving dynamic distribution of backend connection resources and further saving network resources. The release timeout allows for scaling down the process pool under low workload conditions. Moving idle backend connections from the thread pool to the process pool allows for scaling up the process pool.

[0058] For example, Figure 4 This is a schematic diagram of an example architecture of a two-level connection pool provided in an embodiment of this application. Figure 4The diagram illustrates thread pools 31 and 32, and process pool 33. Backend connections 1 through 10 are idle backend connections. After backend connection 1 is used, it becomes an idle backend connection again and enters thread pool 31. Backend connection 4's idle time exceeds the switching timeout period, so it is moved from thread pool 31 to process pool 33 and locked. Backend connection 7 in process pool 33's idle time exceeds the release timeout period, so it is released and locked. Backend connection 6 in process pool 33 can be moved from process pool 33 to thread pool 32, and locked. Backend connection 10 in thread pool 32 is popped to be used as a target connection for transmitting database operation statements.

[0059] In some embodiments, if there are no idle backend connections in the connection pool, a new backend connection is created and designated as the target connection. The absence of idle backend connections in the connection pool indicates the lack of reusable idle backend connections. To respond promptly to client requests, a new backend connection is created as the target connection to send database operation statements. If the connection pool includes a thread pool and a process pool, and the thread pool does not have an idle backend connection, the process pool is searched; if the process pool also does not have an idle backend connection, a new backend connection is created and designated as the target connection.

[0060] In some embodiments, a performance requirement range for the connection pool can be set. This range may include the range of idle backend connections in the connection pool that enables the distributed data node access system to operate at high performance, or the capacity range of idle backend connections in the connection pool. If the number of idle backend connections in the connection pool is less than the lower limit of the performance requirement range, a new backend connection is created as the target connection. This target connection is placed in the connection pool after use, provided that no database transaction has been initiated or has been closed, thus ensuring that the number of idle backend connections in the connection pool quickly reaches the performance requirement range. If the number of idle backend connections in the connection pool is greater than the upper limit of the performance requirement range, and there are no idle backend connections in the connection pool, the process waits until an idle backend connection becomes available, at which point that idle backend connection is identified as the target connection. If the number of idle backend connections in the connection pool exceeds the upper limit of the performance requirement range, creating a new backend connection would negatively impact system performance. Therefore, database operation statements can be queued, waiting for an idle backend connection to appear in the connection pool again, thereby controlling the number of backend connections connecting to data nodes in the connection pool to remain within the performance requirement range and ensuring system performance.

[0061] In some embodiments, if a database transaction has been initiated by a database operation statement, the data nodes of the backend connections bound to the frontend connection corresponding to the database operation statement can be obtained; if the data nodes of the backend connections bound to the frontend connection corresponding to the database operation statement include the target data node, the bound backend connection is determined as the target connection; if the data nodes of the backend connections bound to the frontend connection corresponding to the database operation statement do not include the target data node, an idle backend connection is obtained from the connection pool as the target connection; if there is no idle backend connection in the connection pool, a new backend connection is created as the target connection.

[0062] If a database operation statement has started a database transaction, then the front-end connection of the database operation statement is already bound to a back-end connection. You can first check whether the data node of the back-end connection bound to the front-end connection has covered the target data node of this database operation statement. If it has covered, you can directly use the already bound back-end connection to send to the target data node; if it has not covered, you need to use an unbound idle back-end connection as the target connection to send the database operation statement to the target data node.

[0063] In some embodiments, a first linked list may be set, which may be implemented as a priority list. The first linked list includes at least one level of linked lists, each level having a key and at least one value. The key includes a capability identifier, and the value includes a connection identifier. The capability identifier can characterize the permissions and capabilities of the front-end connection. The connection identifier can indicate an idle back-end connection; for example, the connection identifier may include, but is not limited to, the number of the idle back-end connection, a pointer to the idle back-end connection, etc. The capability identifier of the idle back-end connection indicated by the connection identifier in a level of linked list is the same as the capability identifier indicated by the key in the level of linked list. The connection identifiers in each level of linked list are arranged in the order in which the idle back-end connections indicated by the connection identifiers entered the connection pool. The above step S202 can be further refined as follows: obtain the first capability identifier of the front-end connection; search for a level of linked list in the preset first linked list whose key matches the first capability identifier, and determine the idle back-end connection indicated by the first connection identifier in the level of linked list whose key matches the first capability identifier as the target connection. The first capability identifier is the capability identifier of the front-end connection. By using the first linked list, we can find the next linked list in the first linked list that has the same power identifier as the front-end connection, so that the power identifier of the target connection used is consistent with the power identifier of the front-end connection, ensuring that database operation statements and response information can be transmitted normally.

[0064] For example, Figure 5 A schematic diagram illustrating an example of a first linked list provided in an embodiment of this application, as shown below. Figure 5As shown, the first linked list consists of n levels of linked lists, where n is a positive integer. The key of the first level linked list is the power identifier b1, and the values ​​of the first level linked list are connection identifiers b11, b12, b13, ..., b1m. The key of the second level linked list is the power identifier b2, and the values ​​of the second level linked list are connection identifiers b21, b22, b23, ..., b2m. The key of the third level linked list is the power identifier b3, and the values ​​of the third level linked list are connection identifiers b31, b32, b33, ..., b3m. The key of the nth level linked list is the power identifier bn, and the values ​​of the nth level linked list are connection identifiers bn1, bn2, bn3, ..., bnm. The connection identifiers in the first level linked list are arranged from earliest to latest according to the time when the idle backend connection entered the connection pool. The order of the connection identifiers in the other levels of linked lists is also the same, and will not be repeated here. If the capability identifier of the front-end connection of this database operation statement is capability identifier b2, then search the second-level linked list from the first linked list, and pop the first connection identifier b21 in the second-level linked list, which is the connection identifier of the idle back-end connection that entered the connection pool earliest in the second-level linked list. The idle back-end connection indicated by connection identifier b21 is determined as the target connection and the database operation statement is transmitted to the target data node.

[0065] In some embodiments, a second linked list may be provided, which can be implemented as a total list. The second linked list includes connection identifiers of idle backend connections in the connection pool; that is, the second linked list includes connection identifiers of all idle backend connections in the connection pool. The connection identifiers in the second linked list are arranged in the order in which the idle backend connections indicated by the connection identifiers entered the connection pool. For example... Figure 6 This is a schematic diagram illustrating an example of a second linked list provided in an embodiment of this application. Figure 6 The second linked list shown includes Figure 5 All connection identifiers in the first linked list are shown, and the connection identifiers in the second linked list are arranged in the order in which the idle backend connection identifier indicated by the connection identifier enters the connection pool. For example, the idle backend connection indicated by connection identifier b21 enters the connection pool after the idle backend connection indicated by connection identifier b11.

[0066] Correspondingly, if the first linked list does not have a layer of linked lists with keys matching the first capability identifier, the idle backend connection indicated by the first connection identifier in the preset second linked list can be obtained as a candidate connection. If the candidate connection passes the outer logic applicability verification, it is determined as the target connection. If the first linked list does not have a layer of linked lists with keys matching the first capability identifier, the idle backend connection indicated by the first connection identifier in the second linked list is popped from the connection pool as a candidate connection. The outer logic can then determine whether the capability represented by the capability identifier of the candidate connection is applicable to the capability represented by the first capability identifier. If it is applicable, the candidate connection passes the outer logic applicability verification and is determined as the target connection. If the candidate connection fails the outer logic applicability verification, the frontend connection needs to be re-established. By using the second linked list, the success rate of distributed data node access can be improved.

[0067] To facilitate understanding, the following explanation uses SQL statements as an example to illustrate the overall process of a client accessing a data node through a proxy layer. Figure 7 The flowchart illustrates an example of a client accessing a data node through a proxy layer, as provided in this application embodiment. The process of the client accessing the data node through the proxy layer includes steps c1 to c10.

[0068] In step c1, a front-end connection is established. The front-end connection is the connection between the client and the proxy layer.

[0069] In step c2, the SQL statement is parsed to obtain the database sharding result. The database sharding result represents the target data node. If the SQL statement has not started a database transaction, proceed to step c3; if the SQL statement has started a database transaction, proceed to step c10.

[0070] In step c3, determine whether the thread pool has an idle backend connection. If an idle backend connection is available, proceed to step c4; otherwise, proceed to step c8.

[0071] In step c4, an idle backend connection is popped up and checked for synchronization to obtain the target connection.

[0072] In step c5, an SQL statement is sent and executed using the target connection. If the database transaction is not started or has been closed, proceed to step c6; if the database transaction is started, proceed to step c7.

[0073] In step c6, the target connection is returned to the thread pool.

[0074] In step c7, the binding between the target connection and the front-end connection is maintained.

[0075] In step c8, determine whether the process pool has an idle backend connection. If an idle backend connection is available, proceed to step c4; otherwise, proceed to step c9.

[0076] In step c9, a new backend connection is created as the target connection, and step c5 is executed.

[0077] In step c10, it is determined whether the bound backend connection covers the target data node. If it covers the target data node, the bound backend connection is used as the target connection and step c5 is executed; if it does not cover the target data node, step c3 is executed.

[0078] For details of steps c1 to c10 above, please refer to the relevant descriptions in the above embodiments, which will not be repeated here.

[0079] The distributed data node access method provided in this application can save network resources and keep the number of data nodes within a range that allows for better data node performance. For example, under the same test conditions, the related technologies of keeping backend connections and frontend connections bound until the frontend binding is broken, as well as the distributed data node access method in this application, can be executed respectively. Test conditions may include: the client establishes a total of 2000 frontend connections with the proxy layer; each business transaction operation may include 5 database operations and other business logic; the 5 database operations are completed through select, commit, insert, update, and commit statements, respectively; the system throughput of the data node cluster is 50,000 transactions per second (TPS); the average cumulative latency of the 5 database operations reaches 10 milliseconds; the latency of other business logic reaches 90 milliseconds; and to achieve better data node performance, the number of backend connections corresponding to each data node needs to be 500 to 1000. If the related technology of binding backend connections to frontend connections until the frontend binding is broken is adopted, 12 proxy layers and 12 data nodes are required to establish 2000 frontend connections. Each proxy layer is allocated 166 frontend connections, and each frontend connection needs to bind to a backend connection of each data node. Therefore, each proxy layer needs to establish 166 backend connections with each data node, and each data node has a total of 2000 backend connections. If the distributed data node access method in this application embodiment is adopted, since the backend connections can be reused, only 10 proxy layers and 10 data nodes are needed. Compared with related technologies, network resources such as hardware resources can be saved by 20%. Each data node has a total of about 1000 backend connections, which is 50% less than the number of backend connections per data node compared with related technologies. It can also make the performance of data nodes better and improve the performance of the data node cluster.

[0080] A second aspect of this application provides an access device for distributed data nodes. Figure 8 This is a schematic diagram of the structure of a distributed data node access device provided in an embodiment of this application, as shown below. Figure 8 As shown, the access device 400 for the distributed data node may include a receiving module 401, a connection processing module 402, and a sending module 403.

[0081] The receiving module 401 can be used to determine the target data node accessed by the database operation statement based on the database operation statement transmitted by the client through the established front-end connection.

[0082] The connection processing module 402 can be used to obtain an idle backend connection from a preset connection pool as the target connection.

[0083] The connection pool includes idle backend connections, and the target connection is checked and synchronized to ensure that the target connection can be used normally.

[0084] The sending module 403 can be used to send database operation statements to the target data node using the target connection;

[0085] The connection processing module 402 can also be used to put the target connection into the connection pool as an idle backend connection when the database operation statement has not started a database transaction or has closed a database transaction.

[0086] In some embodiments, the connection pool includes a thread pool and a process pool, wherein the thread pool includes idle backend connections and the process pool includes idle backend connections.

[0087] The connection processing module 402 can be specifically used to: determine whether there is an idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement; if there is an idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement, obtain an idle backend connection from the thread pool and determine it as the target connection; if there is no idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement, obtain an idle backend connection from the process pool corresponding to the process to which the thread occupied by the database operation statement belongs and determine it as the target connection.

[0088] In some examples, the connection processing module 402 may be specifically used to: skip the lock waiting step from the thread pool and obtain an idle backend connection as the target connection.

[0089] In some embodiments, the connection processing module 402 can also be used to: move the idle backend connection into the process pool when the idle time of the idle backend connection in the thread pool exceeds a preset switching timeout; and release the idle backend connection when the idle time of the idle backend connection in the process pool exceeds a preset release timeout.

[0090] In some embodiments, the connection processing module 402 may also be used to: create a new backend connection when there is no idle backend connection in the connection pool, and determine the newly created backend connection as the target connection.

[0091] In some embodiments, the connection processing module 402 can also be used to: when a database transaction has been started by a database operation statement, obtain the data nodes of the backend connections that the frontend connection corresponding to the database operation statement is bound to; and when the data nodes of the backend connections that the frontend connection corresponding to the database operation statement is bound to include the target data node, determine the bound backend connection as the target connection.

[0092] The connection processing module 402 can be specifically used to: obtain an idle backend connection from the connection pool as the target connection when the data node of the backend connection bound to the frontend connection corresponding to the database operation statement does not include the target data node.

[0093] In some embodiments, the connection processing module 402 can also be used to: if the number of idle backend connections in the connection pool is less than the lower limit of the performance requirement range, then create a new backend connection as the target connection; if the number of idle backend connections in the connection pool is greater than the upper limit of the performance requirement range, and there are no idle backend connections in the connection pool, then wait until there are idle backend connections in the connection pool, and determine the idle backend connection as the target connection.

[0094] In some embodiments, the check synchronization process includes a liveness detection check process and a synchronization process.

[0095] The liveness detection process includes: checking the cluster topology of the target connection; if the cluster topology of the target connection changes, releasing the target connection, re-establishing the backend connection with the target data node, and identifying the re-established backend connection as the new target connection; or, periodically checking the cluster topology of idle backend connections in the connection pool; if the cluster topology of idle backend connections changes, releasing the idle backend connection.

[0096] Synchronization processing includes setting the database settings variables of the target connection to be consistent with those of the front-end connection.

[0097] In some embodiments, the connection processing module 402 may be specifically used to: obtain the first capability identifier of the front-end connection; search for a layer of linked list in a preset first linked list whose key matches the first capability identifier, and determine the idle back-end connection indicated by the first connection identifier in the layer of linked list whose key matches the first capability identifier as the target connection.

[0098] The first linked list includes at least one layer of linked lists. Each layer of linked lists has a key and at least one value. The key includes a capability identifier, and the value includes a connection identifier. The capability identifier of the idle backend connection indicated by the connection identifier in a layer of linked lists is the same as the capability identifier indicated by the key in the layer of linked lists. The connection identifiers in each layer of linked lists are arranged in the order in which the idle backend connection indicated by the connection identifier entered the connection pool.

[0099] In some embodiments, the connection processing module 402 may also be specifically used to: when there is no first-level linked list in the first linked list whose key is consistent with the first capability identifier, obtain the idle backend connection indicated by the first connection identifier in the preset second linked list as a candidate connection, wherein the second linked list includes the connection identifiers of idle backend connections in the connection pool, and the connection identifiers in the second linked list are arranged in the order of time when the idle backend connections indicated by the connection identifiers entered the connection pool; if the candidate connection passes the outer logic applicability verification, the candidate connection is determined as the target connection.

[0100] It should be noted that the distributed data node access device 400 is a device corresponding to the above-described distributed data node access method. All implementation methods in the above method embodiments are applicable to the embodiments of this device and can achieve the same technical effect, so they will not be described again here.

[0101] A third aspect of this application provides an agent device. Figure 9 This is a schematic diagram of the structure of an agent device provided in an embodiment of this application, as shown below. Figure 9 As shown, the agent device 500 includes a memory 501, a processor 502, and a computer program stored in the memory 501 and executable on the processor 502.

[0102] In some examples, the processor 502 described above may include a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits that may be configured to implement the embodiments of this application.

[0103] Memory 501 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the distributed data node access method according to embodiments of this application.

[0104] The processor 502 runs a computer program corresponding to the executable program code by reading the executable program code stored in the memory 501, so as to implement the distributed data node access method in the above embodiment.

[0105] In some examples, the agent device 500 may also include a communication interface 503 and a bus 504. For example, Figure 9 As shown, the memory 501, processor 502, and communication interface 503 are connected through bus 504 and complete communication with each other.

[0106] The communication interface 503 is mainly used to enable communication between various modules, devices, units, and / or equipment in the embodiments of this application. Input devices and / or output devices can also be connected through the communication interface 503.

[0107] Bus 504 includes hardware, software, or both, that couples the components of agent device 500 together. For example, and not limitingly, bus 504 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-E) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 504 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0108] A fourth aspect of this application provides a distributed data node access system, which may include a client, a proxy device, and data nodes. The client is communicatively connected to the proxy device, and the proxy device is communicatively connected to the data nodes. The proxy device can be understood as the proxy layer, the distributed data node access device, and the proxy device in the above embodiments. Specific details regarding the client, proxy device, data nodes, and the distributed data node access method can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0109] A fifth aspect of this application provides a computer-readable storage medium storing computer program instructions. When executed by a processor, these instructions implement the distributed data node access method described in the above embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. The aforementioned computer-readable storage medium may include non-transitory computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, etc., and is not limited thereto.

[0110] The sixth aspect of this application provides a computer program product, which includes a computer program that, when executed by a processor, implements the distributed data node access method in the above embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0111] It should be clarified that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For the device embodiments, equipment embodiments, system embodiments, computer-readable storage medium embodiments, and computer program product embodiments, the relevant parts can be referred to the description section of the method embodiments. This application is not limited to the specific steps and structures described above and shown in the figures. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application. Furthermore, for the sake of brevity, detailed descriptions of known methods and techniques are omitted here.

[0112] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0113] Those skilled in the art will understand that the above embodiments are exemplary and not restrictive. Different technical features appearing in different embodiments can be combined to achieve beneficial effects. Based on a study of the drawings, specification, and claims, those skilled in the art should be able to understand and implement other variations of the disclosed embodiments. In the claims, the term "comprising" does not exclude other means or steps; the quantifier "a" does not exclude a plurality; the terms "first" and "second" are used to identify names and not to indicate any particular order. No reference numerals in the claims should be construed as limiting the scope of protection. The functionality of multiple parts appearing in the claims can be implemented by a single hardware or software module. The appearance of certain technical features in different dependent claims does not mean that these technical features cannot be combined to achieve beneficial effects.

Claims

1. A method for accessing a distributed data node, characterized in that, include: Based on the database operation statement transmitted by the client through the established front-end connection, determine the target data node accessed by the database operation statement; An idle backend connection is obtained from a preset connection pool as the target connection. The connection pool includes idle backend connections. The target connection is checked and synchronized to ensure that the target connection can be used normally. The connection pool includes a thread pool and a process pool. The thread pool includes idle backend connections. The process pool includes idle backend connections. The database operation statement is sent to the target data node using the target connection; If the database operation statement has not started a database transaction or has already closed a database transaction, the target connection will be placed into the connection pool as an idle backend connection. The step of obtaining an idle backend connection from a preset connection pool as the target connection includes: Determine whether there is an idle backend connection in the thread pool corresponding to the thread used by the database operation statement; If the thread pool corresponding to the thread occupied by the database operation statement has an idle backend connection, obtain an idle backend connection from the thread pool and determine it as the target connection; If the thread pool corresponding to the thread occupied by the database operation statement does not have an idle backend connection, an idle backend connection is obtained from the process pool corresponding to the process to which the thread occupied by the database operation statement belongs and determined as the target connection.

2. The method according to claim 1, characterized in that, The step of determining the target connection by obtaining an idle backend connection from the thread pool includes: Skip the lock waiting step from the thread pool and obtain an idle backend connection to identify as the target connection.

3. The method according to claim 1, characterized in that, Also includes: If the idle time of an idle backend connection in the thread pool exceeds the preset switching timeout, the idle backend connection will be moved to the process pool. If the idle time of an idle backend connection in the process pool exceeds the preset release timeout, the idle backend connection will be released.

4. The method according to claim 1, characterized in that, Also includes: If there are no available backend connections in the connection pool, a new backend connection is created, and the newly created backend connection is designated as the target connection.

5. The method according to claim 1, characterized in that, Also includes: If a database transaction has been started by the database operation statement, obtain the data node of the backend connection that the frontend connection corresponding to the database operation statement is bound to. If the data node of the backend connection that is bound to the frontend connection corresponding to the database operation statement includes the target data node, the bound backend connection is determined as the target connection. Obtaining an idle backend connection from a preset connection pool as the target connection includes: If the target data node is not included in the data nodes of the backend connections that are already bound to the frontend connection corresponding to the database operation statement, an idle backend connection is obtained from the connection pool as the target connection.

6. The method according to claim 1, characterized in that, Also includes: If the number of idle backend connections in the connection pool is less than the lower limit of the performance requirement range, then a new backend connection is created as the target connection. If the number of idle backend connections in the connection pool is greater than the upper limit of the performance requirement range, and there are no idle backend connections in the connection pool, then wait until there are idle backend connections in the connection pool, and then determine the idle backend connection as the target connection.

7. The method according to claim 1, characterized in that, The inspection synchronization process includes liveness detection processing and synchronization processing; The liveness detection process includes: Check the cluster topology of the target connection. If the cluster topology of the target connection changes, release the target connection, re-establish the backend connection with the target data node, and determine the re-established backend connection as the new target connection. or, The cluster topology of idle backend connections in the connection pool is periodically checked. If the cluster topology of idle backend connections changes, the idle backend connections are released. The synchronization process includes: Set the database settings variables of the target connection to be consistent with those of the front-end connection.

8. The method according to claim 1, characterized in that, Obtaining an idle backend connection from a preset connection pool as the target connection includes: Obtain the first authority identifier for the front-end connection; Search for a linked list whose key matches the first capability identifier within a pre-defined first linked list. Then, identify the free backend connection indicated by the first connection identifier in the linked list whose key matches the first capability identifier as the target connection. The first linked list includes at least one layer of linked lists. Each layer of linked lists has a key and at least one value. The key includes a capability identifier, and the value includes a connection identifier. The capability identifier of the idle backend connection indicated by the connection identifier in a layer of linked lists is the same as the capability identifier indicated by the key in the layer of linked lists. The connection identifiers in each layer of linked lists are arranged in the order in which the idle backend connection indicated by the connection identifier enters the connection pool.

9. The method according to claim 8, characterized in that, Also includes: If there is no first-level linked list in the first linked list whose key is consistent with the first capability identifier, the idle backend connection indicated by the first connection identifier in the preset second linked list is obtained as a candidate connection. The second linked list includes the connection identifiers of the idle backend connections in the connection pool, and the connection identifiers in the second linked list are arranged in the order of time when the idle backend connections indicated by the connection identifiers entered the connection pool. If the candidate connection passes the outer logic applicability verification, the candidate connection is determined as the target connection.

10. A distributed data node access device, characterized in that, include: The receiving module is used to determine the target data node accessed by the database operation statement based on the database operation statement transmitted by the client through the established front-end connection. A connection processing module is used to obtain an idle backend connection from a preset connection pool as a target connection. The connection pool includes idle backend connections. The target connection undergoes a check and synchronization process to ensure that the target connection can be used normally. The connection pool includes a thread pool and a process pool. The thread pool includes idle backend connections. The sending module is used to send the database operation statement to the target data node using the target connection; The connection processing module is also used to place the target connection as an idle backend connection into the connection pool when the database operation statement has not started a database transaction or has closed a database transaction. The connection processing module is specifically used to: determine whether there is an idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement; if there is an idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement, obtain an idle backend connection from the thread pool and determine it as the target connection; if there is no idle backend connection in the thread pool corresponding to the thread occupied by the database operation statement, obtain an idle backend connection from the process pool corresponding to the process to which the thread occupied by the database operation statement belongs and determine it as the target connection.

11. An agent device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the distributed data node access method as described in any one of claims 1 to 9.

12. A distributed data node access system, characterized in that, include: Client; The proxy device as described in claim 11, wherein the proxy device is communicatively connected to the client; The data node is connected to the agent device.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the distributed data node access method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method for accessing distributed data nodes as described in any one of claims 1 to 9.

Images