Fault processing method and device for data storage resources

Abstract

The invention discloses a fault processing method and device for data storage resources, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining real-time monitoring data of data storage resources, judging a fault type and a corresponding fault processing strategy based on fault data in the data, obtaining target resources based on the fault processing strategy, and supplementing the data storage resources. According to the embodiment, the fault data can be automatically complemented, and the processing efficiency of the fault data is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for handling failures of data storage resources. Background Art

[0002] At present, in the field of Internet technology, databases, as important data storage resources, occupy an important position in the system. During use, once a database fails, it will affect the normal provision of system services and the user experience, so it is necessary to handle the faulty data in a timely manner. The common fault handling method is to notify the relevant technical personnel after the failure occurs, and the technical personnel will judge the database information, manually connect the new database machine to the current architecture, and perform subsequent verification and other operations.

[0003] However, in an environment with high-concurrency network failures, large-scale downtime, and multiple machine failures, the time nodes for manual fault handling cannot be fully controlled and the processing time is long, which will delay the implementation of operations and extend the database failure time; and the high error rate of manual operations may also cause data inconsistency or further damage to the system. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method and device for handling faults of data storage resources, which can improve the handling efficiency and accuracy of faults of data storage resources.

[0005] To achieve the above objective, according to one aspect of an embodiment of the present invention, a method for handling a data storage resource failure is provided, comprising:

[0006] Obtain real-time monitoring data of the data storage resource cluster;

[0007] In response to the real-time monitoring data containing fault data, determining the fault type based on the fault data;

[0008] Determine the corresponding fault handling strategy based on the fault type;

[0009] Obtain target resources based on the fault handling strategy and supplement data storage resources based on the target resources.

[0010] Optionally, before acquiring the target resource based on the fault handling strategy, the method further includes: determining that a resource supplement operation is required based on the fault handling strategy;

[0011] The method also includes:

[0012] In response to determining that resource replenishment is not required based on the fault handling strategy, the faulty data is repaired online.

[0013] Optionally, if the fault type is a hardware fault, it is determined based on the fault handling strategy that data storage resources need to be supplemented, including:

[0014] Determine the cause of the fault based on the fault data;

[0015] Determine whether the hardware fault can be automatically repaired based on the fault cause and the preset automatic repair fault table;

[0016] In response to the hardware failure not being able to be automatically repaired, it is determined that data storage resources need to be supplemented.

[0017] Optionally, target resources are acquired based on the fault handling strategy, and data storage resources are supplemented according to the target resources, including:

[0018] Obtaining hardware configuration information and a first topological relationship corresponding to the fault data;

[0019] Based on the configuration information of the hardware corresponding to the fault data and the first topological relationship, a target resource is acquired from the available resource list, and data storage resources are supplemented according to the target resource.

[0020] Optionally, if the fault type is a database capacity fault, it is determined based on the fault handling strategy that data storage resources need to be supplemented, including:

[0021] Obtaining capacity adjustment information of data storage resources corresponding to the fault data;

[0022] Determining whether the adjusted data storage resources can meet the usage conditions of the data storage resources based on the capacity adjustment information;

[0023] In response to the adjusted data storage resources being able to meet the usage conditions of the data storage resources, it is determined that the data storage resources need to be supplemented.

[0024] Optionally, target resources are acquired based on the fault handling strategy, and data storage resources are supplemented according to the target resources, including:

[0025] Acquire a second topological relationship corresponding to the fault data;

[0026] Determine a corresponding resource acquisition method according to the second topological relationship type;

[0027] The target resource is acquired from the available resource list by using the resource acquisition method corresponding to the second topological relationship type, and data storage resources are supplemented according to the target resource.

[0028] Optionally, acquiring the target resource from the available resource list by using a resource acquisition method corresponding to the second topological relationship type, and supplementing the data storage resource according to the target resource, including:

[0029] In response to the second topological relationship being a sub-library and sub-table type, determining a target computer room with the most resources in the computer room set;

[0030] Based on the capacity adjustment information, determine whether the resources in the target computer room meet the application conditions;

[0031] In response to the resources in the target computer room satisfying the usage conditions, the target resources are determined from the resources in the target computer room, and data storage resources are supplemented according to the target resources.

[0032] Optionally, the fault type is a network fault or a downtime fault, and it is determined based on the fault handling strategy that data storage resources need to be supplemented, including:

[0033] Perform survivability detection on the data storage resources corresponding to the fault data according to a preset period;

[0034] In response to a survival check result of failure, it is determined that data storage resources need to be supplemented.

[0035] Optionally, target resources are acquired based on the fault handling strategy, and data storage resources are supplemented according to the target resources, including:

[0036] Obtain configuration information and a third topological relationship corresponding to the fault data;

[0037] Based on the configuration information of the hardware corresponding to the fault data and the third topological relationship, the target resource is obtained from the available resource list, and the data storage resource is supplemented according to the target resource.

[0038] Optionally, data storage resources are supplemented according to the target resources, including:

[0039] Data synchronization is performed based on the target resource and the data storage resource corresponding to the fault data to obtain a target data storage resource containing synchronized data.

[0040] Optionally, after data synchronization is performed based on the target resource and the data storage resource corresponding to the fault data to obtain the target data storage resource containing the synchronized data, the method further includes:

[0041] Verify whether the data of the target data storage resource is consistent with the data storage resource corresponding to the fault data;

[0042] In response to the verification result being consistent, the data storage resource corresponding to the faulty data is taken offline.

[0043] According to another aspect of an embodiment of the present invention, a device for handling a fault of a data storage resource is provided, comprising:

[0044] An acquisition module is used to obtain real-time monitoring data of a data storage resource cluster;

[0045] A judgment module, configured to judge the fault type based on the fault data in response to the real-time monitoring data containing the fault data;

[0046] A determination module is used to determine a corresponding fault handling strategy according to the fault type;

[0047] Obtain target resources based on the fault handling strategy and supplement data storage resources based on the target resources.

[0048] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the fault handling method for data storage resources provided by an embodiment of the present invention.

[0049] According to another aspect of an embodiment of the present invention, a computer readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for handling a fault of a data storage resource provided by an embodiment of the present invention is implemented.

[0050] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method for handling a fault of a data storage resource provided by an embodiment of the present invention is implemented.

[0051] An embodiment of the above invention has the following advantages or beneficial effects: it can monitor the data storage resource cluster and determine whether there is fault data through monitoring data. When fault data exists, a corresponding fault handling strategy is selected according to the fault type, and the target resources are obtained through the strategy. The data storage resources are supplemented based on the target resources, thereby realizing automatic supplementation of the data storage resources corresponding to the fault data and improving the efficiency and accuracy of fault handling.

[0052] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0054] Figure 1 is a schematic diagram of main steps of a method for handling a data storage resource failure according to an embodiment of the present invention;

[0055] Figure 2 A flowchart for the automatic schema completion function for the database;

[0056] Figure 3 Assign logical diagrams to different interfaces;

[0057] Figure 4 is a schematic diagram of main modules of a device for handling a fault of a data storage resource according to an embodiment of the present invention;

[0058] Figure 5 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0059] Figure 6 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0061] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0062] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0063] The existing fault handling method is mainly implemented manually, which is time-consuming, not accurate, and has low processing efficiency. Based on this, the present invention provides a database automatic architecture completion function. By implementing this function through the system, it is possible to automatically handle the failure of data storage resources and improve the efficiency of fault handling. Database automatic architecture completion function: Determine the computer room information that needs to be found according to the database role information (master database, slave database, disaster recovery) where the failure occurs. After determining the computer room information, find the idle machines and database package (CPU / memory / hard disk) information under this computer room, obtain the IP address information of this machine and lock it, and then perform the database deployment task. After the automated deployment task is completed, perform the database recovery operation. This recovery operation is full recovery + incremental recovery. After completion, the existing production master database can be synchronized in real time.

[0064] Figure 1 1 is a schematic diagram of the main process of a method for handling a fault of a data storage resource according to an embodiment of the present invention, including:

[0065] Step S101: acquiring real-time monitoring data of a data storage resource cluster.

[0066] By monitoring the data storage resource cluster through the entire link, real-time monitoring data can be obtained. The data storage resource cluster can be a database resource. Specifically, the monitored data includes whether the machine monitoring the database can be pinged (end-to-end connectivity), whether the database domain name information can be used to access the specific database, whether the telnet (remote login) database is normal, and whether the CPU (central processing unit), memory, and disk status of the database machine are normal. Through monitoring data, faults in the data storage resource cluster can be discovered in a timely manner, and corresponding fault handling strategies can be executed.

[0067] Step S102 : in response to the real-time monitoring data containing fault data, determining the fault type based on the fault data.

[0068] For the acquired real-time monitoring data, if it contains fault data, it is necessary to determine its fault type. For example, the fault data of CPU / memory / hard disk and other resources corresponds to the fault type of database machine failure, and the fault type corresponding to receiving a large number of database alarm information and being unable to access the database is the large-scale database downtime. The fault types mainly include database and its faults, database network faults, large-scale database downtime, database expansion and contraction, and database splitting and merging.

[0069] Optionally, judgment rules for monitoring data are pre-set to determine fault data in the monitoring data. For example, a CPU usage threshold is set to 90%. If it exceeds the threshold, it is determined to be fault data, and the corresponding fault type is database machine failure. A threshold for the number of database alarm messages is set. If alarm messages exceeding the threshold are received within a certain period of time, they are determined to be fault data, and the corresponding fault type is large-scale database downtime.

[0070] Optionally, the fault data may be database error information, including:

[0071] A. CPU:

[0072] A1: If it is a motherboard failure, analyze the error

[0073] A2: If it is a CPU failure, analyze the error

[0074] A3: Suspected CPU failure

[0075] B. Memory:

[0076] B1: If it is a memory failure, the error is analyzed as follows: There is a problem with the memory device: The dimm a1 correction error rate has exceeded the allowed threshold

[0077] B2: If it is a memory failure, the analysis error is: the memory has an uncorrectable ECC (Error Checking and Correcting) error

[0078] B3: If it is a memory failure, the analysis error is: A multi-bit memory error is detected at the DIMM_B6 position of the memory device.

[0079] B4: If it is a memory failure, the error is analyzed as follows: An uncorrectable memory error was detected on memory module (DIMM) 11, the specific address is 0000001E40AD72C0. 278CBA61-VC10-FRU 01KR354

[0080] C. Hard disk:

[0081] C1: If the hard disk array card fails, analyze the error

[0082] C2: RAID card fault analysis error: array card cache failure

[0083] C3: RAID card fault analysis error: RAID card reset

[0084] C4: Fan failure analysis error

[0085] C5: If it is a power module fault, the error report is: PERC1 battery power is low.

[0086] C6: If the hard disk array card fails, the error is: the array card physical disk is lost: raid pd islost.

[0087] Step S103: determine a corresponding fault handling strategy according to the fault type.

[0088] After determining the fault type, the fault handling strategy corresponding to the fault data is determined according to the pre-stored correspondence between the fault type and the fault handling strategy for subsequent fault handling operations. Among them, the fault types include hardware faults, database capacity faults, network faults or downtime faults, and the corresponding fault handling strategies include hardware fault handling strategies, database capacity fault handling strategies and network fault or downtime fault handling strategies.

[0089] Optionally, the fault type can be selectively set according to actual conditions, and the number of set fault types can also be changed, without limitation.

[0090] Step S104: acquiring target resources based on the fault handling strategy, and supplementing data storage resources according to the target resources.

[0091] According to the determined fault handling strategy, the corresponding target resource is obtained from the available resource pool, and the target resource is used as a new data storage resource to replace the data storage resource where the faulty data is located to complete the database architecture, and execute subsequent functions. Specifically, the resources of the determined new database are used to replace the resources of the faulty database to execute functions, and the faulty database is offline.

[0092] Optionally, when the fault is a hardware fault, a target database with sufficient resources for the database of the faulty data is selected from the available resource pool according to parameter information of the database where the faulty data is located, and after performing data synchronization operation on the target database, the target database is used to execute functions instead of the faulty database.

[0093] The fault handling method for data storage resources provided in an embodiment of the present invention can monitor the data storage resource cluster and determine whether there is fault data through monitoring data. When fault data exists, a corresponding fault handling strategy is selected according to the fault type, and target resources are obtained through the strategy. Data storage resources are supplemented based on the target resources, thereby realizing automatic supplementation of data storage resources corresponding to the fault data and improving the efficiency and accuracy of fault handling.

[0094] Optionally, before acquiring the target resource based on the fault handling strategy, it also includes: determining the need to supplement the data storage resources based on the fault handling strategy; the fault handling method also includes: in response to determining that the data storage resources do not need to be supplemented based on the fault handling strategy, performing online repair on the faulty data. After determining the fault handling strategy, before acquiring the target resource, it is also necessary to determine whether the database failure requires a resource supplement operation. The judgment condition is that the faulty data can be repaired online in the database. If it can be repaired online, the database of the faulty data does not need to be taken offline. If it cannot be repaired online, the database of the faulty data needs to be repaired offline, and a new database is required to replace it to implement the corresponding function. Among them, the method of judging whether it can be repaired online can be achieved through operations such as task retry and master-slave database switching.

[0095] Optionally, the fault type is a hardware fault, and the need to supplement data storage resources is determined based on the fault handling strategy, including: determining the cause of the fault based on the fault data; determining whether the hardware fault can be automatically repaired based on the fault cause and a preset automatic repair fault table; and determining that data storage resources need to be supplemented in response to the hardware fault not being able to be automatically repaired. When the fault type is a hardware fault, it is necessary to determine whether to supplement data storage resources. First, it is necessary to determine the cause of the fault based on the fault data. For example, the fault data is an error message: Slot7 hard disk failure, and the cause of the fault is determined to be hard disk failure B2. Obtain the preset automatic repair fault table from the pre-stored data, and determine whether the cause of the fault exists in the automatic repair fault table. If so, it means that it can be automatically repaired, otherwise it cannot be automatically repaired. If it cannot be automatically repaired, resources need to be supplemented.

[0096] Optionally, the preset automatic repair fault table includes the following faults:

[0097] A. Replace the network cable:

[0098] A1: The error reported after analysis is: bond0: bond id negotiation is inconsistent, please check the network card and network cable; network card with bond rate less than 1000M: eth5; bond rate is inconsistent, please check the network cable: eth5; network card status is abnormal, please check the network card and network cable. Network card: eth5;

[0099] A2: The analysis error is: bnxt_en 0000:1a:00.1 eth3: NIC Link is Down.

[0100] B. Hard disk:

[0101] B1: Hard disk fault analysis error: Fault description ens4f0 down;

[0102] B2: Hard disk failure analysis error: Slot7 hard disk failure;

[0103] B3: Hard disk failure analysis error: kernel: EXT4-fs warning (device sdad): __ext4_read_dirblock:902: error reading directory block (ino 2, block 0);

[0104] B4: Hard disk failure analysis error: Slot7 hard disk failure;

[0105] B5: Hard disk fault analysis error: Hard disk light alarm;

[0106] B6: Hard disk fault analysis error: Hard disk slot 2 alarm;

[0107] B7: Hard disk failure analysis error: slot0 disk failure.

[0108] C. Memory failure

[0109] C1: Memory failure analysis error: The system restarted due to unknown reasons.

[0110] C2: Memory failure analysis error: An uncorrectable error occurred in the server's memory (DIMM). Error code: 0x2C00000F.

[0111] Optionally, the target resource is obtained based on the fault handling strategy, and the data storage resource is supplemented according to the target resource, including: obtaining the configuration information and the first topological relationship of the hardware corresponding to the fault data; based on the configuration information and the first topological relationship of the hardware corresponding to the fault data, the target resource is obtained from the available resource list, and the data storage resource is supplemented according to the target resource. When the fault is a hardware fault, the database system calls the resource supplement function through the interface. The hardware configuration information and the first topological relationship of the database where the fault data is located, that is, the master-slave relationship of the database, are obtained. The master library corresponding to the fault data is determined according to the first topological relationship. If the database where the fault data is located is the master library, no further determination is required. If the library where the fault data is located is a slave library or a disaster recovery library, the master library corresponding to it needs to be determined according to the first topological relationship. After determining the master library, the database resources and the corresponding topological relationship that can cover the resource are searched from the available resource list according to the resources of the master library, that is, the master-slave-disaster database resources are obtained for supplementing the data storage resources.

[0112] Specifically, resource acquisition is achieved through the allocate_server interface, and the input parameters of the interface are the server package configuration and the specified computer room information. Among them, the specified computer room information is the information of the computer room where the faulty data is located. The corresponding package is queried from the available server resource pool of the specified computer room, that is, the number of machines that meet the resources of the database where the faulty data is located, and the list of available servers in the computer room is obtained. A server information is randomly selected from the list of available servers. Resources are supplemented based on the determined package and the selected server information. Resource supplementation specifically includes automatically executing deployment and recovery tasks based on the determined package and server to complete the topology, and realizing the full and incremental synchronization of the faulty database data.

[0113] Optionally, the interface input parameter may also include a non-selectable server list and optional network segment information. When randomly selecting server information, remove the servers in the non-selectable server list from the available server list, and randomly select a server as a supplementary server from the remaining servers within the optional network segment range.

[0114] Optionally, the fault type is a database capacity fault, and it is determined based on the fault handling strategy that data storage resources need to be supplemented, including: obtaining capacity adjustment information of the data storage resources corresponding to the fault data; judging whether the adjusted data storage resources can meet the use conditions of the data storage resources based on the capacity adjustment information; and judging that the data storage resources need to be supplemented in response to the adjusted data storage resources being able to meet the use conditions of the data storage resources. When the fault type is a database capacity fault, that is, when operations such as expansion and contraction, splitting / merging, etc. are required, it is necessary to call the cluster_allocate_server interface for an automatic supplement process, including the following steps. Obtain capacity adjustment information, that is, information on the adjusted database capacity, such as expanding the database capacity to twice the original capacity. Determine whether the adjusted database capacity information can meet the use of the database, that is, whether the adjusted database capacity can meet the normal function of the database. If it cannot meet the requirements, the subsequent automatic supplement operation will not be performed; if it can meet the requirements, it is determined that automatic supplementation is required. Among them, the capacity adjustment information includes disk, memory and other data; the input parameters of the cluster_allocate_server interface include: a topology list of multiple sets of mysql corresponding to the fault data database, database hardware configuration information and the main database information of the database corresponding to the fault data.

[0115] Optionally, the acquired capacity adjustment information is data generated after a DBA (database administrator) intervenes in the communication to confirm relevant risk points and confirms that operations such as database expansion and contraction, database splitting and merging can be performed.

[0116] Optionally, target resources are acquired based on the fault handling strategy, and data storage resources are supplemented according to the target resources, including: acquiring a second topological relationship corresponding to the fault data; determining a corresponding resource acquisition method according to the second topological relationship type; acquiring the target resource from a list of available resources through the resource acquisition method corresponding to the second topological relationship type, and data storage resources are supplemented according to the target resource.

[0117] For database capacity failure, when the data storage resources adjusted based on the capacity adjustment information can meet the usage conditions, the data storage resources before adjustment, that is, the second topological relationship of the data storage resources corresponding to the faulty data, are obtained to determine the type of topological relationship. The corresponding method is selected according to the type of topological relationship. If it is a sub-database and sub-table type, it is necessary to determine the computer room with the most resources in the preset computer room set as the target computer room, which can be the computer room with the largest number of computer rooms, so as to give priority to the allocation of computer room resources with more idle resources. If the selected target computer room cannot allocate all server resources, the second largest computer room is selected as the new target computer room, and then the mysql_allocate_server interface is called to perform the following steps. Based on the resources of the target computer room, it is determined whether the number of available machines meets the adjusted database capacity, specifically whether the number of the adjusted master library + slave library is met. In addition, it is necessary to further determine whether other computer rooms other than the target computer room can meet the number of the adjusted disaster recovery library. When the resources of the target computer room do not meet the adjusted database capacity, or other computer rooms do not meet the number of disaster recovery libraries, the result of insufficient resources and inability to allocate is returned. When the corresponding number can be met, it means that resources can be supplemented based on the target computer room.

[0118] Specifically, resources are replenished by calling the allocate_server interface, including randomly obtaining a server information from the available server list of the determined target computer room, and calling the interface multiple times to select a server for allocation according to the required number. Each selected server needs to be marked as allocated to avoid repeated selection.

[0119] Optionally, the servers marked as allocated are also written to the list of unavailable servers.

[0120] Optionally, if the topological relationship type is determined to be MySQL (relational database management system) type, the mysql_allocate_server interface is called to supplement resources. The input parameters of this interface include topological relationship information, database hardware configuration information, the target computer room / the computer room where the main database corresponding to the fault data is located, and a list of unselectable servers. According to the database hardware configuration information, select which computer room has the most resources, and determine whether the resources of the selected computer room can meet the adjusted database capacity. If not, the result is returned directly, and insufficient resources cannot be allocated. If satisfied, the allocate_server interface is called to allocate the required number of servers. Specifically, server information is randomly obtained from the list of available servers in the selected computer room as the selected server.

[0121] Optionally, the fault type is a network fault or a downtime fault, and the need for data storage resource supplementation is determined based on the fault handling strategy, including: performing a survival test on the data storage resource corresponding to the fault data according to a preset period; in response to the survival test result being a failure, determining that data storage resources need to be supplemented. For network faults and downtime roadblocks, since there is the possibility of problems that can be repaired in a short time, such as network fluctuations and flash disconnections, it is necessary to determine whether a resource supplement operation is required before performing resource supplementation. The judgment method is to detect and access the database through methods such as ping and telnet. If jitter occurs, the detection failure will be returned. After an interval of 3 seconds, the detection will be performed again. If the database can be accessed normally, it is confirmed to be jitter or restart, and no other operations are required. If the current detection access fails, a detection will be initiated again after waiting for 3 seconds. If it fails again this time, it is confirmed that there is a problem with the database network, and it is no longer jitter, so the completion operation is performed.

[0122] Optionally, the judgment method also includes that if a large number of database alarm messages are received, it indicates that a large area of ​​the database is abnormal, and if the alarm information includes a downtime alarm, it is determined that automatic completion is required.

[0123] Optionally, target resources are obtained based on the fault handling strategy, and data storage resources are supplemented according to the target resources, including: obtaining configuration information and a third topological relationship corresponding to the fault data; obtaining target resources from a list of available resources based on the configuration information and a third topological relationship of the hardware corresponding to the fault data, and data storage resources are supplemented according to the target resources. When the fault type is a network fault or a downtime fault, the main database where each fault data is located is determined, and the allocate_server interface is called to allocate resources. Specifically, the configuration information of the database corresponding to the fault data and the third topological relationship of the database are obtained, and the main database corresponding to the fault database is determined according to the third topological relationship. The target resource is obtained from the list of available resources in the computer room where the main database is located, including the server information in the list of available servers. A server is randomly selected from the list of available servers for allocation.

[0124] Optionally, after determining that the database machine is down, before performing the resource replenishment operation, it is necessary to perform a master-slave switch of the database. After the master-slave switch is completed, the new master database is used to provide functions, and then the resource replenishment operation is performed. When performing the resource replenishment operation, the master database corresponding to the fault data is determined to be the master database before the master-slave switch.

[0125] The fault handling method for data storage resources provided in an embodiment of the present invention can select different handling strategies according to the fault type, determine the type of resources that need to be acquired and the target resources to be acquired according to the strategy, realize automatic supplementation of resources for fault data, and improve the processing efficiency and accuracy of fault data.

[0126] Optionally, data storage resources are supplemented according to the target resources, including: data synchronization based on the target resources and the data storage resources corresponding to the faulty data, to obtain the target data storage resources containing the synchronized data. After the target resources are determined, data synchronization operations need to be performed based on the data in the database where the faulty data is located. Among them, the determined target resources are the resources of the new database, and data synchronization is performed based on the faulty data database and the new database, including synchronization of full data and incremental data, to ensure the data integrity of the new database.

[0127] Optionally, after data synchronization is performed based on the target resource and the data storage resource corresponding to the fault data, and the target data storage resource containing the synchronized data is obtained, the fault handling method further includes: verifying whether the data of the target data storage resource is consistent with the data storage resource corresponding to the fault data; in response to the verification result being consistent, taking the data storage resource corresponding to the fault data offline. Before the new database is online and the faulty database is offline, data verification is required to check whether the new database data is consistent with the faulty database data. If they are inconsistent, resynchronization is required. If they are consistent, synchronization is completed, the faulty machine is offline, and the corresponding old server is offline; the new machine is online, and the new server is online.

[0128] According to the data storage resource fault handling method provided by the embodiment of the present invention, it is possible to monitor the database cluster data, and when fault data exists, determine the fault type and whether it can be automatically repaired. For fault data that cannot be automatically repaired, resources need to be supplemented through the database automatic architecture completion function, and after determining the target resource to be supplemented, synchronize the data of the new and old databases to ensure data integrity and avoid data loss. After data synchronization, the machine of the faulty data database is offline, and the machine in the target resource is online to complete the database topology, thereby realizing the automatic supplement of data storage resources.

[0129] like Figure 2The figure shows the flowchart of the automatic database architecture completion function. When the database usage rate is too low or reaches a performance bottleneck, the relevant person in charge will submit a database expansion and contraction work order, and the work order will call this system for processing. Among them, this system is a system that realizes the automatic database architecture completion function. This system obtains the topological architecture of the original database and executes the subsequent process according to the newly applied hardware configuration information. Perform a pre-check before allocating the machine to determine whether the submitted database hardware configuration information (disk, memory, etc.) can meet the database usage. Determine the original database topology type. If it is a sub-library and sub-table type, call the cluster_allocate_server interface to allocate the machine and obtain the allocation result; if it is a mysql type, call the mysql_allocate_server interface to allocate the machine and obtain the allocation result. When a hardware failure occurs in the database (such as hard disk, memory, etc.), the system operation and maintenance tool automatically notifies this system through an interface call. This system obtains relevant information (hardware configuration information of the faulty instance, master-slave relationship, etc.) based on the faulty instance and executes the subsequent process based on this information. Connect to other replica sets in the topology to find the current master database, and call the allocate_server interface to allocate a database server based on the current master database, deploy and restore a single replica to complete the topology, and obtain the result of machine allocation. When the database monitoring has a downtime exception or network anomaly, this system obtains the downtime information by polling the interface of the database monitoring system. Every few seconds, the database and server are detected (i.e., whether the database and server are alive). If the detection fails after exceeding the specified time threshold, it is determined that the machine is down. If the detection is successful, it is determined to be a restart or network jitter, and the logic of automatically completing the topology architecture when the downtime is not triggered. If the detection fails, it is determined that the database server is down, connect to other replica sets in the topology to find the current master database, and call the allocate_server interface to allocate a database server based on the current master database, deploy and restore a single replica to complete the topology, and obtain the result of machine allocation. After obtaining the result of machine allocation, determine whether the resources can meet the allocation result. If not, it means that the resources are insufficient, the allocation fails, and wait for manual intervention. If it is satisfied, it means that the resources are sufficient and the allocation is successful. Automatically execute deployment and recovery tasks to complete the topology, and synchronize the existing master database in real time. Automatically refresh, detect high-availability information metadata, verify architecture topology, master-slave synchronization, etc., and ensure that high-availability switching is normal under the current architecture to ensure metadata accuracy. Before going offline, verify that real-time data synchronization is normal, that the GTID (global transaction identifier) ​​of the new and old databases is consistent, and that there is no data loss or other problems, and then call the offline interface to perform offline operations. Call the offline interface to automatically offline the faulty machine / automatically offline the old server before scaling up or down.

[0130] like Figure 3The following is a schematic diagram of the allocation logic of different interfaces. allocate_server interface (process identifier 1): allocate a single MySQL database server. Method input parameters: server package configuration, specified computer room information, unavailable server list, and available network segment information. Find the server list of the corresponding package and computer room from the available server resource pool. In the available server list, remove the information in the unavailable server list. Randomly obtain a server information from the available server list (to avoid uneven use of the resource pool), perform a pre-use check (check the disk status, etc.). If it does not meet the requirements, remove the server and repeat the above operation. mysql_allocate_server interface (process identifier 2): automatically allocate a topology architecture server. Process input parameters: topology architecture information, database hardware configuration information, computer room selected by the main database, unavailable server list, etc. According to the input configuration information, query which computer room group has the most packages (supports searching from the specified computer room). The purpose is to prioritize the allocation of the most idle packages to avoid uneven use of server resources in different computer rooms. Determine whether the available number of packages in the computer room is more than the number of master libraries + slave libraries in the same computer room applied for, and determine whether the available number of packages in other computer rooms is more than the number of disaster recovery libraries in different computer rooms applied for. If not, the result is returned directly, indicating that insufficient resources cannot be allocated; if so, the allocate_server interface is called multiple times to allocate the required number of servers. After each call to obtain a server, the server is marked as allocated (written to the list of unselectable servers). The allocate_server interface randomly obtains a server from the list of available servers in the specified computer room and allocates it. cluster_allocate_server interface (process identifier 3): Automatic allocation of database servers for cluster types with separate databases and tables (multiple MySQL topologies) Input parameters: multiple MySQL topology lists, database hardware configuration information, and specified master database information. According to the hardware configuration information passed in, find which computer room group has the most packages (supports searching from the specified computer room list) in order to prioritize the most idle packages and avoid uneven use of server resources in different computer rooms. The mysql_allocate_server interface is called first from the computer room with the most resources to allocate servers. When all servers cannot be allocated from one computer room, the allocation logic is continued from the computer room with the second most resources and the automatically allocated server information (IP, computer room, etc.) is returned. If the resources are insufficient, it is impossible to allocate enough machines and the insufficient resources information is returned. The mysql_allocate_server interface makes a judgment based on the determined computer room to determine whether the available number of packages in the computer room is more than the number of master libraries + slave libraries in the same computer room applied for, and whether the available number of packages in other computer rooms is more than the number of disaster recovery libraries applied for in different computer rooms.If not, the result is returned directly, indicating that insufficient resources cannot be allocated; if so, the allocate_server interface is called multiple times to allocate the required number of servers. After each call to obtain a server, the server is marked as allocated (written to the list of unavailable servers).

[0131] Figure 4 FIG. 4 is a schematic diagram of main modules of a device 400 for handling a fault of a data storage resource according to an embodiment of the present invention. Figure 4 As shown, including:

[0132] Acquisition module 401, used to acquire real-time monitoring data of the data storage resource cluster;

[0133] A determination module 402, configured to determine a fault type based on the fault data in response to the real-time monitoring data containing the fault data;

[0134] The determination module 403 is used to determine the corresponding fault handling strategy according to the fault type;

[0135] The supplement module 404 is used to obtain target resources based on the fault handling strategy and supplement data storage resources according to the target resources.

[0136] The fault handling device for data storage resources provided in an embodiment of the present invention can monitor the data storage resource cluster and determine whether there is fault data through monitoring data. When fault data exists, a corresponding fault handling strategy is selected according to the fault type, and target resources are obtained through the strategy. Data storage resources are supplemented based on the target resources, thereby realizing automatic supplementation of data storage resources corresponding to the fault data and improving the efficiency and accuracy of fault handling.

[0137] Optionally, the device also includes: a judgment module 402, which is used to determine whether data storage resources need to be supplemented based on the fault handling strategy; the device also includes: a judgment module 402, which is used to perform online repair of faulty data in response to determining that data storage resources do not need to be supplemented based on the fault handling strategy.

[0138] Optionally, the fault type is a hardware fault, and a judgment module 402 is used to determine the need for data storage resource supplementation based on a fault handling strategy, including: a determination module 403, used to determine the cause of the fault based on the fault data; a judgment module 402, used to determine whether the hardware fault can be automatically repaired based on the fault cause and a preset automatic repair fault table; in response to the hardware fault not being able to be automatically repaired, determining the need for data storage resource supplementation.

[0139] Optionally, a supplement module 404 is used to acquire target resources based on a fault handling strategy and to supplement data storage resources according to the target resources, including: an acquisition module 401, used to acquire the configuration information and the first topological relationship of the hardware corresponding to the fault data; a supplement module 404, used to acquire the target resources from a list of available resources based on the configuration information and the first topological relationship of the hardware corresponding to the fault data, and to supplement data storage resources according to the target resources.

[0140] Optionally, the fault type is a database capacity fault, and a judgment module 402 is used to determine the need to supplement data storage resources based on a fault handling strategy, including: an acquisition module 401, used to obtain capacity adjustment information of the data storage resources corresponding to the fault data; a judgment module 402, used to determine whether the adjusted data storage resources can meet the use conditions of the data storage resources based on the capacity adjustment information; in response to the adjusted data storage resources being able to meet the use conditions of the data storage resources, determining that the data storage resources need to be supplemented.

[0141] Optionally, a supplement module 404 is used to acquire target resources based on a fault handling strategy and to supplement data storage resources according to the target resources, including: an acquisition module 401, used to acquire a second topological relationship corresponding to fault data; a determination module 403, used to determine a corresponding resource acquisition method according to the second topological relationship type; a supplement module 404, used to acquire the target resource from a list of available resources through a resource acquisition method corresponding to the second topological relationship type, and to supplement data storage resources according to the target resource.

[0142] Optionally, the supplement module 404 is used to obtain the target resource from the available resource list through the resource acquisition method corresponding to the second topological relationship type, and supplement the data storage resources according to the target resource, including: a determination module 403, used to determine the target computer room with the most resources in the computer room set in response to the second topological relationship being a sub-library and sub-table type; a judgment module 402, used to judge whether the resources in the target computer room meet the application conditions according to the capacity adjustment information; a supplement module 404, used to determine the target resource from the resources in the target computer room in response to the resources in the target computer room meeting the usage conditions, and supplement the data storage resources according to the target resource.

[0143] Optionally, the fault type is a network fault or a downtime fault, and the judgment module 402 is used to determine the need for data storage resource replenishment based on the fault handling strategy, including: a detection module 405 (not shown in the figure), which is used to perform a survival check on the data storage resources corresponding to the fault data according to a preset period; a determination module 403, which is used to determine the need for data storage resource replenishment in response to a survival check result of failure.

[0144] Optionally, the supplement module 404 is used to obtain the target resources based on the fault handling strategy and supplement the data storage resources according to the target resources, including: the acquisition module 401 is used to obtain the configuration information and the third topological relationship corresponding to the fault data; the supplement module 404 is used to obtain the target resources from the list of available resources based on the configuration information and the third topological relationship of the hardware corresponding to the fault data, and supplement the data storage resources according to the target resources.

[0145] Optionally, the supplement module 404 is used to supplement the data storage resources according to the target resources, including: the supplement module 404 is used to synchronize data based on the target resources and the data storage resources corresponding to the fault data to obtain the target data storage resources containing synchronized data.

[0146] Optionally, the device further includes: a verification module 406, used to verify whether the data of the target data storage resource is consistent with the data storage resource corresponding to the fault data; and an offline module, used to offline the data storage resource corresponding to the fault data in response to the verification result being consistent.

[0147] The fault handling device for data storage resources provided by the embodiment of the present invention can monitor the database cluster data, and when fault data exists, determine the fault type and whether it can be automatically repaired. For fault data that cannot be automatically repaired, resources need to be supplemented through the database automatic architecture completion function, and after determining the target resource to be supplemented, the data of the new and old databases are synchronized to ensure the integrity of the data and avoid data loss. After data synchronization, the machine of the faulty data database is offline, and the machine in the target resource is online to complete the database topology, thereby realizing the automatic supplement of data storage resources.

[0148] Figure 5 An exemplary system architecture 500 is shown to which the method for handling a data storage resource failure or the device for handling a data storage resource failure according to the embodiment of the present invention can be applied.

[0149] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0150] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0151] The terminal devices 501 , 502 , and 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0152] The server 505 may be a server that provides various services, such as a background management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 501, 502, and 503. The background management server may analyze and process the received data such as resource supplement requests, and feed back the processing results (such as target resources - only an example) to the terminal device.

[0153] It should be noted that the data storage resource fault handling method provided in the embodiment of the present invention is generally executed by the server 505 , and accordingly, the data storage resource fault handling device is generally disposed in the server 505 .

[0154] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0155] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 6 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0156] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0157] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0158] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present invention are executed.

[0159] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be provided in a processor. For example, they may be described as follows: a processor includes an acquisition module, a judgment module, a determination module, and a supplementary module. The names of these units or modules do not, in certain cases, constitute limitations on the units or modules themselves. For example, the acquisition module may also be described as a "module for acquiring real-time monitoring data of a data storage resource cluster."

[0162] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0163] Obtain real-time monitoring data of the data storage resource cluster;

[0164] In response to the real-time monitoring data containing fault data, determining the fault type based on the fault data;

[0165] Determine the corresponding fault handling strategy based on the fault type;

[0166] Obtain target resources based on the fault handling strategy and supplement data storage resources based on the target resources.

[0167] According to the technical solution of the embodiment of the present invention, it is possible to monitor the data storage resource cluster and determine whether there is fault data through the monitoring data. When fault data exists, a corresponding fault handling strategy is selected according to the fault type, and the target resources are obtained through the strategy. The data storage resources are supplemented based on the target resources, thereby realizing automatic supplementation of the data storage resources corresponding to the fault data and improving the efficiency and accuracy of fault handling.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for handling a data storage resource failure, characterized in that: include: Obtain real-time monitoring data of the data storage resource cluster; In response to the real-time monitoring data containing fault data, determining the fault type based on the fault data; Determine a corresponding fault handling strategy according to the fault type; A target resource is acquired based on the fault handling strategy, and data storage resources are supplemented according to the target resource.

2. The method according to claim 1, characterized in that Before acquiring the target resource based on the fault handling strategy, the method further includes: determining the need to supplement data storage resources based on the fault handling strategy; The method further comprises: In response to determining that no data storage resource supplement is required based on the fault handling strategy, the fault data is repaired online.

3. The method according to claim 2, characterized in that The fault type is a hardware fault, and the determining based on the fault handling strategy that data storage resources need to be supplemented includes: determining a fault cause based on the fault data; Determine whether the hardware fault can be automatically repaired based on the fault cause and a preset automatic repair fault table; In response to the hardware failure being unable to be automatically repaired, it is determined that data storage resources need to be supplemented.

4. The method according to claim 3, characterized in that The acquiring of target resources based on the fault handling strategy and supplementing of data storage resources according to the target resources includes: Obtaining hardware configuration information and a first topological relationship corresponding to the fault data; Based on the configuration information of the hardware corresponding to the fault data and the first topological relationship, a target resource is acquired from an available resource list, and data storage resources are supplemented according to the target resource.

5. The method according to claim 2, characterized in that: The fault type is a database capacity fault, and the determination based on the fault handling strategy that data storage resources need to be supplemented includes: Acquire capacity adjustment information of the data storage resource corresponding to the fault data; Determining whether the adjusted data storage resources can meet the usage conditions of the data storage resources based on the capacity adjustment information; In response to the adjusted data storage resources being able to satisfy the usage conditions of the data storage resources, it is determined that the data storage resources need to be supplemented.

6. The method according to claim 5, characterized in that The acquiring of target resources based on the fault handling strategy and supplementing of data storage resources according to the target resources includes: Acquire a second topological relationship corresponding to the fault data; Determining a corresponding resource acquisition method according to the second topological relationship type; The target resource is acquired from the available resource list by using the resource acquisition method corresponding to the second topological relationship type, and data storage resources are supplemented according to the target resource.

7. The method according to claim 6, characterized in that The step of acquiring the target resource from the available resource list by using the resource acquisition method corresponding to the second topological relationship type, and supplementing the data storage resource according to the target resource includes: In response to the second topological relationship being of a sub-library and sub-table type, determining a target computer room with the most resources in the computer room set; According to the capacity adjustment information, determining whether the resources in the target computer room meet the application conditions; In response to the resources in the target computer room satisfying the usage conditions, a target resource is determined from the resources in the target computer room, and data storage resources are supplemented according to the target resource.

8. The method according to claim 2, characterized in that: The fault type is a network fault or a downtime fault, and the need to supplement data storage resources based on the fault handling strategy includes: Performing survivability detection on the data storage resource corresponding to the fault data according to a preset period; In response to a survival check result of failure, it is determined that data storage resources need to be supplemented.

9. The method according to claim 8, characterized in that The acquiring of target resources based on the fault handling strategy and supplementing of data storage resources according to the target resources includes: Acquire configuration information and a third topological relationship corresponding to the fault data; Based on the configuration information of the hardware corresponding to the fault data and the third topological relationship, a target resource is acquired from the available resource list, and data storage resources are supplemented according to the target resource.

10. The method according to any one of claims 1 to 9, characterized in that: The supplementing of data storage resources according to the target resources includes: Data synchronization is performed based on the target resource and the data storage resource corresponding to the fault data to obtain a target data storage resource containing synchronized data.

11. The method according to claim 10, characterized in that After synchronizing the data based on the target resource and the data storage resource corresponding to the fault data to obtain the target data storage resource containing the synchronized data, the method further includes: Verify whether the data of the target data storage resource is consistent with the data storage resource corresponding to the fault data; In response to the verification result being consistent, the data storage resource corresponding to the fault data is taken offline.

12. A data storage resource fault handling device, characterized in that: include: An acquisition module is used to obtain real-time monitoring data of a data storage resource cluster; A judgment module, configured to judge a fault type based on the fault data in response to the real-time monitoring data containing the fault data; A determination module, used to determine a corresponding fault handling strategy according to the fault type; The supplement module is used to obtain target resources based on the fault handling strategy and supplement data storage resources according to the target resources.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 11.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images