Mirror source management method, device and equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose a mirror source management method, device and equipment and a computer readable storage medium. The method comprises: receiving an access request of a client for requesting to download a target software package; resolving a domain name of the access request to obtain a virtual address corresponding to the domain name, so that the client accesses a load-matched mirror source service node according to the virtual address; obtaining the target software package from a plurality of first software packages in an internal network mirror source through the load-matched mirror source service node, the plurality of first software packages in the internal network mirror source being synchronously obtained from an external network mirror source, so that the client can complete the downloading of the software package without opening an external network strategy additionally, and the data acquisition efficiency is improved; and sending the target software package to the client through a connection established by the access request and the virtual address, so that the client can download the software package through the internal network access to the mirror source, and the data acquisition efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a mirror source management method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] A software repository (Yellow dog Updater Modified, yum repository), also known as a mirror repository, is a read-hat package manager (RPM) specifically designed to resolve software package dependencies. A yum repository stores all RPM packages on a server and records the dependencies between these packages in a file. When a client uses the yum repository to install an RPM package, the yum repository first downloads the dependency file from the server. By analyzing the dependency file, it downloads all relevant RPM packages from the server at once and installs them, thus reducing the number of downloads.

[0003] In existing technologies, in public cloud scenarios, if a client needs to install an RPM package, it needs to enable external network access. Then, the yum repository file needs to be modified, setting the baseurl (uniform resource locator) to the required link. After executing `yum clean all && yum makecache` to clear the cache and create new metadata, the client can then install the necessary RPM packages and their dependencies.

[0004] Because existing technologies require an external network connection to download and install RPM packages, data acquisition efficiency is reduced. Summary of the Invention

[0005] This invention aims to provide a mirror source management method, apparatus, device, and computer-readable storage medium. The client accesses a load-matched mirror source service node based on a virtual address, and obtains the target software package from the intranet mirror source through the load-matched mirror source service node. The software package can be downloaded without the need for additional external network access policies, thus improving data acquisition efficiency.

[0006] The technical solution of this invention is implemented as follows:

[0007] In a first aspect, embodiments of the present invention provide a mirror source management method, the method comprising: receiving an access request from a client to request the download of a target software package; resolving the domain name of the access request to obtain a virtual address corresponding to the domain name, such that the client accesses an intranet mirror source through a load-matched mirror source service node based on the virtual address; obtaining the target software package from a plurality of first software packages in the intranet mirror source through the load-matched mirror source service node, wherein the plurality of first software packages in the intranet mirror source are synchronously obtained from an external mirror source, and the plurality of first software packages include the target software package; and sending the target software package to the client through a connection established between the access request and the virtual address.

[0008] Secondly, embodiments of the present invention provide a mirror source management device, the device comprising: a receiving module, configured to receive an access request from a client requesting to download a target software package; a parsing module, configured to parse the domain name of the access request to obtain a virtual address corresponding to the domain name, enabling the client to access a load-matched mirror source service node based on the virtual address; an obtaining module, configured to obtain the target software package from multiple first software packages of an intranet mirror source through the load-matched mirror source service node, wherein the multiple first software packages in the intranet mirror source are synchronously obtained from an external network mirror source, and the multiple first software packages include the target software package; and a sending module, configured to send the target software package to the client through a connection established between the access request and the virtual address.

[0009] Thirdly, embodiments of the present invention provide a mirror source management device, the device including a memory for storing executable instructions and a processor for implementing the above-mentioned mirror source management method when executing the executable instructions stored in the memory.

[0010] Fourthly, embodiments of the present invention provide a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, implement the above-described image source management method.

[0011] This invention provides a mirror source management method, apparatus, device, and computer-readable storage medium. According to the solution provided by this invention, an access request from a client to download a target software package is received; the domain name of the access request is resolved to obtain the virtual address corresponding to the domain name, enabling the client to access a load-matched mirror source service node based on the virtual address; the target software package is obtained from multiple first software packages in an intranet mirror source through the load-matched mirror source service node, where the multiple first software packages include the target software package; the matching of mirror source service nodes is performed through load balancing, improving network resource utilization. Specifically, the multiple first software packages in the intranet mirror source are synchronously obtained from an external network mirror source, allowing the client to complete the software package download without additional external network access policies, reducing network congestion and improving data acquisition efficiency; the target software package is sent to the client through the connection established between the access request and the virtual address, realizing the process of the client accessing the mirror source to download the software package via the intranet, further improving data acquisition efficiency. Attached Figure Description

[0012] Figure 1 An exemplary architecture diagram of a yum repository is provided for an embodiment of the present invention;

[0013] Figure 2 A flowchart of optional steps for a mirror source management method provided in an embodiment of the present invention;

[0014] Figure 3 An exemplary architecture diagram for containerized management is provided for embodiments of the present invention;

[0015] Figure 4 A flowchart of optional steps for another mirror source management method provided in an embodiment of the present invention;

[0016] Figure 5 This is a schematic diagram of the structure of a mirror source management device provided in an embodiment of the present invention;

[0017] Figure 6 This is a schematic diagram of the structure of a mirror source management device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that some embodiments described herein are merely illustrative of the technical solutions of the present invention and are not intended to limit the scope of the present invention.

[0019] To better understand the mirror source management method provided in the embodiments of the present invention, the relevant technologies will be explained before introducing the technical solutions of the embodiments of the present invention.

[0020] In related technologies, in public cloud scenarios, yum repositories are improved RPM package managers that solve the package dependency problem faced by RPM packages. Linux systems provide yum repositories for downloading, installing, upgrading, and uninstalling software packages. A yum repository can be built in three ways: (1) Local yum repository: A yum repository is built on a host based on a collection of packages in a local directory. The host acts as both the client and the server, usually in the form of an ISO mount. (2) Network yum repository: A network yum repository is built based on software repositories on a File Transfer Protocol (FTP) server. Similar to a Hyper Text Transfer Protocol (HTTP) server, it is generally used for remote installation scenarios in PrebooteXecution Environment (PXE). (3) Network yum repository: A network yum repository is built based on software repositories on an HTTP server. The architecture diagram of a yum repository is shown below. Figure 1 As shown, Figure 1 This is an exemplary architecture diagram of a yum repository provided in an embodiment of the present invention. Figure 1 The architecture diagram of the yum repository includes a yum repository server and a yum repository client on a Linux file server. Data is transferred between the yum repository client and the yum repository server via an interface (port). The `createrepo` command is used to create yum repositories, indexing numerous RPM packages stored in a specific local location, describing the dependencies of each RPM package, and forming metadata. The yum repository server includes multiple RPM packages. The `createrepo` command generates a metadata package data source file, which includes a list of RPM packages and their dependencies. This allows the RPM package management tool on the yum repository client to synchronize the metadata package data source file from the yum repository server, then download the RPM packages to the local machine based on the metadata, and automatically delete the RPM packages after installation.

[0021] In related technologies, in the HTTP server (i.e. Figure 1 All packages (i.e., ...) on the yum repository server Figure 1After placing the RPM packages, create a repo and start the HTTP daemon (http-daemon, httpd) service. Configure the corresponding path on the client for access. If the client needs to install RPM packages, it needs to enable external network access. Next, modify the ` / etc / yum.repo.d / *.repo` file, setting the `baseurl` to the link to be accessed. Execute `yum cleanall && yum makecache` to clear the cache and create metadata. The client can then successfully install the required packages and their dependencies.

[0022] While the above solution enables the normal download and installation of software packages by connecting to the external network, users need to pay extra to enable the external network policy, which increases user costs. In addition, a large number of accesses to a single mirror source will increase the server load on the server side, resulting in increased network latency and reduced data acquisition efficiency.

[0023] This invention provides a mirror source management method, such as... Figure 2 As shown, Figure 2 This is an optional flowchart of a mirror source management method provided in an embodiment of the present invention. The mirror source management method includes the following steps:

[0024] S201. Receive the client's access request to download the target software package.

[0025] In this embodiment of the invention, the executing entity of the mirror source management method is a mirror source management device, which is a server. For ease of description, the mirror source management device is referred to as a server. The server receives access requests sent by clients. These access requests can be HTTP requests, which are sent to the server in the form of request messages. The request messages include, but are not limited to, request methods, URLs, protocol versions, request headers, and request data. In this embodiment of the invention, the access request is used to request the server to download the target software package.

[0026] S202. Resolve the domain name of the access request to obtain the virtual address corresponding to the domain name, so that the client can access the internal network mirror source through the mirror source service node matched by the load based on the virtual address.

[0027] In this embodiment of the invention, the domain name of the HTTP request is resolved by a Domain Name Server (DNS) to obtain the virtual IP (VIP) corresponding to the domain name. The virtual IP is used to provide users with an accessible IP address. The DNS server is responsible for converting domain names to their corresponding IP addresses. After obtaining the virtual IP through domain name resolution, the client can access the internal network mirror source based on the virtual IP. When the client accesses the internal network mirror source using the virtual IP, it needs to find the access path to the internal network mirror source. In this embodiment, multiple mirror source service nodes represent different access paths. The server includes multiple mirror source service nodes, and the internal network mirror source can be accessed through any one of these nodes. This avoids the reduced data acquisition efficiency caused by multiple clients simultaneously accessing the internal network mirror source through a single path.

[0028] In this embodiment of the invention, the domain name of the access request is resolved to obtain the virtual address corresponding to the domain name. The client accesses the intranet mirror source through the mirror source service node of load matching. This load matching method improves the utilization of network resources.

[0029] S203. Obtain the target software package from multiple first software packages in the intranet mirror source through the load-matched mirror source service node. The multiple first software packages in the intranet mirror source are synchronously obtained from the external mirror source, and the multiple first software packages include the target software package.

[0030] In this embodiment of the invention, "first" and "second" are merely distinguishing names and do not represent an order relationship. They should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated, such as a first software package and a second software package. This is illustrated using an example of an intranet mirror source containing multiple software packages. Since these multiple software packages are synchronously obtained from an external network mirror source, and the intranet mirror source includes multiple software packages and descriptions of their dependencies, the client can obtain the target software package and its dependencies by accessing the intranet mirror source, thereby enabling the subsequent download and installation of the target software package. The client does not need to access the external network mirror source. Therefore, the client does not need to implement additional external network access policies to complete the software package download, reducing network congestion and improving data acquisition efficiency.

[0031] It should be noted that the mirror source service node for load matching can be the mirror source service node with the lowest load, or a mirror source service node with a load less than a first preset threshold, or a mirror source service node whose physical distance from the client is less than a preset distance and whose load is less than a second preset threshold. The preset threshold and preset distance can be set by those skilled in the art according to actual conditions. For example, they can be determined through analysis of a large number of load matching thresholds and distances, and this embodiment of the invention does not impose any limitations on this.

[0032] In this embodiment of the invention, the access path for the client to access the intranet mirror source is the mirror source service node with load matching, which avoids the problem of resource waste caused by unbalanced load and improves the utilization rate of network resources.

[0033] S204. The target software package is sent to the client through the connection established by the access request and the virtual address.

[0034] In this embodiment of the invention, after the server obtains the target software package from the intranet mirror source through the load-matched mirror source service node, it also needs to send the target software package to the client through the connection established by the access request and the virtual address.

[0035] It should be noted that in the embodiments of the present invention, the external network can be referred to as the public network, the internal network can be referred to as the private network or local area network, the client can be referred to as the cloud virtual machine, virtual machine, client, or cloud host, the software package can be referred to as the RPM package or installation package, and the task can be referred to as the business. The embodiments of the present invention do not limit these.

[0036] According to the solution provided in this embodiment of the invention, an access request from a client to download a target software package is received; the domain name of the access request is resolved to obtain the virtual address corresponding to the domain name, enabling the client to access a load-matched mirror source service node based on the virtual address; through the load-matched mirror source service node, the target software package is obtained from multiple first software packages in the intranet mirror source, where the multiple first software packages include the target software package. Matching of mirror source service nodes through load balancing improves network resource utilization. Specifically, the multiple first software packages in the intranet mirror source are synchronously obtained from an external network mirror source, allowing the client to complete the software package download without additional external network access policies, reducing network congestion. By establishing a connection between the access request and the virtual address, the target software package is sent to the client, realizing the process of the client accessing the mirror source to download the software package through the intranet, thus improving data acquisition efficiency.

[0037] In some embodiments, multiple first software packages in the intranet mirror source are stored in a network file system server. Before S203 above, the mirror source management method further includes a step of transferring and storing software packages: transferring multiple second software packages in the external network mirror source to the network file system server corresponding to the intranet mirror source synchronously, wherein the network file system server is connected to the external network mirror source through network permissions.

[0038] In this embodiment of the invention, before the client accesses the intranet mirror source via the intranet, some preparatory work is required to synchronize multiple second software packages from the external network mirror source to the intranet mirror source. In this embodiment, multiple first software packages from the intranet mirror source are stored in a Network File System (NFS) storage server. NFS allows remote clients to access files over the network in a manner similar to a local file system; it can be understood that NFS makes accessing files elsewhere on the network feel like using one's own computer.

[0039] It should be noted that the multiple second software packages in the external network mirror source are the same as the multiple first software packages in the internal network mirror source. The first and second are only used to distinguish whether the software package is located in the external network mirror source or the internal network mirror source. It is also possible to describe them as multiple software packages. This embodiment of the invention does not limit this.

[0040] In this embodiment of the invention, the NFS storage server can have a large storage space, for example, 10 terabytes (TB), and the NFS storage server has network permissions to connect to the external network mirror source in order to synchronize multiple second software packages and their required dependencies in the external network mirror source in real time.

[0041] In this embodiment of the invention, the NFS storage server connects to the external network mirror source through network permissions; multiple second software packages in the external network mirror source are synchronously transferred to the NFS storage server corresponding to the internal network mirror source, so that users can access the internal network mirror source through the internal network and obtain the target software package from the NFS storage server, thereby improving data acquisition efficiency.

[0042] In some embodiments, the intranet mirror source is built according to the following rules: deploy at least one intranet mirror source within a preset range.

[0043] In this embodiment of the invention, the intranet mirror source can be built in two ways. In the first way, one or two central mirror sources are built across the entire cloud resource pool, with multiple regions sharing one mirror source, theoretically reducing operational pressure. However, this approach has the following problems: high network latency occurs when accessing the mirror source across regions; and if multiple regions share one mirror source, and the mirror source fails (e.g., is compromised), the server becomes unavailable, affecting a large area. Therefore, the intranet mirror source built using the first method suffers from high latency, low availability, and high risk. In the second way, one mirror source is built for each region, and multiple availability zones (AZs) within that region share one mirror source. Even if the mirror source in one region fails, the mirror sources in other regions can still operate normally. Therefore, the mirror source built using the second method has low network latency, high security, and high availability. After the actual mirror source deployment is completed, synchronization and other operations are automated, requiring no operational intervention.

[0044] In this embodiment of the invention, after the mirror source is deployed, subsequent synchronization and update operations are all completed by automated scripts and scheduled tasks, requiring minimal operational intervention. Considering availability, security, and risk, as well as the actual situation of subsequent operations, at least one internal network mirror source is deployed within a preset scope. For example, one internal network mirror source is deployed in one region. Because in the first setup method described above, the entire cloud resource pool uses only one or two central mirror sources, and these span physical regions, latency cannot be guaranteed. If any problems occur that cause the mirror source server to become unavailable, the geographical impact is very large. Therefore, this embodiment of the invention chooses the second setup method described above, deploying at least one internal network mirror source within a preset scope to avoid large-scale failures, resulting in low network latency, high security, and high availability.

[0045] It should be noted that the preset range in the embodiments of the present invention is a physical geographical range. The preset range can be appropriately set by those skilled in the art according to actual needs, and the embodiments of the present invention do not limit this.

[0046] In some embodiments, prior to S203 above, the mirror source management method further includes the step of determining a load-matching mirror source service node: determining a load-matching mirror source service node through a load balancing component based on a dynamic scheduling algorithm.

[0047] In this embodiment of the invention, HTTP requests typically use port 80 by default. The mirror source service node provides port 80 for connecting to the load balancer (LB) component to provide HTTP services to the client. The load balancer component provides virtual IP records to the module used for domain name resolution, and returns the virtual IP corresponding to the domain name to the client when the client requests access.

[0048] In this embodiment of the invention, the load balancing component can be any form of load balancer, and there are no restrictions on the composition of the load balancing component used, as long as it can select a mirror source service node with a smaller load for the access path of the virtual IP. This includes, but is not limited to, Linux Virtual Server (LVS), DPDK-LVS (dpvs), DNS component Bind (Berkeley Internet Name Domain), Nginx (software), F5 (hardware), keepalived, apache (software), and any combination of the above software and hardware. In this embodiment of the invention, the dynamic scheduling algorithm includes, but is not limited to: round-robin, random, least-connection, weight, ip_hash, fair, and url_hash.

[0049] In this embodiment of the invention, the load balancing component determines the load-matching mirror source service node according to the dynamic scheduling algorithm, so as to access the intranet mirror source through the load-matching mirror source service node and obtain the target software package, thereby improving the utilization of network resources.

[0050] In some embodiments, the software package synchronization transfer task and the global load balancing task are implemented by containerizing the task. The image source management method further includes the following steps: deploying a first container based on a container cluster to implement the synchronization transfer task, so as to decompose the synchronization transfer task into the container cluster; or, deploying a second container based on a container cluster to implement the global load balancing task, so as to decompose the global load balancing task into the container cluster.

[0051] In this embodiment of the invention, the synchronous transfer of multiple second software packages from an external network mirror source to an NFS storage server constitutes one task, and the determination of the mirror source service node with load matching by the load balancing component based on a dynamic scheduling algorithm constitutes another task. When the server used to execute these tasks, it consumes significant CPU and memory. Without an isolation mechanism, server instability can occur, affecting the operation of other tasks. Therefore, in this embodiment, the synchronous transfer task and the global load balancing task are decomposed into a container cluster. Multiple containers are deployed within the container cluster, each handling different tasks. This process of integrating tasks into containers and running them is called "containerizing." Containers are designed for tasks, and their lifecycle can be the same as the task's lifecycle. Containers simplify the construction, deployment, and execution of tasks.

[0052] In this embodiment of the invention, the containerization task is illustrated through the following two examples. Example 1: In the first container, the NFS storage server connects to the external network mirror source via network permissions; multiple second software packages from the external network mirror source are synchronously transferred to the NFS storage server corresponding to the internal network mirror source, thus realizing the synchronous transfer task. Example 2: In the second container, a load balancing component determines the mirror source service node with load matching based on a dynamic scheduling algorithm, thus realizing a global load balancing task; wherein, the load balancing component is connected to port 30080 provided by the second container to provide HTTP service, and the client is configured with a domain name to use the internal network mirror source.

[0053] It should be noted that containers can be created for more tasks in subsequent task processing, and this embodiment of the invention does not limit this.

[0054] For example, this section describes how a load balancing component, consisting of an Nginx node and an NFS storage server, synchronizes an external network mirror source using rsync (a data mirroring and backup tool for Linux systems). The servers used on the server side are containerized, and the synchronization-to-storage task and the global load balancing task are decomposed into a Kubernetes (k8s) cluster. For the synchronization-to-storage task, a mirror server and a mirror sync scheduled task are deployed in the k8s cluster. These tasks execute the synchronization mirroring work every preset time (e.g., every hour), synchronizing the software packages from the external network mirror source to the NFS storage server within the cloud resource pool. The pod's port 30080 is exposed to the load balancing component to provide HTTP service. Clients within the cloud resource pool can use the internal network mirror source normally by configuring a domain name.

[0055] In embodiments of the present invention, a containerized software architecture is as follows: Figure 3 As shown, Figure 3 This is an exemplary architecture diagram for containerized management provided in an embodiment of the present invention. Taking a Kubernetes (k8s) cluster as an example, Nginx and rsync services are deployed in the k8s cluster. Both Nginx and rsync services are scheduled and managed by k8s. Typically, Nginx and rsync services are deployed on four nodes each. Figure 3 The cluster is represented by nodes 1, 2, 3, and 4. The Nginx and rsync services on these four nodes are mounted via NFS, serving as external shared NFS storage servers. Each node includes an Nginx Kubernetes pod and several rsync tasks. The Nginx service acts as a proxy for yum / apt repositories, providing proxy and HTTP services, allowing access to packages in the internal network mirror repository. The rsync service runs as a scheduled task within the Kubernetes cluster, synchronizing multiple secondary packages from the external network mirror repository. Figure 3 The system also deploys DNS services on bare metal nodes 1 and 2, exposing VIP1 to provide DNS lookup services, such as DNS lookupmirrors.ecloud.com. On bare metal nodes 3 and 4, the system deploys DPVS, a high-performance Layer 4 load balancer based on DPDK, acting as a Layer 4 traffic proxy for the Nginx service. Nodes 3 and 4 expose VIP2 to provide high-performance mirror source services, for example, via HTTP GET http: / / mirrors.ecloud.com / centos / 7 / os / x86_64. All four Nginx services operate on port 30080, and the DPVS instances on nodes 3 and 4 can connect to port 30080 via network traffic.

[0056] In this embodiment of the invention, tasks are containerized. By setting up separate tasks (mirror server and mirrorsync), multiple software packages can be synchronized from an external network mirror source. By setting up a separate Nginx service, HTTP services can be provided to the load balancing component. Implementing different tasks within containers ensures high availability and high performance, guarantees that clients can quickly and securely access the internal network mirror source, and improves data acquisition efficiency.

[0057] In some embodiments, the load balancing component includes a front-end load balancing node and a back-end load balancing node. The front-end load balancing node provides a directory for storing virtual address paths, enabling clients to access the corresponding intranet mirror source based on the virtual address. The front-end load balancing node and the back-end load balancing node are connected via TCP, and the back-end load balancing node is connected to a preset port provided by the mirror source service node. The above S203 can be implemented in the following way: The front-end load balancing node advertises the virtual address to the service switch based on a routing protocol component; the back-end load balancing node parses the content of the access request to obtain the parsing result; the service switch selects a mirror source service node with load matching based on a scheduling algorithm and the parsing result.

[0058] In this embodiment of the invention, an example is taken where the front-end load balancing node is a front-end SLB node and the back-end load balancing node is a back-end Nginx node. The load balancing component includes a front-end Server Load Balancer (SLB) node and a back-end Nginx node. The front-end SLB node and the back-end Nginx node cooperate to determine the mirror source service node for load matching. For example, the front-end SLB node can be LVS supporting Layer 4 switching technology, and the back-end Nginx node can be Nginx supporting Layer 7 switching technology. Since the back-end Nginx node provides services externally in the form of an Nginx cluster, an additional load balancer, namely the front-end SLB node, is added on top of the Nginx cluster. The front-end SLB node has one external entry point, serving as the system's sole entry point. The Nginx cluster possesses scalability and high availability. Through horizontal scaling, the number of Nginx nodes is increased, thereby improving the bandwidth of the Nginx cluster, increasing throughput performance, enhancing network data processing capabilities, and improving network resource utilization.

[0059] In this embodiment of the invention, HTTP requests use port 80 by default. The mirror source service node provides port 80, and the backend Nginx node connects to port 80 provided by the mirror source service node.

[0060] In this embodiment of the invention, the load balancing component adopts a Layer 4 load balancing + Layer 7 load balancing scheme. The front-end SLB nodes advertise their virtual IPs to the service switches via the Quagga / FRR routing protocol (BGP) component. The Nginx nodes parse the content of HTTP requests and obtain the parsing results. The service switches select a mirror source service node with matching load based on the scheduling algorithm and the parsing results. In this embodiment, load balancing of the SLB cluster is achieved through the equal-cost multipath routing (ECMP) mechanism of the service switches. ECMP, also known as equal-cost multipath, means that there are multiple paths with the same cost to the same mirror source service node. When the server supports equal-cost routing, requests to the mirror source service node can be distributed across different paths, achieving network load balancing. Furthermore, if some paths fail, other paths can take over the forwarding process, achieving route redundancy backup.

[0061] In this embodiment of the invention, when a user in the cloud resource pool accesses the intranet mirror source through the intranet, the business switch in the load balancing component implements global load balancing according to the dynamic scheduling algorithm. When a server / virtual machine where a component of a backend Nginx node is located goes down, the intranet mirror source can still provide HTTP services to the user, ensuring the high availability of the intranet mirror source service.

[0062] In some embodiments, the mirror source management method further includes the following step: expanding the capacity of the front-end load nodes through the service switch, thereby increasing the number of front-end load nodes.

[0063] For example, taking LVS as the front-end load balancer, all HTTP requests first reach LVS as the front-end node of the load balancing component. If LVS goes down, the back-end server will be unable to provide services. Therefore, to avoid this situation, it is necessary to scale up LVS by establishing multiple LVSs that collaborate to improve server high availability. LVS can be in dual-master or master-slave mode. In master-master mode, both LVSs perform load balancing using DNS round-robin. In master-slave mode, the master LVS performs the work; if the master LVS goes down, the slave LVS performs load balancing; if the master LVS recovers, it performs load balancing again. For example, high-availability monitoring programs run on both the master and slave LVSs, monitoring each other's operational status by sending messages such as "I am alive". When the backup LVS fails to receive such a message within a certain period of time, it takes over the service IP of the primary LVS and continues to provide load balancing services; when the backup LVS receives the "I am alive" message from the primary LVS again, it releases the service IP address, and the primary LVS starts providing load balancing services again.

[0064] In this embodiment of the invention, the front-end load balancing nodes can also be expanded through the service switch to increase the number of front-end load balancing nodes, thereby improving bandwidth, increasing throughput performance, and enhancing network data processing capabilities. Furthermore, the front-end SLB nodes of the load balancing component have a health check mechanism. When an anomaly is detected in the service of a certain SLB node, the routing protocol (BGP) connection will be actively disconnected, and an available SLB node will be used to provide load balancing services, thus improving the high availability of the load balancing component.

[0065] In some embodiments, the number of backend load nodes is multiple, and the mirror source management method further includes the following steps: checking multiple backend load nodes through a preset health check mechanism to identify abnormal nodes; when an abnormal node is found, disconnecting the abnormal node.

[0066] In this embodiment of the invention, the load balancing component includes multiple backend Nginx nodes, which are checked by a preset health check mechanism. When an abnormal Nginx node is detected, the abnormal Nginx node is disconnected to avoid access path errors, ensure the normal operation of the business, and improve the high availability of the load balancing component.

[0067] It should be noted that the preset health check mechanism in the embodiments of the present invention can be set by those skilled in the art according to actual needs, as long as it can detect abnormal Nginx nodes, and the embodiments of the present invention do not impose any restrictions on this.

[0068] The following will describe an exemplary application of the embodiments of the present invention in a practical application scenario.

[0069] In this embodiment of the invention, the example is a load balancer with a front-end node of LVS (Layer 4 load balancer), a back-end load balancer of Nginx (Layer 7 load balancer), a yum repository as the image source, a virtual machine within a cloud resource pool as the client, and an HTTP request as the access request. To facilitate understanding of the functional architecture of this solution, the process of a client accessing an internal yum repository is described using three functional modules: an NFS storage service module, a yum repository service module, and a DNS resolution module, along with the load balancer component. Specifically, the NFS storage service module is used to transfer software packages from the external yum repository to an NFS storage server in the cloud resource pool. The yum repository service module provides an HTTP daemon (http-daemon, httpd) service, which mounts the directory corresponding to the storage path of the software packages on the NFS storage server and provides port 80 access permissions for HTTP requests. The load balancing component is divided into a front-end and a back-end. The front-end uses the Quagga / FRR BGP component to connect to the service switch and provides a virtual IP to the DNS resolution module. The back-end uses Nginx nodes to distribute traffic to the less loaded mirror source service nodes. The load balancing component connects to port 80 provided by the yum source service module. The DNS resolution module records the virtual IP provided by the load balancing component and returns the virtual IP corresponding to the domain name of the HTTP request to the virtual machine when a virtual machine in the cloud resource pool makes an HTTP request.

[0070] For example, the data access process of an HTTP request is as follows: Figure 4 As shown, Figure 4 A flowchart illustrating optional steps of another mirror source management method provided in an embodiment of the present invention. Figure 4 In this example, VM users represents the client, DNS represents the DNS server, VIP represents the virtual IP, yum source represents the image source, and NFS Cloud represents the NFS storage server in the cloud resource pool. Figure 4 The load balancing component consists of two LVSs that implement master-master mode through routing protocol components (BGP or OSPF), and global load balancing of the BC-SLB cluster corresponding to the load balancing component is achieved through service switch ECMP. Figure 4The storage space of the disk array (Redundant Arrays of Independent Disks, RAID) corresponding to NFS Cloud is greater than 10TB. NFS Cloud can be understood as an internal network yum repository. NFS Cloud synchronizes multiple software packages from the external network yum repository through the rsync service (a data mirroring and backup tool under the Linux system) and stores multiple software packages in RAID, so that subsequent clients can access the internal network yum repository through the load-matched yum repository service node to obtain the target software package. Figure 4 This document explains the mirror source management method through the interaction process between the client and the server, including the following steps:

[0071] S401. The client accesses the DNS server corresponding to the yum repository.

[0072] When a client initiates an HTTP request in the cloud resource pool to download a software package from the internal yum repository, the DNS resolution module resolves the domain name of the HTTP request. For example, the DNS resolution module resolves the domain name of the HTTP request to a virtual IP address through a DNS server, where the virtual IP address is provided by the load balancing component.

[0073] In this embodiment of the invention, the domain name of the HTTP request is resolved using a DNS server to obtain a virtual IP corresponding to the BC-SLB cluster. A nameserver needs to be configured in resolv.conf, pointing to the virtual IP of the DNS server. resolv.conf is the configuration file used by the resolver library. Whenever a program needs to access other servers on the network via a domain name, it needs to use this library to convert the domain name into the corresponding IP address before access can be made. The virtual IP of the DNS server uses BGP Anycast. BGP Anycast technology means that different servers use the same IP address, that is, the virtual IP is the same for each region. If the BGP Anycast scheme is used, the virtual IP corresponding to each region is the same, and a virtual IP is obtained based on the DNS server resolving the domain name of the HTTP request.

[0074] S402. The client obtains the virtual IP address obtained by the DNS resolution module from the domain name resolution.

[0075] This virtual IP can be understood as the IP address of the yum repository.

[0076] S403: The client accesses the yum repository based on the virtual IP.

[0077] The client accesses the server corresponding to the yum repository based on the virtual IP. The load balancing component forwards the access to the appropriate yum repository service node through LVS and Nginx proxies. The yum repository service node provides the directory where the package is located to the access request link. The required package is then sent to the virtual machine where the client is located via an HTTP request to the virtual IP established with the virtual IP.

[0078] In an embodiment of the present invention, Figure 4 The BC-SLB cluster corresponding to the load balancing component adopts a 4+7 scheme, namely LVS Layer 4 load balancing + Nginx Layer 7 load balancing. The LVS Layer 4 load balancer advertises virtual IPs to the uplink service switch through the Quagga / FRR BGP component. The load balancing of the BC-SLB cluster is achieved through the service switch's equal-cost routing + ECMP mechanism. Simultaneously, the service switch can also expand the BC-SLB, thereby increasing the number of LVS Layer 4 load balancers. Furthermore, the BC-SLB cluster has a health check mechanism; if abnormal service is detected on the local server, the BGP connection will be proactively disconnected.

[0079] In this embodiment of the invention, LVS Layer 4 load balancer acts as the front end of the BC-SLB cluster. Client HTTP requests are input to LVS, where load balancing forwards them to the backend services. Multiple Nginx nodes in the backend services collectively perform Layer 7 load balancing and output load-matched yum source service nodes to the client. Nginx Layer 7 load balancer acts as the back end of the BC-SLB cluster. It selects the yum source service node with the lower load using a scheduling algorithm (WLC algorithm), improving network resource utilization. Furthermore, the BC-SLB cluster has a health check mechanism for Nginx nodes. When an Nginx node fails, it shuts down the failed node to ensure normal business operation. The backend Nginx Layer 7 load balancer establishes a TCP connection with the frontend LVS Layer 4 load balancer and performs deep parsing of the client's request content to obtain the parsing results. Based on the specified scheduling algorithm and the parsing results, a load-matched yum source service node is selected. This load-matched yum source service node can be understood as the backend real server (RS). The BC-SLB cluster can use FULLNAT forwarding mode to change the real client address (clientIP) to the BC-SLB cluster's internal network address (Local IP). Therefore, the TOA (Address of TCP Option) module needs to be installed on the Nginx Layer 7 load balancer to record the real client address in Nginx logs. The TOA module ensures that the backend real servers see the real client address, rather than the LVS virtual IP.

[0080] It should be noted that, Figure 4The example uses four Nginx Layer 7 load balancers, but this embodiment of the invention does not limit the number of Nginx Layer 7 load balancers.

[0081] The mirror source management method proposed in this invention is applied in a public cloud environment. It provides clients with a fast, convenient, and configuration-free internal yum repository containing multiple software packages. The load balancing component selects a load-matched yum repository service node based on the load of different servers, enabling virtual machines within the cloud resource pool to access the internal yum repository through the load-matched yum repository service node, thereby improving network resource utilization. Virtual machines within the cloud resource pool do not need to additionally enable public network policies to use the internal yum repository normally, enabling software package installation, reducing network congestion issues, and improving data acquisition efficiency.

[0082] To implement the mirror source management method of this invention, this invention also provides a mirror source management device, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of a mirror source management device provided in an embodiment of the present invention. The mirror source management device 50 includes: a receiving module 501, used to receive an access request from a client requesting to download a target software package; a parsing module 502, used to parse the domain name of the access request to obtain a virtual address corresponding to the domain name, so that the client accesses a load-matched mirror source service node according to the virtual address; an obtaining module 503, used to obtain the target software package from multiple first software packages of an intranet mirror source through the load-matched mirror source service node, wherein the multiple first software packages in the intranet mirror source are synchronously obtained from an external network mirror source, and the multiple first software packages include the target software package; and a sending module 504, used to send the target software package to the client through a connection established between the access request and the virtual address.

[0083] In some embodiments, multiple first software packages in the intranet mirror source are stored on a network file system server;

[0084] The mirror source management device 50 further includes an NFS storage service module; the NFS storage service module is used to synchronously transfer multiple second software packages in the external network mirror source to the network file system server corresponding to the internal network mirror source, wherein the network file system server is connected to the external network mirror source through network permissions.

[0085] In some embodiments, the intranet mirror source is established according to the following rules: at least one intranet mirror source is deployed within a preset range.

[0086] In some embodiments, the mirror source management device 50 further includes a matching module, which is used to determine the mirror source service node for load matching according to a dynamic scheduling algorithm through a load balancing component.

[0087] In some embodiments, the image source management device 50 further includes a containerization module, which is used to deploy a first container based on a container cluster to implement a synchronous transfer storage task, so as to decompose the synchronous transfer storage task into the container cluster; or, based on a container cluster, to deploy a second container based on a container cluster to implement a global load balancing task, so as to decompose the global load balancing task into the container cluster.

[0088] In some embodiments, the load balancing component includes a front-end load node and a back-end load node. The front-end load node is used to provide a directory for the virtual address storage path, so that the client can access the corresponding intranet mirror source according to the virtual address. The front-end load node and the back-end load node are connected via TCP, and the back-end load node is connected to a preset port provided by the mirror source service node.

[0089] The matching module is also used to announce the virtual address to the service switch through the front-end load node based on the routing protocol component; parse the content of the access request based on the back-end load node to obtain the parsing result; and select the mirror source service node for load matching through the service switch according to the scheduling algorithm and the parsing result.

[0090] In some embodiments, the mirror source management device 50 further includes a capacity expansion module, which is used to expand the capacity of the front-end load nodes through the service switch, thereby increasing the number of the front-end load nodes.

[0091] In some embodiments, the number of backend load nodes is multiple;

[0092] The mirror source management device 50 also includes an inspection module, which is used to inspect multiple backend load nodes through a preset health check mechanism to identify abnormal nodes; when an abnormal Nginx node is found, the abnormal Nginx node is disconnected.

[0093] It should be noted that the mirror source management device provided in the above embodiments is only illustrated by the division of the above-described program modules when performing mirror source management. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the processing described above. Furthermore, the mirror source management device and the mirror source management method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process and beneficial effects are detailed in the method embodiments, which will not be repeated here. For technical details not disclosed in this device embodiment, please refer to the description of the method embodiments of this invention for understanding.

[0094] In an embodiment of the present invention, Figure 6 This is a schematic diagram of the composition structure of the mirror source management device proposed in an embodiment of the present invention, as shown below. Figure 6 As shown, the device 60 proposed in this embodiment of the invention may further include a processor 601 and a memory 602 storing executable instructions of the processor 601. In some embodiments, the mirror source management device 60 may further include a communication interface 603 and a bus 604 for connecting the processor 601, the memory 602 and the communication interface 603.

[0095] In this embodiment of the invention, the processor 601 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and this embodiment of the invention does not specifically limit the specific device used.

[0096] In this embodiment of the invention, bus 604 is used to connect communication interface 603, processor 601, and memory 602, as well as the mutual communication between these devices.

[0097] In this embodiment of the invention, the processor 601 is configured to receive an access request from a client requesting to download a target software package; resolve the domain name of the access request to obtain a virtual address corresponding to the domain name, enabling the client to access an intranet mirror source through a load-matched mirror source service node based on the virtual address; obtain the target software package from multiple first software packages in the intranet mirror source through the load-matched mirror source service node, wherein the multiple first software packages in the intranet mirror source are synchronously obtained from an external mirror source, and the multiple first software packages include the target software package; and send the target software package to the client through the connection established between the access request and the virtual address.

[0098] The memory 602 in the mirror source management device 60 can be connected to the processor 601. The memory 602 is used to store executable program code and data, including computer operation instructions. The memory 602 may contain high-speed RAM and may also include non-volatile memory, such as at least two disk drives. In practical applications, the memory 602 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor 601.

[0099] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0100] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] This invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the mirror source management method as described in any of the above embodiments.

[0102] For example, the program instructions corresponding to a mirror source management method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to a mirror source management method in the storage media are read or executed by an electronic device, the mirror source management method as described in any of the above embodiments can be implemented.

[0103] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0104] This invention is described with reference to schematic and / or block diagrams illustrating the implementation of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, as well as combinations of blocks in the schematic and / or block diagrams. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0105] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0106] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0107] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A method for managing mirror sources, characterized in that, The method includes: Receives access requests from clients requesting the download of the target software package; The domain name of the access request is resolved to obtain the virtual address corresponding to the domain name, so that the client can access the intranet mirror source through the mirror source service node with load matching according to the virtual address; The target software package is obtained from multiple first software packages in the intranet mirror source through the load-matched mirror source service node. The multiple first software packages in the intranet mirror source are synchronously obtained from the external mirror source, and the multiple first software packages include the target software package. The target software package is sent to the client through the connection established between the access request and the virtual address; Before obtaining the target software package from the intranet mirror source through the load-matched mirror source service node, the method further includes: The load balancing component determines the mirror source service node that matches the load based on a dynamic scheduling algorithm. The load balancing component includes a front-end load node and a back-end load node. The front-end load node provides a directory for storing the virtual address, enabling the client to access the corresponding intranet mirror source based on the virtual address. The front-end load node and the back-end load node are connected via TCP, and the back-end load node is connected to a preset port provided by the mirror source service node. The step of determining the load-matching mirror source service node by the load balancing component according to the dynamic scheduling algorithm includes: the front-end load node advertising the virtual address to the service switch based on the routing protocol component; parsing the content of the access request based on the back-end load node to obtain the parsing result; and the service switch selecting the load-matching mirror source service node according to the scheduling algorithm and the parsing result.

2. The method according to claim 1, characterized in that, The internal network mirror source contains multiple first software packages stored on a network file system server; Before obtaining the target software package from the intranet mirror source through the load-matched mirror source service node, the method further includes: Multiple second software packages from the external network mirror source are synchronously transferred to the network file system server corresponding to the internal network mirror source, wherein the network file system server is connected to the external network mirror source through network permissions.

3. The method according to claim 1 or 2, characterized in that, The internal network mirror source is set up according to the following rules: at least one internal network mirror source is deployed within a preset range.

4. The method according to claim 1 or 2, characterized in that, The method further includes: The first container is deployed based on a container cluster to implement the synchronous transfer and storage task, thereby decomposing the synchronous transfer and storage task into the container cluster. Alternatively, a second container can be deployed on top of the container cluster to implement the global load balancing task, thereby decomposing the global load balancing task into the container cluster.

5. The method according to claim 1, characterized in that, The method further includes: The number of front-end load balancing nodes can be increased by expanding the capacity of the front-end load balancing nodes through the service switch.

6. The method according to claim 1, characterized in that, The number of backend load nodes is multiple, and the method further includes: A preset health check mechanism is used to check multiple backend load nodes to identify abnormal nodes; When the abnormal node is detected, disconnect the abnormal node.

7. A mirror source management device, characterized in that, The device includes: The receiving module is used to receive access requests from clients requesting the download of the target software package; The resolution module is used to resolve the domain name of the access request to obtain the virtual address corresponding to the domain name, so that the client can access the load-matched mirror source service node according to the virtual address; The acquisition module is used to acquire a target software package from multiple first software packages in an intranet mirror source through the load-matched mirror source service node, wherein the multiple first software packages in the intranet mirror source are synchronously acquired from an external mirror source, and the multiple first software packages include the target software package; The sending module is used to send the target software package to the client through the connection established between the access request and the virtual address; The matching module is used to determine the mirror source service node for load matching based on a dynamic scheduling algorithm using a load balancing component; the matching module is also used to advertise the virtual address to the service switch through a front-end load node based on a routing protocol component; to parse the content of the access request based on a back-end load node to obtain a parsing result; and to select the mirror source service node for load matching by the service switch based on a scheduling algorithm and the parsing result. The load balancing component includes a front-end load node and a back-end load node. The front-end load node provides a directory for storing the virtual address, enabling the client to access the corresponding intranet mirror source based on the virtual address. The front-end load node and the back-end load node are connected via TCP, and the back-end load node is connected to a preset port provided by the mirror source service node.

8. A mirror source management device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that can run on the processor, the processor executing the program to implement the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, implement the method described in any one of claims 1-6.

Images