Mirror image warehouse implementation method and device based on multiple fragments

Abstract

The invention provides a method and a device for realizing a mirror image warehouse based on multiple fragments, and relates to the artificial intelligence fields of container technology, enterprise cloud, distributed storage, large models, code generation and the like. The method comprises the following steps: acquiring a data processing request from a client; determining a second fragment used for processing the data processing request from the deployed M first fragments, wherein M is a positive integer greater than 1; the data processing request is sent to the second fragment, one Harbor instance independently runs in each first fragment, and the Harbor instances are used for processing the sent data processing request.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method and device for implementing a multi-shard image repository in the fields of container technology, enterprise cloud, and distributed storage. Background Art

[0002] Currently, the backend of the image repository relies on a large-scale enterprise-level image repository service (Harbor) instance, which supports the enterprise's image management work. Summary of the invention

[0003] The present disclosure provides a method and device for implementing a multi-shard image repository.

[0004] A method for implementing a multi-shard image repository includes:

[0005] Get data processing request from client;

[0006] Determine a second shard for processing the data processing request from the deployed M first shards, where M is a positive integer greater than 1;

[0007] The data processing request is sent to the second shard, wherein an enterprise-level image repository service instance is independently run in each of the first shards, and the enterprise-level image repository service instance is used to process the sent data processing request.

[0008] A multi-shard-based image warehouse implementation device includes: a request acquisition module, a target determination module, and a request processing module;

[0009] The request acquisition module is used to obtain a data processing request from a client;

[0010] The target determination module is used to determine a second shard for processing the data processing request from the deployed M first shards, where M is a positive integer greater than 1;

[0011] The request processing module is used to send the data processing request to the second shard, wherein an enterprise-level image repository service instance independently runs in each of the first shards, and the enterprise-level image repository service instance is used to process the sent data processing request.

[0012] An electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0017] A computer program product comprises a computer program / instruction, wherein the computer program / instruction implements the above method when executed by a processor.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0020] Figure 1 It is a flowchart of an embodiment of a method for implementing a multi-shard image repository according to the present disclosure;

[0021] Figure 2 A schematic diagram of the relationship between the client, proxy and shards described in the present disclosure;

[0022] Figure 3 A schematic diagram of the interaction between the client, the proxy, the primary shard and other shards when the data processing request is a new request as described in the present disclosure;

[0023] Figure 4 A schematic diagram of the interaction between the client, the proxy and each shard when the data processing request is an update request or a delete request as described in the present disclosure;

[0024] Figure 5 A schematic diagram of the interaction between the client, the proxy and the primary shard when the data processing request is a query request as described in the present disclosure;

[0025] Figure 6 It is a schematic diagram of the composition structure of an embodiment 600 of the device for implementing a multi-shard image repository according to the present disclosure;

[0026] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] In addition, it should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0029] Figure 1 Flow chart of an embodiment of the method for implementing a multi-shard image repository according to the present disclosure. Figure 1 As shown, the following specific implementation methods are included.

[0030] In step 101, a data processing request is obtained from a client.

[0031] In step 102, a second shard for processing the data processing request is determined from the deployed M first shards, where M is a positive integer greater than 1.

[0032] In step 103, the data processing request is sent to the second shards, wherein a Harbor instance is independently run in each second shard, and the Harbor instance is used to process the sent data processing request.

[0033] With the in-depth implementation of enterprise cloud strategies, more and more services are being migrated to the cloud. Correspondingly, the demand for image resources has also increased dramatically. This growth is not only reflected in the number of images, but also in multiple dimensions such as image size, pull frequency, and concurrent request volume.

[0034] Faced with such a huge demand, the specifications of existing Harbor instances gradually reveal their limitations. For example, with the significant increase in the number of connections that Harbor instances need to handle, system resources are facing tremendous pressure. Moreover, for specific high-load scenarios, such as a user pulling large images in batches at a certain point in time (such as a research and development team centrally downloading multiple large images for testing), then this sudden high demand is likely to cause the Harbor instance to fall into an abnormal state due to resource exhaustion or untimely processing. Not only will the user's operation be affected, but it may also trigger a chain reaction, resulting in all users in the entire enterprise who rely on the Harbor instance for image management being unable to push and pull images normally, thereby affecting the stability and availability of the system.

[0035] In response to the above problems, the solution disclosed in the present invention proposes a method for implementing a mirror repository based on multiple shards, which adopts the Harbor instance sharding strategy, that is, deploying multiple first shards, and independently running a Harbor instance on each first shard. In this way, when a first shard cannot provide services normally due to an abnormal state, other first shards will not be affected and can still operate normally, thereby greatly reducing the impact range of the abnormality and further improving the overall stability and availability of the system. In addition, for each data processing request obtained, the corresponding second shard can be first determined from the multiple first shards, and then the data processing request can be sent to the determined second shard for processing, thereby ensuring that each data processing request can be processed correctly, thereby improving the accuracy of the processing results, etc.

[0036] The specific value of M can be determined according to actual needs. M first shards can be deployed, and each first shard can run a Harbor instance independently. In addition, each first shard can be bound to at least one project / space (project), that is, each first shard carries a group of specific projects, thereby achieving effective isolation of resource usage and data access between projects, reducing the abnormalities of other projects caused by problems in a single project, and thus improving the stability of the system.

[0037] in addition, Figure 1 The execution subject of the illustrated embodiment may be a proxy service located at the upper layer of each slice. Figure 2 Schematic diagram of the relationship between the client, proxy and shards described in this disclosure. Figure 2As shown, assuming that there are four first shards, namely shard 1, shard 2, shard 3 and shard 4, the Proxy can act as a bridge between the client and the first shards, responsible for routing and sending requests. For the client, it cannot perceive the existence of the first shards and only needs to interact with the Proxy without directly connecting to the first shards, which greatly simplifies the client configuration and operation. In addition, after the Proxy obtains the data processing request from the client, it can intelligently send it to the corresponding second shard, thereby ensuring that each data processing request can be correctly processed.

[0038] For the acquired data processing request, the Proxy may first determine a second shard for processing the data processing request from the deployed M first shards, and then send the data processing request to the second shard for processing.

[0039] In some embodiments of the present disclosure, the data processing request may be a new request. Accordingly, the main shard among the M first shards can be determined as the second shard, that is, the second shard includes the main shard in the first shard, and the main shard is a shard pre-set as the default route. After the data processing request is sent to the second shard, the data processing result returned by the second shard after adding the new configuration information corresponding to the data processing request to its own first key configuration table can be obtained, and the data processing result can be broadcast to the third shard, so that the third shard can add the new configuration information to its own second key configuration table according to the data processing result, wherein the third shard is the other shard in the first shard except the second shard, and the second key configuration table is consistent with the first key configuration table.

[0040] In order to enhance the flexibility and scalability of the system, the concept of primary shard is introduced in the solution described in the present disclosure. Which shard is determined as the primary shard can be determined according to actual needs. Figure 2 The shard 1 shown in is determined as the primary shard (i.e., determined as the default route). Accordingly, all data processing requests that do not explicitly specify a shard can be sent to the primary shard for processing, thereby simplifying the processing flow and further improving the processing efficiency of data processing requests.

[0041] Figure 3 This is a schematic diagram of the interaction between the client, proxy, primary shard and other shards when the data processing request is a new request as described in this disclosure. Figure 3As shown, assuming that there are four first shards, namely shard 1, shard 2, shard 3 and shard 4, and assuming that shard 1 is the primary shard, then the Proxy can send the new request obtained from the client to the primary shard, and the primary shard can add the new configuration information corresponding to the new request to its own first key configuration table, and can generate a data processing result, and then return the data processing result to the Proxy. Further, the Proxy can broadcast the data processing result to shard 2, shard 3 and shard 4, so that shard 2, shard 3 and shard 4 add the new configuration information to their own second key configuration table according to the data processing result and in accordance with the principle of consistency with the key configuration table in the primary shard.

[0042] Key configuration tables usually refer to configuration tables related to the object-relational database management system (postgresql) database, such as user group (user_group), harbor_user, project, project member / space member (project_member), project metadata (project_metadat), immutable tag rule (immutable_tag_rule), retention policy (retention_polic), etc.

[0043] Accordingly, by adopting the above-mentioned processing method for new requests, the new configuration information can be synchronized to different shards, thereby ensuring that the key configuration tables in each shard remain consistent.

[0044] For ease of description, in the solution described in the present disclosure, the key configuration tables corresponding to the second shard, the third shard, and the first shard are respectively referred to as the first key configuration table, the second key configuration table, and the third key configuration table.

[0045] In some embodiments of the present disclosure, the data processing request may also be an update request or a deletion request. Accordingly, the M first shards may all be determined as second shards, that is, the second shards include the M first shards. In addition, in response to determining that the data processing request is an update request, the update request may be broadcast to each second shard so that each second shard can update the configuration information corresponding to the update request in its own first key configuration table. In response to determining that the data processing request is a deletion request, the deletion request may be broadcast to each second shard so that each second shard can delete the configuration information corresponding to the deletion request in its own first key configuration table.

[0046] Figure 4 This is a schematic diagram of the interaction between the client, proxy, and each shard when the data processing request is an update request or a delete request as described in this disclosure. Figure 4As shown, assuming that there are four first shards (i.e., second shards) including shard 1, shard 2, shard 3, and shard 4, the Proxy can broadcast the update request or deletion request obtained from the client to each second shard. Accordingly, for the update request, each second shard can respectively update the configuration information corresponding to the update request in its own first key configuration table; for the deletion request, each second shard can respectively delete the configuration information corresponding to the deletion request in its own first key configuration table.

[0047] Through the above processing, it can be ensured that the key configuration tables in each shard remain consistent after update and delete operations.

[0048] For example, the above-mentioned new request may be a request to create a new project. The primary shard may generate corresponding new configuration information (new project information) for the newly created project and add it to its own first key configuration table, and may generate a corresponding project identifier (project_id). The project identifier may be used to identify the storage location of the new configuration information in the first key configuration table, etc., and then the new configuration information and the project identifier may be used to form a data processing result. The Proxy may carry the data processing result in the new project request and broadcast it to a third shard outside the primary shard. In actual applications, the interface for creating a space may be rewritten on the third shard to support carrying the data processing result in the message body (body) of the new project request.

[0049] For another example, the above-mentioned new request may be a request to add a space member (such as a member in a project). The main shard may generate corresponding new configuration information (new member information) for the added space member and add it to its own first key configuration table, and may generate a corresponding space member identifier (member_id). The space member identifier may be used to identify the storage location of the new configuration information in the key configuration table, etc., and then the new configuration information and the space member identifier may be used to form a data processing result. The Proxy may carry the data processing result in the new member request and broadcast it to a third shard outside the main shard. In actual applications, the corresponding interface may be rewritten on the third shard to support carrying the data processing result in the body of the new member request.

[0050] For another example, the update request may be a request to update a project, and the update request may be directly broadcast to all second shards (ie, all first shards) so that each second shard updates the configuration information corresponding to the update request in its own first key configuration table.

[0051] The above-mentioned addition request, update request and deletion request are all data processing requests involving configuration information changes in key configuration tables. When it comes to adding, deleting and updating (modifying) projects, adding, deleting and updating space members, as well as space configuration, user management and mail group management, corresponding data processing requests can be generated, and the above-mentioned processing methods can be used to ensure the consistency of key configuration tables in each shard, thereby improving the overall consistency and reliability of the system.

[0052] In some embodiments of the present disclosure, the data processing request may also be a query request. Accordingly, the primary shard among the M first shards may be determined as the second shard, that is, the second shard includes the primary shard, and after sending the data processing request to the second shard, the query result obtained by the second shard after querying its own first key configuration table can be obtained, and the query result can be returned to the client.

[0053] Figure 5 Schematic diagram of the interaction between the client, proxy and primary shard when the data processing request is a query request as described in the present disclosure. Figure 5 As shown, the query request may be a query request for project information or member information, etc. The Proxy may send the query request obtained from the client to the primary shard (assuming it is shard 1), and the primary shard obtains the query result by querying its own first key configuration table and returns it to the Proxy, which then returns the query result to the client.

[0054] Since the key configuration tables in each shard remain consistent, the query request can be directly processed by the primary shard to avoid shard selection, thereby simplifying the processing flow. However, in theory, it is also possible to send the query request to any other shard other than the primary shard for processing.

[0055] In some embodiments of the present disclosure, in response to determining that the M first shards are expanded, the third key configuration table in any first shard before the expansion may be copied to the shard after the expansion.

[0056] That is to say, when performing horizontal expansion of shards, the consistency of the key configuration tables of each first shard can be used to simplify operations. By simply copying the third key configuration table in any existing first shard to the database of the new shard, the new shard can be quickly enabled, thereby improving the scalability of the system, simplifying the expansion process, and reducing operational complexity. Moreover, the system performance can be significantly improved as the shards expand, thereby providing smoother and more efficient performance when dealing with high-concurrency requests and large-scale image management, and improving the stability and accuracy of the system.

[0057] In addition, in some embodiments of the present disclosure, in response to each predetermined period of time, the following processing can be performed respectively: the third key configuration tables in each first shard are compared, and in response to determining that there is an inconsistency, the third key configuration tables in each first shard are adjusted to a consistent state.

[0058] The specific value of the cycle duration can be determined according to actual needs, such as 5 minutes or 10 minutes.

[0059] In order to deal with the problem of inconsistent configuration between shards caused by network problems or other unexpected situations, the above-mentioned scheduled synchronization processing method can be adopted, that is, regularly checking whether there are differences in the key configuration tables between the shards, and automatically synchronizing when inconsistencies are found, so that the key configuration tables on all shards are kept up to date and consistent, thereby further improving the overall stability and reliability of the system.

[0060] In some embodiments of the present disclosure, the data processing request may also be a retention operation request, which may carry a retention policy identifier (retention_id). Accordingly, when determining the second shard for processing the data processing request from the deployed M first shards, the first project information corresponding to the target retention policy can be first determined by querying the first mapping relationship table, and then the second shard corresponding to the first project information can be determined by querying the second mapping relationship table. The target retention policy is the retention policy corresponding to the retention policy identifier. The first mapping relationship table stores the correspondence between different retention policies and corresponding project information, and the second mapping relationship table stores the correspondence between different first shards and corresponding project information.

[0061] Retention operations usually involve global or project-specific retention policy configurations, and the corresponding projects are usually not clearly stated in the retention operation request. Therefore, it is necessary to determine the first project information corresponding to the target retention policy based on the first mapping relationship table in the Proxy, and then further determine the second shard corresponding to the first project information based on the second mapping relationship table, and send the retention operation request to the second shard for processing, thereby improving the accuracy and efficiency of the retention operation.

[0062] The first mapping table stores the correspondence between the retention policy and the project information, and the second mapping table stores the correspondence between the first shard and the project information. For example, the second mapping table can store the correspondence between the virtual IP address (VIP, Virtual IP Address) of the Harbor instance in each first shard and the corresponding project information. In this way, when the client initiates a data processing request for a specific project, the Proxy can determine which first shard to send the data processing request to by querying the second mapping table, thereby improving the accuracy of routing, that is, each data processing request can be sent to the corresponding shard for processing. Moreover, this correspondence is dynamically configurable, so that it is convenient to make flexible adjustments when performing operations such as shard expansion or system maintenance.

[0063] In some embodiments of the present disclosure, the data processing request may also be a push-pull mirror operation request. Accordingly, when determining the second shard for processing the data processing request from the deployed M first shards, the request type of the push-pull mirror operation request may be determined first, and then the second shard may be determined from the first shard according to the shard determination method corresponding to the request type. That is, a flexible processing strategy may be adopted for different request types in the push-pull mirror operation.

[0064] In some embodiments of the present disclosure, the request type may be an initiation request ( / v2 / request). In response to determining that the initiation request carries account password information, the shard determination method may include: random determination or determination according to a predetermined strategy, that is, a first shard is randomly selected from each first shard to be determined as the second shard, or a first shard can be selected from each first shard according to a predetermined strategy to be determined as the second shard. In response to determining that the initiation request does not carry account password information, the initiation request may be refused to respond.

[0065] If it is determined that the initiating request does not carry account password information, the initiating request can be directly rejected, such as returning a 401 status code response directly to the client to reject unauthorized access. If it is determined that the initiating request carries account password information, a first shard can be randomly selected from each first shard to be determined as the second shard, or a first shard can be selected from each first shard according to a predetermined strategy to be determined as the second shard. There is no restriction on the specific predetermined side strategy. For example, the first shard with the smallest load can be selected as the second shard, which is very flexible and convenient.

[0066] In some embodiments of the present disclosure, the request type may also be a token acquisition request ( / service / token request). Then, the second project information corresponding to the token acquisition request may be first determined according to the query parameter carried in the token acquisition request. Then, the second shard corresponding to the second project information may be determined from the first shard by querying the second mapping relationship table.

[0067] In addition, in some embodiments of the present disclosure, the request type may also be a push-pull mirror request ( / v2 / ** type request), then the second shard corresponding to the third project information carried in the push-pull mirror request may be determined from the first shard by querying the second mapping relationship table.

[0068] The token acquisition request usually does not directly carry the project information, so the corresponding second project information can be determined according to the query parameters carried therein, while the push-pull image request usually directly carries the project information. The push-pull image request can be a block data (blob) and form (Manifest) operation request, etc.

[0069] It can be seen that, by adopting the above processing method, no matter which type of push or pull mirror operation request is, the corresponding second shard can be correctly determined, thereby improving the accuracy of subsequent processing results.

[0070] In combination with the foregoing introduction, it can be seen that in the scheme described in the present disclosure, the data processing requests obtained by the Proxy from the client may include new requests, update requests, deletion requests, query requests, retention operation requests, and push-pull mirror operation requests, etc. For different data processing requests, the second shard can be determined in a corresponding manner, and the data processing request can be sent to the second shard to complete subsequent related processing.

[0071] In addition, it should be noted that, for the aforementioned method embodiments, for the sake of simplicity of description, they are expressed as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0072] The above is an introduction to the method embodiment. The following is a further explanation of the scheme disclosed in the present invention through an apparatus embodiment.

[0073] Figure 6 FIG. 6 is a schematic diagram of the composition structure of an embodiment 600 of the image repository implementation device based on multiple shards described in the present disclosure. Figure 6As shown, it includes: a request acquisition module 601, a target determination module 602 and a request processing module 603.

[0074] The request acquisition module 601 is used to acquire a data processing request from a client.

[0075] The target determination module 602 is used to determine a second shard for processing the data processing request from the deployed M first shards, where M is a positive integer greater than 1.

[0076] The request processing module 603 is used to send the data processing request to the second shard, wherein a Harbor instance is independently run in each first shard, and the Harbor instance is used to process the sent data processing request.

[0077] By adopting the scheme described in the above-mentioned device embodiment, multiple first shards can be deployed, and a Harbor instance can be independently run on each first shard. In this way, when a first shard cannot provide services normally due to falling into an abnormal state, other first shards will not be affected and can still operate normally, thereby greatly reducing the impact range of the abnormality and improving the overall stability and availability of the system. In addition, for each data processing request obtained, the corresponding second shard can be first determined from the multiple first shards, and then the data processing request can be sent to the determined second shard for processing, thereby ensuring that each data processing request can be processed correctly, thereby improving the accuracy of the processing results, etc.

[0078] In some embodiments of the present disclosure, the data processing request may be a new request. Accordingly, the target determination module 602 may determine the primary shard among the M first shards as the second shard, that is, the second shard includes the primary shard in the first shard, and the primary shard is a shard pre-set as the default route. After sending the data processing request to the second shard, the request processing module 603 may also obtain the data processing result returned by the second shard after adding the new configuration information corresponding to the data processing request to its own first key configuration table, and may broadcast the data processing result to the third shard so that the third shard can add the new configuration information to its own second key configuration table based on the data processing result. The third shard is the other shard in the first shard except the second shard, and the second key configuration table is consistent with the first key configuration table.

[0079] In some embodiments of the present disclosure, the data processing request may also be an update request or a deletion request. Accordingly, the target determination module 602 may determine all of the M first shards as second shards, that is, the second shards include the M first shards. In addition, in response to determining that the data processing request is an update request, the request processing module 603 may broadcast the update request to each second shard, so that each second shard updates the configuration information corresponding to the update request in its own first key configuration table. In response to determining that the data processing request is a deletion request, the deletion request may be broadcast to each second shard, so that each second shard deletes the configuration information corresponding to the deletion request in its own first key configuration table.

[0080] In some embodiments of the present disclosure, the data processing request may also be a query request. Accordingly, the target determination module 602 may determine the primary shard among the M first shards as the second shard, that is, the second shard includes the primary shard. In addition, after sending the data processing request to the second shard, the request processing module 603 may obtain the query result obtained by the second shard after querying its own first key configuration table, and then return the query result to the client.

[0081] In some embodiments of the present disclosure, in response to determining that the M first shards are expanded, the request processing module 603 may copy the third key configuration table in any first shard before the expansion to the shard after the expansion.

[0082] That is to say, when performing horizontal expansion of shards, the consistency of the key configuration tables of each shard can be used to simplify the operation. The new shard can be quickly enabled by simply copying the third key configuration table in any existing first shard to the database of the new shard.

[0083] In addition, in some embodiments of the present disclosure, the request processing module 603 may perform the following processing in response to each predetermined period: compare the third key configuration tables in each first shard, and in response to determining that the third key configuration tables in each first shard are inconsistent, adjust each third key configuration table to a consistent state.

[0084] In some embodiments of the present disclosure, the data processing request may also be a retention operation request, which may carry a retention policy identifier. Accordingly, when the target determination module 602 determines the second shard for processing the data processing request from the deployed M first shards, it may first determine the first project information corresponding to the target retention policy by querying the first mapping relationship table, and then determine the second shard corresponding to the first project information from the first shard by querying the second mapping relationship table. The target retention policy is the retention policy corresponding to the retention policy identifier. The first mapping relationship table stores the correspondence between different retention policies and corresponding project information, and the second mapping relationship table stores the correspondence between different first shards and corresponding project information.

[0085] In some embodiments of the present disclosure, the data processing request may also be a push-pull mirror operation request. Accordingly, when the target determination module 602 determines the second shard for processing the data processing request from the deployed M first shards, it can first determine the request type of the push-pull mirror operation request, and then determine the second shard from the first shards according to the shard determination method corresponding to the request type.

[0086] Among them, in some embodiments of the present disclosure, the request type may be an initiation request. In response to determining that the initiation request carries account password information, the target determination module 602 may randomly select a first shard from each first shard to be determined as the second shard, or may select a first shard from each first shard to be determined as the second shard according to a predetermined strategy, that is, the shard determination method may include: random determination or determination according to a predetermined strategy. In addition, in response to determining that the initiation request does not carry account password information, the initiation request may be refused to be responded to.

[0087] In some embodiments of the present disclosure, the request type may also be a token acquisition request. Then, the target determination module 602 may first determine the second project information corresponding to the token acquisition request based on the query parameters carried in the token acquisition request, and then determine the second shard corresponding to the second project information from the first shard by querying the second mapping relationship table.

[0088] In addition, in some embodiments of the present disclosure, the request type may also be a push-pull mirror request, then the target determination module 602 may determine the second shard corresponding to the third project information carried in the push-pull mirror request from the first shard by querying the second mapping relationship table.

[0089] Figure 6 The specific working process of the illustrated device embodiment can refer to the relevant description in the aforementioned method embodiment and will not be described in detail.

[0090] In summary, by adopting the solution described in this disclosure and using the Harbor instance sharding strategy, etc., the stability, availability, scalability, and data consistency of the system are significantly improved, and the management and maintenance costs of the system are reduced, thereby providing strong technical support for the enterprise cloud strategy and providing enterprises with a more efficient and reliable image warehouse solution.

[0091] The solution disclosed in this disclosure can be applied to the field of artificial intelligence, especially to the fields of container technology, enterprise cloud, distributed storage, large models, and code generation. Artificial intelligence is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It has both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing. Artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0092] In addition, the data processing requests in the embodiments described in this disclosure are not for a specific user and do not reflect the personal information of a specific user. In the technical solution of this disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good customs.

[0093] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0094] Figure 7 A schematic block diagram of an electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0095] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0096] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0097] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI, Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP, Digital Signal Processing), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the methods described in the present disclosure. For example, in some embodiments, the methods described in the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the methods described in the present disclosure may be executed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method described in the present disclosure in any other appropriate manner (for example, by means of firmware).

[0098] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system on chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general programmable processor, which can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0100] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM, Electronically Programmable Read-Only Memory), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0102] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0103] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0104] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0105] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for implementing a multi-shard image repository, comprising: Get data processing request from client; Determine a second shard for processing the data processing request from the deployed M first shards, where M is a positive integer greater than 1; The data processing request is sent to the second shard, wherein an enterprise-level image repository service instance is independently run in each of the first shards, and the enterprise-level image repository service instance is used to process the sent data processing request.

2. The method according to claim 1, wherein: The data processing request includes: a new addition request; The second shard includes a primary shard in the first shard, and the primary shard is a shard preset as a default route; After sending the data processing request to the second shard, the method further includes: Obtaining a data processing result returned by the second shard after adding the newly added configuration information corresponding to the data processing request to its own first key configuration table; Broadcast the data processing result to the third shard, so that the third shard adds the newly added configuration information to its own second key configuration table according to the data processing result, wherein the third shard is the other shards in the first shard except the second shard, and the second key configuration table is consistent with the first key configuration table.

3. The method according to claim 2, wherein: The data processing request includes: an update request or a deletion request; The second shard includes M of the first shards; The sending the data processing request to the second shard comprises: In response to determining that the data processing request is the update request, broadcasting the update request to the second shard, so that the second shard updates the configuration information corresponding to the update request in the first key configuration table; In response to determining that the data processing request is the deletion request, broadcasting the deletion request to the second shard, so that the second shard deletes the configuration information corresponding to the deletion request in the first key configuration table.

4. The method according to claim 2, wherein: The data processing request includes: a query request; The second shard includes the primary shard; After sending the data processing request to the second shard, the method further includes: Obtain a query result obtained by the second shard after querying the first key configuration table, and return the query result to the client.

5. The method according to claim 2, 3 or 4, further comprising: In response to determining that the first shard is expanded, the third key configuration table in any of the first shards before the expansion is copied to the shard after the expansion.

6. The method according to claim 2, 3 or 4, further comprising: In response to each predetermined period of time, the following processing is performed respectively: in response to determining that the third key configuration tables in each of the first slices are inconsistent, each of the third key configuration tables is adjusted to a consistent state.

7. The method according to any one of claims 1 to 4, wherein: The data processing request includes: a retention operation request, wherein the retention operation request carries a retention policy identifier; The determining, from the deployed M first shards, a second shard for processing the data processing request comprises: Determine first project information corresponding to the target retention policy by querying the first mapping relationship table; The second shard corresponding to the first project information is determined from the first shard by querying the second mapping relationship table, the target retention policy is the retention policy corresponding to the retention policy identifier, the first mapping relationship table stores the correspondence between different retention policies and corresponding project information, and the second mapping relationship table stores the correspondence between different first shards and corresponding project information.

8. The method according to any one of claims 1 to 4, wherein: The data processing request includes: a push-pull mirroring operation request; The determining, from the deployed M first shards, a second shard for processing the data processing request comprises: Determine the request type of the push-pull mirroring operation request; The second shard is determined from the first shard according to the shard determination method corresponding to the request type.

9. The method according to claim 8, wherein: The request types include: initiating a request; In response to determining that the initiation request carries account password information, the shard determination method includes: randomly determining or determining according to a predetermined strategy; The method further comprises: In response to determining that the initiation request does not carry the account password information, refusing to respond to the initiation request.

10. The method according to claim 8, wherein: The request types include: token acquisition request; The determining the second slice from the first slice according to the slice determination method corresponding to the request type includes: Determine second item information corresponding to the token acquisition request according to the query parameter carried in the token acquisition request; The second shard corresponding to the second project information is determined from the first shard by querying the second mapping relationship table, wherein the second mapping relationship table stores corresponding relationships between different first shards and corresponding project information.

11. The method according to claim 8, wherein: The request types include: push and pull image requests; The determining the second slice from the first slice according to the slice determination method corresponding to the request type includes: The second shard corresponding to the third project information carried in the push-pull mirror request is determined from the first shard by querying the second mapping relationship table, wherein the second mapping relationship table stores corresponding relationships between different first shards and corresponding project information.

12. The method according to any one of claims 1 to 4, wherein: The method is applied to the proxy server at the upper layer of each shard.

13. A device for implementing a multi-shard image repository, comprising: Request acquisition module, target determination module and request processing module; The request acquisition module is used to obtain a data processing request from a client; The target determination module is used to determine a second shard for processing the data processing request from the deployed M first shards, where M is a positive integer greater than 1; The request processing module is used to send the data processing request to the second shard, wherein an enterprise-level image repository service instance independently runs in each of the first shards, and the enterprise-level image repository service instance is used to process the sent data processing request.

14. The device according to claim 13, wherein: The data processing request includes: a new addition request; The second shard includes a primary shard in the first shard, and the primary shard is a shard preset as a default route; The request processing module is further used to, after sending the data processing request to the second shard, obtain the data processing result returned by the second shard after adding the newly added configuration information corresponding to the data processing request to its own first key configuration table, and broadcast the data processing result to the third shard, so that the third shard can add the newly added configuration information to its own second key configuration table according to the data processing result, wherein the third shard is a shard other than the second shard in the first shard, and the second key configuration table is consistent with the first key configuration table.

15. The device according to claim 14, wherein: The data processing request includes: an update request or a deletion request; The second shard includes M of the first shards; In response to determining that the data processing request is the update request, the request processing module broadcasts the update request to the second shard so that the second shard updates the configuration information corresponding to the update request in the first key configuration table. In response to determining that the data processing request is the deletion request, the request processing module broadcasts the deletion request to the second shard so that the second shard deletes the configuration information corresponding to the deletion request in the first key configuration table.

16. The device according to claim 14, wherein: The data processing request includes: a query request; The second shard includes the primary shard; The request processing module is further used to, after sending the data processing request to the second shard, obtain a query result obtained by the second shard after querying the first key configuration table, and return the query result to the client.

17. The device according to claim 14, 15 or 16, wherein: The request processing module is further configured to, in response to determining that the first shard has been expanded, copy the third key configuration table in any of the first shards before the expansion to the shard after the expansion.

18. The device according to claim 14, 15 or 16, wherein: The request processing module is further used to perform the following processing in response to each predetermined period: in response to determining that the third key configuration tables in each of the first slices are inconsistent, adjust each of the third key configuration tables to a consistent state.

19. The device according to any one of claims 13 to 16, wherein: The data processing request includes: a retention operation request, wherein the retention operation request carries a retention policy identifier; The target determination module determines the first project information corresponding to the target retention policy by querying the first mapping relationship table, and determines the second shard corresponding to the first project information from the first shard by querying the second mapping relationship table. The target retention policy is the retention policy corresponding to the retention policy identifier. The first mapping relationship table stores the correspondence between different retention policies and corresponding project information, and the second mapping relationship table stores the correspondence between different first shards and corresponding project information.

20. The device according to any one of claims 13 to 16, wherein: The data processing request includes: a push-pull mirroring operation request; The target determination module determines a request type of the push-pull mirroring operation request, and determines the second slice from the first slice according to a slice determination method corresponding to the request type.

21. The device according to claim 20, wherein: The request types include: initiating a request; In response to determining that the initiation request carries account password information, the shard determination method includes: randomly determining or determining according to a predetermined strategy; The target determination module is further configured to, in response to determining that the initiation request does not carry the account password information, refuse to respond to the initiation request.

22. The device according to claim 20, wherein: The request types include: token acquisition request; The target determination module determines the second project information corresponding to the token acquisition request based on the query parameters carried in the token acquisition request, and determines the second shard corresponding to the second project information from the first shard by querying the second mapping relationship table. The second mapping relationship table stores the correspondence between different first shards and corresponding project information.

23. The device according to claim 20, wherein: The request types include: push and pull image requests; The target determination module determines the second shard corresponding to the third project information carried in the push-pull image request from the first shard by querying the second mapping relationship table, and the second mapping relationship table stores the correspondence between different first shards and corresponding project information.

24. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

25. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1 to 12.

26. A computer program product, comprising a computer program / instruction, which implements the method according to any one of claims 1 to 12 when executed by a processor.

Images