Object-oriented storage heterogeneous computing resource management system storage fusion method

Abstract

The application discloses a heterogeneous computing resource management system storage fusion method for object-oriented storage, and relates to the technical field of storage.The heterogeneous computing resource management system storage fusion method is applied to a heterogeneous computing resource management system.The method comprises the following steps: in response to a file operation event triggered based on a storage management platform, obtaining a target storage system, a storage path and an operation to be performed of a file to be operated; in the case that the target storage system is an object storage system, searching for object metadata of the file to be operated in the object storage system based on the storage path to obtain a search result; and performing the operation to be performed on the file to be operated based on the search result.The application can connect the object storage system to the heterogeneous computing resource management system, so that the heterogeneous computing resource management system can provide storage services for a heterogeneous computing cluster in Huawei Cloud.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of storage, in particular to a heterogeneous computing power resource management system storage fusion method for object storage. BACKGROUND

[0002] With the development of digital economy and the promotion of Internet of Things, the amount of data is growing explosively, which on the one hand promotes the development of high-performance computing, big data, artificial intelligence and other applications, and these applications need to process larger and larger data; on the other hand, it gives birth to file systems with different characteristics, which can meet the needs of different application scenarios and performance, capacity and cost.

[0003] At present, supercomputing centers, intelligent computing centers and cloud computing computing power platforms can use computing clusters of different architectures, and provide support for different types of applications and jobs through a heterogeneous computing power resource management system, and the storage system accessed by the heterogeneous computing power resource management system is usually a parallel file system and a distributed file system. However, when the computing power platforms use the heterogeneous computing clusters in Huawei Cloud, the object storage system used by the heterogeneous computing clusters in Huawei Cloud cannot be directly accessed to the heterogeneous computing power resource management system.

[0004] Therefore, how to access the object storage system to the heterogeneous computing power resource management system to provide storage services for the heterogeneous computing clusters in Huawei Cloud through the heterogeneous computing power resource management system is a problem that needs to be solved at present. SUMMARY

[0005] The main purpose of the present application is to provide a heterogeneous computing power resource management system storage fusion method for object storage, which aims to access the object storage system to the heterogeneous computing power resource management system to provide storage services for the heterogeneous computing clusters in Huawei Cloud through the heterogeneous computing power resource management system.

[0006] To achieve the above purpose, the present application provides a heterogeneous computing power resource management system storage fusion method for object storage, which is applied to a heterogeneous computing power resource management system, the heterogeneous computing power resource management system comprising a storage management platform and a file gateway, the file gateway being in communication connection with an object storage system.

[0007] The heterogeneous computing power resource management system storage fusion method for object storage comprises the following steps:

[0008] In response to a file operation event triggered based on the storage management platform, the target storage system, storage path and operation to be performed of a file to be operated are obtained.

[0009] In a case where the target storage system is the object storage system, object metadata of the to-be-operated file is found in the object storage system based on the storage path, and a finding result is obtained;

[0010] The to-be-operated file is subjected to the to-be-executed operation based on the finding result.

[0011] In an embodiment, in a case where the to-be-executed operation is file addition, the step of subjecting the to-be-operated file to the to-be-executed operation based on the finding result comprises:

[0012] If the finding result is that the object metadata does not exist, an object operation interface is invoked through the file gateway to add the to-be-operated file in the object storage system;

[0013] If the finding result is that the object metadata exists, a first prompt is returned to the storage management platform through the file gateway, wherein the first prompt represents that the to-be-operated file already exists in the object storage system.

[0014] In an embodiment, in a case where the to-be-executed operation is file access or file modification, the step of subjecting the to-be-operated file to the to-be-executed operation based on the finding result comprises:

[0015] If the finding result is that the object metadata does not exist, a second prompt is returned to the storage management platform through the file gateway, wherein the second prompt represents that the to-be-operated file does not exist in the object storage system;

[0016] If the finding result is that the object metadata exists, an object operation interface is invoked through the file gateway to access or modify at least one object corresponding to the object metadata in the object storage system.

[0017] In an embodiment, the step of finding object metadata of the to-be-operated file in the object storage system based on the storage path, and obtaining a finding result comprises:

[0018] The storage path is parsed to obtain a bucket name and an object name of the to-be-operated file;

[0019] The object metadata of the to-be-operated file is found in the object storage system based on the bucket name and the object name, and a finding result is obtained.

[0020] In an embodiment, after the step of finding object metadata of the to-be-operated file in the object storage system based on the storage path, and obtaining a finding result, the method further comprises:

[0021] In a case that the search result is that the object metadata exists, first user information in the object metadata is acquired;

[0022] The first user information is compared with second user information triggering the file operation event, and a comparison result is obtained;

[0023] In a case that the comparison result is consistent comparison, the step of performing the to-be-executed operation on the to-be-operated file based on the search result is executed.

[0024] In an embodiment, the storage management platform is in communication connection with the file gateway, and after the step of performing the to-be-executed operation on the to-be-operated file based on the search result, the method further comprises:

[0025] A file operation result returned by the object storage system is received through the file gateway, and the file operation result is returned to the storage management platform.

[0026] In an embodiment, the file gateway is in communication connection with a file storage system, and after the step of acquiring the target storage system, the storage path and the to-be-executed operation of the to-be-operated file in response to the file operation event triggered based on the storage management platform, the method further comprises:

[0027] In a case that the target storage system is the file storage system, a file operation interface is called through the file gateway, so that the to-be-executed operation is performed on the to-be-operated file based on the storage path in the file storage system.

[0028] In addition, to achieve the above-mentioned purpose, the present application also provides a storage fusion device, which is applied to a heterogeneous computing resource management system, the heterogeneous computing resource management system comprising a storage management platform and a file gateway, the file gateway being in communication connection with an object storage system;

[0029] The storage fusion device comprises:

[0030] An acquisition module is configured to acquire, in response to a file operation event triggered based on the storage management platform, a target storage system, a storage path and a to-be-executed operation of a to-be-operated file;

[0031] A search module is configured to, in a case that the target storage system is the object storage system, search object metadata of the to-be-operated file in the object storage system based on the storage path, and obtain a search result;

[0032] An operation module is configured to perform the to-be-executed operation on the to-be-operated file based on the search result.

[0033] In addition, to achieve the above object, the present application also provides a storage medium, which is a computer readable storage medium, and a program for implementing the object-oriented storage heterogeneous computing resource management system storage fusion method is stored on the computer readable storage medium. The program for implementing the object-oriented storage heterogeneous computing resource management system storage fusion method is executed by a processor to implement the steps of the object-oriented storage heterogeneous computing resource management system storage fusion method as described above.

[0034] In addition, to achieve the above object, the present application also provides a computer program product, which comprises a computer program. When the computer program is executed by a processor, the steps of the object-oriented storage heterogeneous computing resource management system storage fusion method as described above are implemented.

[0035] The present application provides an object-oriented storage heterogeneous computing resource management system storage fusion method. The method is applied to a heterogeneous computing resource management system, which comprises a storage management platform and a file gateway. The file gateway is in communication connection with an object storage system. In response to a file operation event triggered based on the storage management platform, the target storage system, storage path and to-be-executed operation of a to-be-operated file are obtained. In the case where the target storage system is an object storage system, object metadata of the to-be-operated file in the object storage system is searched based on the storage path, and a search result is obtained. Then, the to-be-operated file is subjected to the to-be-executed operation based on the search result.

[0036] In this way, the present application establishes a file gateway in the heterogeneous computing resource management system, establishes a communication connection between the file gateway and the object storage system, obtains the target storage system, storage path and to-be-executed operation of the to-be-operated file in response to a file operation event triggered by a user based on the storage management platform, searches for object metadata of the to-be-operated file in the object storage system in the case where the target storage system is an object storage system, obtains a search result, and subjects the to-be-operated file to the to-be-executed operation based on the search result. Thus, the object storage system is connected to the heterogeneous computing resource management system, so as to provide storage services for the heterogeneous computing cluster in the Huawei cloud through the heterogeneous computing resource management system. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0039] Figure 1 This is a flow chart of the first embodiment of the storage fusion method for the heterogeneous computing resource management system for object-oriented storage of the present application;

[0040] Figure 2 This is a schematic diagram of the object metadata acquisition process involved in an embodiment of the storage fusion method for a heterogeneous computing resource management system for object storage of this application;

[0041] Figure 3 This is a schematic diagram of the file operation process involved in an embodiment of the storage fusion method of the heterogeneous computing resource management system for object-oriented storage of this application;

[0042] Figure 4 A schematic diagram of a file operation architecture involved in an embodiment of a storage fusion method for a heterogeneous computing resource management system for object-oriented storage of the present application;

[0043] Figure 5 This is a schematic diagram of the module structure of the storage fusion device of this application;

[0044] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the storage fusion method of the object-oriented heterogeneous computing resource management system in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0047] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0048] The main solution of the present application is: in response to a file operation event triggered based on the storage management platform, obtaining the target storage system, storage path and operation to be performed of the file to be operated; when the target storage system is the object storage system, searching the object metadata of the file to be operated in the object storage system based on the storage path to obtain a search result; and performing the operation to be performed on the file to be operated based on the search result.

[0049] Currently, the computing power platforms such as supercomputing centers, intelligent computing centers and cloud computing can use computing clusters of different architectures, and provide support for different types of applications and jobs through heterogeneous computing power resource management systems, and the storage systems accessed by the heterogeneous computing power resource management systems are usually parallel file systems and distributed file systems. However, when the computing power platforms use the heterogeneous computing clusters in Huawei Cloud, the object storage system used by the heterogeneous computing clusters in Huawei Cloud cannot be directly accessed to the heterogeneous computing power resource management system.

[0050] The application establishes a file gateway in the heterogeneous computing power resource management system, establishes a communication connection between the file gateway and the object storage system, responds to a file operation event triggered by a user based on a storage management platform, obtains a target storage system, a storage path and an operation to be performed of a file to be operated, and when the target storage system is an object storage system, finds object metadata of the file to be operated in the object storage system to obtain a finding result, so as to perform the operation to be performed on the file to be operated based on the finding result, thereby realizing the access of the object storage system to the heterogeneous computing power resource management system to provide storage services for the heterogeneous computing clusters in Huawei Cloud through the heterogeneous computing power resource management system.

[0051] It should be noted that the execution subject of the method in each embodiment of the energy storage system configuration method can be a heterogeneous computing power resource management system, or a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a storage fusion device capable of realizing the above functions, and the present embodiment does not make specific limitation thereon. The present embodiment and each of the following embodiments will be described below with the heterogeneous computing power resource management system as an execution subject. For ease of description, the execution subject is omitted in the following description.

[0052] Based on this, the application proposes a first embodiment of an object storage-oriented heterogeneous computing power resource management system storage fusion method, please refer to Figure 1 , the object storage-oriented heterogeneous computing power resource management system storage fusion method includes steps S10-S30:

[0053] Step S10, in response to a file operation event triggered based on the storage management platform, obtaining a target storage system, a storage path and an operation to be performed of a file to be operated;

[0054] It should be noted that the heterogeneous computing resource management system includes a storage management platform and a file gateway. The storage management platform provides an interactive interface for users to perform file operations. The file gateway is used to connect the object storage system to the heterogeneous computing resource management system, so that users can access data in the object storage system through traditional file protocols. That is, after user authentication, users can perform operations on the storage management platform in the form of files, regardless of whether the computing cluster uses a traditional file storage system or an object storage system. When a user triggers a file operation event on the storage management platform, the user needs to input the storage system (hereinafter referred to as the target storage system for distinction), the storage path, and the operation to be performed on the storage management platform. The operation to be performed includes but is not limited to file addition, file access, and file modification. The file to be operated is a single file or a folder.

[0055] When a user triggers a file operation event based on the storage management platform, the heterogeneous computing resource management system responds to the file operation event and obtains the target storage system, the storage path, and the operation to be performed on the file to be operated through the file gateway.

[0056] In step S20, if the target storage system is the object storage system, the object metadata of the file to be operated is searched in the object storage system based on the storage path, and a search result is obtained.

[0057] It should be noted that, in order to make the file format of the upper-layer heterogeneous computing resource management system consistent with the storage form of the object storage system, the data between the platform and the system can be correctly parsed, accessed, and transmitted. Therefore, the transmitted data is parsed and re-encapsulated in the file gateway. In order to solve the problem that the object storage system does not support traditional file system operations, a new logic is established in the file operation module of the object storage system. The bucket operations and object operations supported by the object storage system are encapsulated into traditional file system operations. Since a new logic is established in the file operation module of the object storage system, the file permission judgment method is also different. Therefore, it is necessary to identify in the file gateway whether the file in the file operation request is from the original traditional file storage system or the newly added object storage system, so as to distribute the file operation to the correct logic board. In this way, the file gateway unifies the input and output formats of different storage systems, provides a unified data interface, and users can obtain object storage system proxy service authorization and operate different storage systems through the unified service gateway.

[0058] Further, it needs to be explained that after the file gateway identifies that the target storage system of the current file to be operated is an object storage system, the logic of the object storage system file processing module is executed. Specifically, the basic composition of the object storage is a bucket and an object. The bucket is a container for storing objects in the OBS, and each bucket has its own storage category, access permission, and region attribute. The object is the basic unit of data storage in the OBS (Object Storage Service), and an object is actually a collection of file data content and its related attribute information, including Key, Metadata, and Data, which represent key value (i.e., the name of the object), object description information (i.e., object metadata, which is uploaded to the OBS in the form of key-value pairs), and file data content. Since object storage is a key-value storage, it does not have the concept of folders compared with traditional storage, so in order to correspond to the traditional file operation, the embodiment of the application uses an empty file with a key name (key) ending with a slash “ / ” to simulate a file or a folder, and the files under the same folder will have a common key prefix ending with “ / ”. Therefore, searching for all files under a folder means searching for all files with the folder name as the prefix. This way of dividing levels according to directories is more in line with the user's operation habits.

[0059] In the case where the target storage system of the current file to be operated is an object storage system, the object metadata of the file to be operated is searched in the object storage system based on the storage path, and a search result is obtained. The search result is that the object metadata exists or does not exist, i.e., the object metadata of the file to be operated can be found in the object storage system and the object metadata of the file to be operated cannot be found in the object storage system, respectively.

[0060] In the embodiment, the step S20 can include steps S201-S202:

[0061] Step S201, the storage path is parsed to obtain the bucket name and object name of the file to be operated;

[0062] It needs to be explained that since the embodiment of the application uses an empty file with a key name ending with a slash “ / ” to simulate a file or a folder, after the file gateway obtains the storage path of the file to be operated, the storage path can be parsed to obtain the bucket name and object name of the file to be operated.

[0063] Step S202, the object metadata of the file to be operated is searched in the object storage system based on the bucket name and the object name, and a search result is obtained.

[0064] Find the object metadata of the to-be-operated file in the object storage system based on the bucket name and the object name of the to-be-operated file, and obtain a search result.

[0065] Exemplarily, as shown in Figure 2 An object metadata acquisition flowchart is shown. The file gateway parses the bucket name and the object name of the to-be-operated file based on the storage path of the to-be-operated file, finds the object metadata in the object storage system of Huawei Cloud based on the bucket name and the object name, obtains the object metadata after finding the corresponding object metadata, finds the OBS storage object (i.e., the to-be-operated file) according to the object metadata, and then performs the to-be-executed operation on the to-be-operated file.

[0066] Step S30: Perform the to-be-executed operation on the to-be-operated file based on the search result.

[0067] Perform the to-be-executed operation on the to-be-operated file according to the search result of finding the object metadata of the to-be-operated file in the object storage system.

[0068] In a feasible implementation, in the case where the to-be-executed operation is file addition, the step S30 can include steps A10-A20.

[0069] Step A10: If the search result is that the object metadata does not exist, call an object operation interface through the file gateway to add the to-be-operated file in the object storage system.

[0070] Step A20: If the search result is that the object metadata exists, return a first prompt to the storage management platform through the file gateway, where the first prompt indicates that the to-be-operated file already exists in the object storage system.

[0071] It should be noted that file addition is file pasting or file creation. The file gateway is in communication connection with the object storage system, and the file gateway can call an object operation structure to access the object storage system.

[0072] In this embodiment, if the search result is that the object metadata does not exist, the file gateway calls an object operation interface to add the to-be-operated file in the object storage system; if the search result is that the object metadata exists, the file gateway returns a prompt (hereinafter referred to as a first prompt for distinction) to the storage management platform, where the first prompt indicates that the to-be-operated file already exists in the object storage system.

[0073] Exemplarily, when a user needs to paste a file or newly create a file to a storage path, the file gateway first searches for file metadata of the file to be pasted or the file to be newly created in the object storage system based on the storage path. If the search result is that the object metadata exists, it indicates that the storage space pointed to by the storage path of the object storage system already has the file to be pasted or the file to be newly created, and therefore a first prompt needs to be output to prompt the user that the file is repeatedly pasted or repeatedly created.

[0074] In another possible implementation, in a case where the operation to be performed is file access or file modification, the step S30 can include steps B10-B20:

[0075] In step B10, if the search result is that the object metadata does not exist, a second prompt is returned to the storage management platform through the file gateway, where the second prompt represents that the file to be operated does not exist in the object storage system.

[0076] In step B20, if the search result is that the object metadata exists, an object operation interface is invoked through the file gateway to access or modify at least one object corresponding to the object metadata in the object storage system.

[0077] It should be noted that the file access is file download, file viewing, file cutting or file copying, and the file modification is file deletion or file renaming.

[0078] If the search result is that the object metadata does not exist, a prompt is returned to the storage management platform through the file gateway, where the second prompt represents that the file to be operated does not exist in the object storage system. If the search result is that the object metadata exists, an object operation interface is invoked through the file gateway to access or modify at least one object corresponding to the object metadata in the object storage system.

[0079] The embodiment of the present application responds to a file operation event triggered based on a storage management platform, obtains a target storage system, a storage path and an operation to be performed of a file to be operated. In a case where the target storage system is an object storage system, object metadata of the file to be operated is searched in the object storage system based on the storage path, a search result is obtained, and then the file to be operated is subjected to the operation to be performed based on the search result.

[0080] Thus, by establishing a file gateway in the heterogeneous computing resource management system, the embodiment of the application establishes a communication connection with the object storage system through the file gateway, obtains the target storage system, storage path and to-be-executed operation of the to-be-operated file in response to a file operation event triggered by a user based on a storage management platform, and finds object metadata of the to-be-operated file in the object storage system when the target storage system is the object storage system, obtains a finding result, and performs the to-be-executed operation on the to-be-operated file based on the finding result, thereby realizing the access of the object storage system to the heterogeneous computing resource management system to provide storage services for the heterogeneous computing cluster in Huawei Cloud through the heterogeneous computing resource management system.

[0081] Based on the first embodiment of the application, in the second embodiment of the application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and the subsequent will not be described in detail. On this basis, after the step S20, the object storage-oriented heterogeneous computing resource management system storage fusion method comprises steps C10-C30:

[0082] Step C10, in the case that the finding result is that the object metadata exists, obtaining first user information in the object metadata;

[0083] Step C20, comparing the first user information and second user information triggering the file operation event to obtain a comparison result;

[0084] Step C30, in the case that the comparison result is consistent, performing the step of performing the to-be-executed operation on the to-be-operated file based on the finding result.

[0085] It should be noted that, in order to provide a simple Unix file permission model, the embodiment of the application sets metadata for controlling file permissions on file objects. The metadata indicates the user, group, and permission bits (indicating read, write, and executable) of the file. When a user creates a file or a folder (a file ending with " / "), the metadata of the file will be automatically set according to the user ID and group ID of the user. Meanwhile, the embodiment of the application also provides an interface for modifying the permission metadata of the file. When a user needs to operate a file, the user needs to check whether the user has the operation permission of the file according to the permission metadata of the corresponding file (folder). In order to improve the efficiency of this operation, the embodiment of the application adds an LRU (Least Recently Used) cache for caching permission metadata to reduce the access of the object storage system and reduce the operation delay.

[0086] In the case that the search result is that the object metadata exists, user information in the object metadata (hereinafter referred to as first user information for the sake of distinction) is acquired, user information of a user triggering the current file operation event (hereinafter referred to as second user information for the sake of distinction) is acquired, the first user information and the second user information are compared, a comparison result is obtained, and in the case that the comparison result is consistent, the step of performing the to-be-executed operation on the to-be-operated file based on the search result is executed. It should be noted that the user information includes a user ID and a group ID.

[0087] In this way, before the file gateway is used to call the object operation interface to operate the file data in the object storage system, the identity information of the user triggering the current file operation event is verified, so that the legality of the file operation performed by the user is ensured.

[0088] In the embodiment, the storage management platform is in communication connection with the file gateway, and after the step S30, the method further includes the following step D10:

[0089] In the embodiment, the storage management platform is in communication connection with the file gateway, and after the step S30, the method further includes the following step D10:

[0090] After the to-be-operated file is operated, the object storage system returns the file operation result to the file gateway, the file operation result is operation success or operation failure, the file operation result returned by the object storage system is received through the file gateway, and the file operation result is returned to the storage management platform, so that the user can view the file operation result on the storage management platform.

[0091] In the specific embodiment, when the user submits a storage management operation such as renaming a file, copying and pasting a file, etc. to the OBS storage in the storage management platform, the storage management platform will send the file operation information to the file gateway by calling the related interface of the file gateway. The file gateway can recognize the file type as the OBS object storage type by analyzing the target path of the operation file, and convert the file object into the OBS object storage type, then perform permission authentication, determine that the user has the permission to operate the target object (i.e. the file to be operated), and according to the mapping logic of the traditional file operation and the OBS object storage operation (such as copying a file to copy the object in the corresponding bucket on the OBS, creating a folder to create an object with “ / ” at the end of the object name on the OBS), call the object operation interface service of the OBS storage service. For synchronous object operations such as creating a folder, renaming a file or a folder, etc., the file gateway waits for the data returned by the object operation interface of the OBS storage service, integrates the returned data and returns the successful operation result to the storage management platform. For asynchronous object operations such as deleting a file, copying and pasting a file, etc., the file operation is initially in execution, and the status of the file operation process is updated into a one-to-one file operation log. For each log, a corresponding data pipeline is established, and the log information, i.e. the file operation execution status, is sent to the storage management platform through the websocket service.

[0092] Exemplarily, as shown in Figure 3 The file operation flowchart shows that the user triggers a file operation event on the OBS in the storage management platform of the heterogeneous computing resource management system, the file gateway obtains the target storage system, storage path and to-be-executed operation of the current to-be-operated file, and then the OBS storage processing module of the file gateway includes a permission judgment unit, a synchronous file operation processing unit and an asynchronous file operation processing unit. The permission judgment unit is used to request the object metadata of the current to-be-operated file, perform permission judgment on the user triggering the file operation event based on the object metadata, and after determining that the user has the permission to perform the file operation, call the object operation interface through the synchronous file operation processing unit to execute the to-be-executed operation on the to-be-operated file in the Huawei Cloud, receive the file operation result, and return the file operation result value to the storage management platform. Alternatively, the asynchronous file operation processing unit includes an asynchronous file log pool, calls the object operation interface through the asynchronous file operation processing unit to execute the to-be-executed operation on the to-be-operated file in the Huawei Cloud, records the file operation result by using the asynchronous file log pool, and then returns the file operation result to the storage management platform through the websocket.

[0093] The embodiments of the present application consider that when a folder-related operation is performed, an operation involving multiple object files can be required. In order to reduce the delay of multi-file operation, the file gateway initiates an OBS file operation request in a multi-file concurrent request manner when the number of operation files reaches a certain number, and finally returns the operation result by one thread. At the same time, due to network problems, large file upload and download failures will cause unnecessary waste of resources. In order to reduce such influence, the job gateway and the related interface of the object storage are aligned and adapted to realize the breakpoint resume and slice download function of the OBS file. In this way, the client can use these functions to avoid the invalidation of previous upload and download due to network anomalies during large file transmission.

[0094] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and the subsequent will not be repeated. On this basis, the file gateway is in communication connection with the file storage system, and after the step S10, the object storage-oriented heterogeneous computing resource management system storage fusion method comprises a step E10:

[0095] Step E10, in the case that the target storage system is the file storage system, calling a file operation interface through the file gateway to perform the to-be-executed operation on the to-be-operated file based on a storage path in the file storage system.

[0096] It should be noted that the file gateway is in communication connection with the file storage system, and the file gateway can call the file operation interface to interact with the file storage system.

[0097] When the file gateway detects that the obtained target storage system is the file storage system, the to-be-operated file is operated in the file storage system based on the storage path in a traditional file operation manner.

[0098] Exemplarily, as shown in Figure 4 is a file operation architecture schematic diagram, a user can access a storage management platform, and the file gateway is in communication connection with the storage management platform, the file storage system and the object storage system.

[0099] The embodiments of the present application also provide a storage fusion device, which is applied to a heterogeneous computing resource management system, the heterogeneous computing resource management system comprising a storage management platform and a file gateway, and the file gateway being in communication connection with an object storage system; please refer to Figure 5 , the storage fusion device comprises:

[0100] The acquisition module 10 is configured to acquire a target storage system, a storage path and a to-be-executed operation of a to-be-operated file in response to a file operation event triggered based on the storage management platform.

[0101] A search module 20 is configured to search for object metadata of the file to be operated in the object storage system based on the storage path when the target storage system is the object storage system, and obtain a search result;

[0102] The operation module 30 is configured to perform the operation to be performed on the file to be operated based on the search result.

[0103] Optionally, when the operation to be performed is adding a new file, the operation module 30 is further configured to:

[0104] If the search result shows that the object metadata does not exist, calling the object operation interface through the file gateway to add the file to be operated in the object storage system;

[0105] If the search result shows that the object metadata exists, a first prompt is returned to the storage management platform through the file gateway, wherein the first prompt indicates that the file to be operated already exists in the object storage system.

[0106] Optionally, when the operation to be performed is file access or file modification, the operation module 30 is further configured to:

[0107] If the search result is that the object metadata does not exist, returning a second prompt to the storage management platform through the file gateway, wherein the second prompt indicates that the file to be operated does not exist in the object storage system;

[0108] If the search result shows that the object metadata exists, the object operation interface is called through the file gateway to access or modify at least one object corresponding to the object metadata in the object storage system.

[0109] Optionally, the search module 20 is further configured to:

[0110] Parsing the storage path to obtain the bucket name and object name of the file to be operated;

[0111] The object metadata of the file to be operated is searched in the object storage system based on the bucket name and the object name to obtain a search result.

[0112] Optionally, the storage fusion device further includes a permission verification module, and the permission verification module is configured to:

[0113] If the search result shows that the object metadata exists, obtaining first user information in the object metadata;

[0114] comparing the first user information with second user information triggering the file operation event to obtain a comparison result;

[0115] In a case where the comparison result is consistent with the comparison, the step of performing the to-be-executed operation on the to-be-operated file based on the search result is executed.

[0116] Optionally, the storage management platform is in communication connection with the file gateway, and the storage fusion device further comprises a result returning module, the result returning module being configured to:

[0117] receive, through the file gateway, a file operation result returned by the object storage system, and return the file operation result to the storage management platform.

[0118] Optionally, the file gateway is in communication connection with a file storage system, and the storage fusion device further comprises a file storage module, the file storage module being configured to:

[0119] In a case where the target storage system is the file storage system, a file operation interface is invoked through the file gateway to perform the to-be-executed operation on the to-be-operated file based on a storage path in the file storage system.

[0120] The storage fusion device provided by the embodiment of the present application adopts the object storage-oriented heterogeneous computing resource management system storage fusion method in the above embodiment, and can solve the technical problem of how to connect an object storage system to a heterogeneous computing resource management system to provide storage services for a heterogeneous computing cluster in Huawei Cloud through the heterogeneous computing resource management system. Compared with the prior art, the storage fusion device provided by the embodiment of the present application has the same beneficial effects as the object storage-oriented heterogeneous computing resource management system storage fusion method provided by the above embodiment, and other technical features in the storage fusion device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0121] The present application provides a storage fusion device, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the object storage-oriented heterogeneous computing resource management system storage fusion method in the above embodiment one.

[0122] The following refers to Figure 6, which shows a schematic diagram of the structure of a storage convergence device suitable for implementing the embodiments of the present application. The storage convergence device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The storage fusion device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0123] like Figure 6 As shown, the storage fusion device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the storage fusion device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 can allow the storage convergence device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a storage convergence device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems can be implemented or have alternatively.

[0124] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0125] The storage fusion device provided by the present application adopts the object-oriented storage heterogeneous computing resource management system storage fusion method in the above-mentioned embodiments, and can solve the technical problem of how to connect the object storage system to the heterogeneous computing resource management system to provide storage services for the heterogeneous computing cluster in the Huawei cloud through the heterogeneous computing resource management system. Compared with the prior art, the storage fusion device provided by the present application has the same beneficial effects as the object-oriented storage heterogeneous computing resource management system storage fusion method provided by the above-mentioned embodiments, and other technical features in the storage fusion device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0126] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0127] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0128] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer program) for executing the object-oriented storage heterogeneous computing resource management system storage fusion method in the above-mentioned embodiments.

[0129] The computer readable storage medium provided in the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0130] The computer readable storage medium described above may be contained in a storage fusion device, or may exist separately without being assembled into the storage fusion device.

[0131] The computer readable storage medium described above carries one or more programs, which, when executed by the storage fusion device, cause the storage fusion device to: in response to a file operation event triggered based on the storage management platform, obtain a target storage system, a storage path, and an operation to be performed for a file to be operated; in a case where the target storage system is the object storage system, find object metadata of the file to be operated in the object storage system based on the storage path to obtain a finding result; and perform the operation to be performed on the file to be operated based on the finding result.

[0132] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0133] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0134] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0135] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the above-mentioned object-oriented storage heterogeneous computing resource management system storage fusion method, and can solve the technical problem of thermal management of the heterogeneous computing resource management system. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the object-oriented storage heterogeneous computing resource management system storage fusion method provided by the above-mentioned embodiments, which will not be repeated here.

[0136] The embodiment of the present application provides a computer program product, comprising a computer program which, when executed by a processor, implements the steps of the object-oriented storage heterogeneous computing resource management system storage fusion method as described above.

[0137] The computer program product provided by the present application can solve the technical problem of how to improve the heat dissipation efficiency of the heterogeneous computing resource management system. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the present application are the same as those of the object-oriented storage heterogeneous computing resource management system storage fusion method provided by the above-mentioned embodiment, and are not described here.

[0138] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, is also included in the patent processing scope of the present application.

Claims

1. A storage fusion method for a heterogeneous computing resource management system based on object storage, characterized in that: The object-oriented storage heterogeneous computing resource management system storage fusion method is applied to the heterogeneous computing resource management system, which includes a storage management platform and a file gateway. The file gateway is communicatively connected to the object storage system, and the OBS storage processing module of the file gateway includes a synchronous file operation processing unit and an asynchronous file operation processing unit. The object-oriented storage heterogeneous computing resource management system storage fusion method includes: In response to a file operation event triggered by the storage management platform, obtaining, through the file gateway, a target storage system, a storage path, and an operation to be performed for the file to be operated; In a case where the target storage system is the object storage system, searching the object metadata of the file to be operated in the object storage system based on the storage path by the file gateway to obtain a search result; Performing the operation to be performed on the file to be operated based on the search result by the file gateway; The step of performing the operation to be performed on the file to be operated includes: The object operation interface is called by the synchronous file operation processing unit or the asynchronous file operation processing unit to perform the operation to be performed on the file to be operated.

2. The method according to claim 1, wherein In a case where the operation to be performed is adding a file, the step of performing the operation to be performed on the file to be operated based on the search result by the file gateway includes: If the search result shows that the object metadata does not exist, calling the object operation interface through the file gateway to add the file to be operated in the object storage system; If the search result shows that the object metadata exists, a first prompt is returned to the storage management platform through the file gateway, wherein the first prompt indicates that the file to be operated already exists in the object storage system.

3. The method according to claim 1, wherein In a case where the operation to be performed is file access or file modification, the step of performing the operation to be performed on the file to be operated based on the search result by the file gateway includes: If the search result is that the object metadata does not exist, returning a second prompt to the storage management platform through the file gateway, wherein the second prompt indicates that the file to be operated does not exist in the object storage system; If the search result shows that the object metadata exists, the object operation interface is called through the file gateway to access or modify at least one object corresponding to the object metadata in the object storage system.

4. The method according to claim 1, wherein The step of searching the object metadata of the file to be operated in the object storage system based on the storage path by the file gateway to obtain a search result includes: Parsing the storage path through the file gateway to obtain the bucket name and object name of the file to be operated; The object metadata of the file to be operated is searched in the object storage system based on the bucket name and the object name to obtain a search result.

5. The method according to claim 1, wherein After the step of searching, by the file gateway, the object metadata of the file to be operated in the object storage system based on the storage path and obtaining a search result, the method further includes: If the search result shows that the object metadata exists, obtaining first user information in the object metadata; Comparing the first user information with the second user information that triggered the file operation event to obtain a comparison result; If the comparison result is consistent, the step of performing the operation to be performed on the file to be operated based on the search result is executed.

6. The method according to any one of claims 1 to 5, characterized in that The storage management platform is in communication with the file gateway. After the step of performing the operation to be performed on the file to be operated based on the search result by the file gateway, the method further includes: The file operation result returned by the object storage system is received through the file gateway, and the file operation result is returned to the storage management platform.

7. The method according to claim 1, wherein The file gateway is in communication with the file storage system. After the step of obtaining, through the file gateway, the target storage system, storage path, and operation to be performed of the file to be operated in response to the file operation event triggered by the storage management platform, the method further includes: In a case where the target storage system is the file storage system, a file operation interface is called through the file gateway to perform the operation to be performed on the file to be operated based on the storage path in the file storage system.

8. A storage fusion device, characterized in that: The storage fusion device is applied to a heterogeneous computing resource management system, which includes a storage management platform and a file gateway. The file gateway is in communication with the object storage system, and the OBS storage processing module of the file gateway includes a synchronous file operation processing unit and an asynchronous file operation processing unit. The storage fusion device includes: An acquisition module, configured to obtain, through the file gateway, a target storage system, a storage path, and an operation to be performed for a file to be operated in response to a file operation event triggered based on the storage management platform; a search module configured to, when the target storage system is the object storage system, search the object metadata of the file to be operated in the object storage system based on the storage path through the file gateway to obtain a search result; An operation module is used to perform the operation to be executed on the file to be operated based on the search result through the file gateway; and call the object operation interface through the synchronous file operation processing unit or the asynchronous file operation processing unit to execute the operation to be executed on the file to be operated.

9. A storage fusion device, characterized in that: The storage fusion device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the storage fusion method of the object-oriented storage heterogeneous computing resource management system as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the storage fusion method of the object-oriented storage heterogeneous computing resource management system are implemented as described in any one of claims 1 to 7.

Images