A data access method, apparatus, device, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a data access method, apparatus, device, and computer-readable storage medium. When a segmented upload request transmitted by a client is obtained, a folder is created, and the name of the folder is set according to a set folder naming method and the object name carried in the segmented upload request. According to the segmented data and segment number carried in the segmented upload request, segmented files for storing each piece of segmented data are established under the folder, and each segmented file is named according to the corresponding segment number. After all the segmented data is successfully uploaded, an object file is created, and metadata for characterizing the file type and the storage locations of each segmented file is recorded in the object file. By creating a folder, the segmented data can be stored in the form of segmented files under the folder. By creating an object file that records the storage locations of the segmented files, subsequent retrieval of the segmented data can be supported, solving the problem that the S3 and NAS integrated interoperable storage system does not support segmented upload.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technologies, and particularly to a data access method, apparatus, device, and computer-readable storage medium. Background Art

[0002] There is a current market demand for the interoperability between the S3 (Simple Storage Service) and NAS (Network Attached Storage) protocols, that is, files written by NAS can be subsequently operated through the S3 protocol, and data written by S3 can also be operated through NAS subsequently. NAS can support multiple types of protocols to share a single piece of data simultaneously.

[0003] There are mainly two implementation methods for the current distributed storage system to support the integration and interoperability of S3 and NAS. The first implementation method is based on the S3 object storage as the base, and deploys the NFS Ganesha gateway to implement the conversion from the NFS file protocol to the S3 protocol; due to the semantic limitations of the underlying S3, common operations on NAS such as modify write, truncate table structure, and rename are not supported, and only the NFS protocol is supported and the CIFS protocol is not supported, which results in the loss of NAS protocol semantics.

[0004] The second implementation method is based on the distributed file system as the base to achieve the interoperability between S3 and NAS. This method can avoid the loss of NAS-related semantics and well implement the interoperability of multiple protocols. However, the current distributed S3-NAS integrated and interoperable storage system only supports the upload and download of ordinary files, and does not support the original large object segmented upload of S3. Because the size of each segment in segmented upload is not fixed and is completely determined by the client, the server does not know the size of each segment when receiving the upload request. This results in the inability to calculate the starting position of the data in this segment, and thus the data cannot be accurately written into a file. And multiple segmented uploads can be executed concurrently and are not sequentially uploaded in the order of segment numbers, which also leads to the inability to calculate the starting position of the current segment based on the already uploaded data, and the data cannot be accurately saved, resulting in data loss or errors.

[0005] It can be seen that how to enable the S3 and NAS integrated and interoperable storage system to support segmented upload of data is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of the embodiments of the present application is to provide a data access method, apparatus, device, and computer-readable storage medium, which can enable the S3 and NAS integrated and interoperable storage system to support segmented upload of data.

[0007] To solve the above technical problems, an embodiment of the present application provides a data access method, including:

[0008] When receiving a segmented upload request transmitted by a client, create a folder, and set the name of the folder according to a set folder naming method and the object name carried in the segmented upload request;

[0009] Based on the segmented data and segment numbers carried in the segmented upload request, create segmented files for storing each segmented data under the folder, and each of the segmented files is named according to the corresponding segment number;

[0010] After all segmented data is successfully uploaded, create an object file; record metadata in the object file for characterizing the file type and the storage locations of each of the segmented files.

[0011] Optionally, the setting the name of the folder according to a set folder naming method and the object name carried in the segmented upload request includes:

[0012] Concatenate a directory identifier, the object name, a segmented upload identifier, and an operation identifier corresponding to the segmented upload request as the name of the folder; wherein, the directory identifier is an identifier of the directory to which the folder belongs.

[0013] Optionally, the naming each of the segmented files according to the corresponding segment number includes:

[0014] Concatenate the segmented upload identifier and a target segment number as the name of a target segmented file; wherein, the target segment number is any one of all the segment numbers; and the target segmented file is the segmented file corresponding to the target segment number.

[0015] Optionally, the creating an object file after all segmented data is successfully uploaded includes:

[0016] For each received segmented data, feedback response information carrying the operation identifier and the segment number to the client;

[0017] When receiving a transmission completion request feedback by the client, use the verification information carried in the transmission completion request to verify all the segmented files included under the folder;

[0018] When all the segmented files included under the folder pass the verification, create an object file under the directory to which the folder belongs.

[0019] Optionally, after verifying all segmented files contained in the folder by using the verification information carried in the transmission completion request, the method further includes:

[0020] If there are segmented files that fail to pass the verification in the folder, a prompt message indicating that the segmented files are abnormal is displayed.

[0021] Optionally, after recording metadata for characterizing the file type and the storage location of each segment file in the object file, the method further includes:

[0022] In case of obtaining a download request transmitted by the client, reading a target object file matching the object name carried in the download request;

[0023] Determine whether the target object file has a multipart upload identifier recorded therein;

[0024] In a case where a multipart upload identifier is recorded in the target object file, obtaining a corresponding target multipart file according to a storage location of each multipart file recorded in the target object file;

[0025] The target segment files are concatenated according to their corresponding segment numbers to obtain a target data packet;

[0026] The target data packet is fed back to the client.

[0027] Optionally, after recording metadata for characterizing the file type and the storage location of each segment file in the object file, the method further includes:

[0028] In the case of obtaining a deletion request transmitted by the client, determining whether there is a target object file matching the object name carried in the deletion request;

[0029] If there is a target object file matching the object name carried in the deletion request, determining whether a multipart upload identifier is recorded in the target object file;

[0030] If the multipart upload identifier is not recorded in the target object file, deleting the target object file;

[0031] In the case where the target object file has a multipart upload identifier recorded therein, each multipart file and its corresponding folder are deleted according to the storage location of each multipart file recorded in the target object file, and the target object file is deleted.

[0032] The embodiment of the present application also provides a data access device, including a first creation unit, a setting unit, a building unit, a second creation unit and a recording unit;

[0033] The first creation unit is used to create a folder when a segmented upload request transmitted by a client is obtained;

[0034] The setting unit is used to set the name of the folder according to a set folder naming method and the object name carried in the segmented upload request;

[0035] The establishment unit is used to establish segmented files for storing each segmented data under the folder according to the segmented data and segment numbers carried in the segmented upload request, and each of the segmented files is named according to the corresponding segment number;

[0036] The second creation unit is used to create an object file after all segmented data are successfully uploaded;

[0037] The recording unit is used to record metadata for characterizing the file type and the storage locations of the segmented files in the object file.

[0038] Optionally, the setting unit is used to splice a directory identifier, the object name, a segmented upload identifier, and an operation identifier corresponding to the segmented upload request as the name of the folder; wherein, the directory identifier is an identifier of the directory to which the folder belongs.

[0039] Optionally, the establishment unit is used to splice the segmented upload identifier and a target segment number as the name of a target segmented file; wherein, the target segment number is any one of all the segment numbers; and the target segmented file is the segmented file corresponding to the target segment number.

[0040] Optionally, the second creation unit includes a feedback subunit, a verification subunit, and a creation subunit;

[0041] The feedback subunit is used to, every time a segmented data is obtained, feedback response information carrying the operation identifier and the segment number to the client;

[0042] The verification subunit is used to, when a transmission completion request feedback by the client is received, verify all the segmented files included under the folder by using the verification information carried in the transmission completion request;

[0043] The creation subunit is used to create an object file under the directory to which the folder belongs when all the segmented files included under the folder pass the verification.

[0044] Optionally, it further includes a display unit;

[0045] The display unit is used to display a prompt message of abnormal segmented file when there is a segmented file that fails to pass the verification under the folder.

[0046] Optionally, it further includes a reading unit, a first judging unit, an acquiring unit, a splicing unit and a feedback unit;

[0047] The reading unit is used to read the target object file matching the object name carried in the download request when the download request transmitted by the client is obtained;

[0048] The first determination unit is used to determine whether the target object file has a multipart upload identifier recorded therein;

[0049] The acquisition unit is configured to acquire the corresponding target segment file according to the storage position of each segment file recorded in the target object file when the segment upload identifier is recorded in the target object file;

[0050] The splicing unit is used to splice the target segment files according to their corresponding segment numbers to obtain a target data packet;

[0051] The feedback unit is used to feed back the target data packet to the client.

[0052] Optionally, it further includes a second judgment unit, a third judgment unit, a first deletion unit and a second deletion unit;

[0053] The second judgment unit is used to judge whether there is a target object file matching the object name carried in the deletion request when obtaining the deletion request transmitted by the client;

[0054] The third judgment unit is configured to judge whether a multipart upload identifier is recorded in the target object file if there is a target object file matching the object name carried in the deletion request;

[0055] The first deleting unit is configured to delete the target object file if the multipart upload identifier is not recorded in the target object file;

[0056] The second deleting unit is configured to, when the target object file has a segment upload identifier recorded therein, delete each segment file and its corresponding folder according to a storage location of each segment file recorded in the target object file, and delete the target object file.

[0057] The present application also provides an electronic device, including:

[0058] Memory for storing computer programs;

[0059] A processor is used to execute the computer program to implement the steps of the above data access method.

[0060] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data access method as described above are implemented.

[0061] It can be seen from the above technical solution that in the case of obtaining a segmented upload request transmitted by a client, a folder is created, and the name of the folder is set according to the set folder naming method and the object name carried in the segmented upload request. By setting the folder, support can be provided for the storage of segmented data, and the data uploaded by different segmented upload requests can be distinguished by the folder name. Considering that the lengths of different segmented data are not fixed, in order to implement the storage of segmented data, segmented files for storing each segmented data can be established under the folder according to the segmented data and the segment number carried in the segmented upload request. Each segmented file is named according to the corresponding segment number, and the segmented data is recorded in the segmented file with the same name as its segment name. In order to implement the retrieval of segmented data, after all segmented data is successfully uploaded, an object file can be created, and metadata for characterizing the file type and the storage location of each segmented file is recorded in the object file. In this technical solution, by creating a folder that supports the storage of segmented data, the segmented data can be stored in the folder in the form of segmented files, and by creating an object file that records the storage location of the segmented files, the subsequent retrieval of segmented data can be supported. The problem that the S3 and NAS integrated and interoperable storage system does not support segmented upload is solved, so that the integrated and interoperable storage system can implement segmented upload of large objects, not only can accurately save and download data, but also can greatly improve the object upload performance, bringing a good user experience to customers. At the same time, the functions of distributed integrated and interoperable storage are enriched, and the market competitiveness of distributed storage products is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application, the drawings required for the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0063] Figure 1 It is a flowchart of a data access method provided by an embodiment of the present application;

[0064] Figure 2 It is a schematic diagram of data segmented upload provided by an embodiment of the present application;

[0065] Figure 3 It is a flowchart of a method for downloading segmented data provided by an embodiment of the present application;

[0066] Figure 4 A schematic diagram of segmented data download provided by an embodiment of the present application;

[0067] Figure 5 A flowchart of a method for deleting segmented data provided by an embodiment of the present application;

[0068] Figure 6 A schematic diagram of deleting segmented data provided by an embodiment of the present application;

[0069] Figure 7 A schematic structural diagram of a data access device provided by an embodiment of the present application;

[0070] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0071] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0072] The terms "including" and "having" in the specification and claims of the present application and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may include steps or units not listed.

[0073] In order to enable those skilled in the art to better understand the solutions of the present application, the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0074] Next, a data access method provided by an embodiment of the present application will be introduced in detail. Figure 1 A flowchart of a data access method provided by an embodiment of the present application, the method includes:

[0075] S101: Create a folder when a segmented upload request transmitted by a client is obtained.

[0076] In the embodiment of the present application, a complete data packet uploaded by a client can be called an object. Considering that the data volumes of different objects vary, for an object with a large data volume, in order to ensure the performance of data upload, the object is often uploaded in segments.

[0077] For segmented upload requests belonging to the same object, these segmented upload requests will carry the same object name and operation identifier. For the same object, there may be multiple operations, and the operation identifier can be used to distinguish each operation.

[0078] After obtaining the segmented upload request transmitted by the client, in order to support the storage of segmented data, a folder can be created in the specified directory.

[0079] S102: Set the name of the folder according to the set folder naming method and the object name carried in the segmented upload request.

[0080] In the embodiments of the present application, in order to effectively distinguish the data transmitted by each segmented upload request, a folder naming method can be set.

[0081] In practical applications, the directory identifier, object name, segmented upload identifier, and the operation identifier corresponding to the segmented upload request can be concatenated as the name of the folder; among them, the directory identifier is the identifier of the directory to which the folder belongs.

[0082] The segmented upload identifier is used to indicate that the currently recorded data is segmented data. In practical applications, in addition to segmented data, there is also ordinary data. By setting the segmented upload identifier for segmented data, it can be used to identify whether the data belongs to segmented data or ordinary data. The data with the segmented upload identifier is segmented data, and the data without the segmented upload identifier is ordinary data.

[0083] It should be noted that in addition to concatenating in the order of the directory identifier, object name, segmented upload identifier, and operation identifier, these four types of parameters can also be randomly sorted and concatenated as the name of the folder, or more parameters can be added for concatenation. It only needs to ensure that the names of each folder are named in a unified order.

[0084] S103: Based on the segmented data and segment number carried in the segmented upload request, create segmented files for storing each segmented data under the folder, and each segmented file is named according to the corresponding segment number.

[0085] For the same object, it can be divided into multiple segmented data. In order to distinguish the front and back order of each segmented data, the segment number can be set based on the front and back order. In practical applications, numbers can be used to represent the segment number. The larger the numerical value, the more backward the position of the segmented data in the entire data packet. Of course, it can also be defined that the smaller the numerical value, the more backward the position of the segmented data in the entire data packet. In the embodiments of the present application, the specific form of the segment number is not limited, as long as it can be used to distinguish the front and back order of different segmented data.

[0086] In a specific implementation, the segmented upload identifier and the target segment number can be concatenated as the name of the target segment file; where the target segment number is any one of all the segment numbers; and the target segment file is the segment file corresponding to the target segment number.

[0087] It should be noted that, in addition to concatenating in the order of the segmented upload identifier and the target segment number, it is also possible to concatenate in the order of the target segment number and the segmented upload identifier as the name of the segment file. It is only necessary to ensure that the names of all segment files are named in a unified order.

[0088] S104: After all segmented data is successfully uploaded, create an object file; record metadata in the object file for characterizing the file type and the storage locations of each segment file.

[0089] In practical applications, during the data transmission phase, the server and the client will interact. The client can transmit segmented data to the server in parallel. For each segmented data obtained by the server, it can feedback response information carrying the operation identifier and the segment number to the client.

[0090] The file type can include a normal file type and a segmented file type.

[0091] After receiving the response information feedback by the server, the client can, based on the segment numbers corresponding to the same operation identifier received, identify whether the server has received all the segmented data of an object. After determining that the server has received all the segmented data of an object, it can feedback a transmission completion request to the server.

[0092] In the case where the server receives the transmission completion request feedback by the client, it can use the verification information carried in the transmission completion request to verify all the segment files included in the folder; in the case where all the segment files included in the folder pass the verification, create an object file in the directory to which the folder belongs.

[0093] The verification of the segment file can include the verification of the correctness and integrity of the segment file. The specific implementation method adopts the commonly used data verification method at present. For example, in practical applications, MD5 (Message-Digest Algorithm 5) can be used to verify the segment file.

[0094] Considering that in practical applications, there may be a situation where the verification of the segment file fails. For this situation, when there are segment files that fail to pass the verification in the folder, a prompt message indicating the abnormality of the segment file can be displayed. By displaying the prompt message, it is convenient for the management personnel to discover and check the segment file in time, and avoid the abnormal segment file from affecting the security of the storage system.

[0095] Figure 2 This is a schematic diagram of data segmented upload provided by an embodiment of this application. Figure 2 In it, "temp dir" represents the created folder, and "object file" represents the object file. The folder is created under the / tenant / bucket directory. In practical applications, the form of the object name can be diverse. Figure 2 The form of the object name is not limited in it. Only taking the concatenation in the order of the directory identifier, the segmented upload identifier, and the operation identifier as an example, that is, bucket_ino_multipart_uploadid. Among them, bucket_ino represents the directory identifier, multipart represents the segmented upload identifier, and uploadid represents the operation identifier. For multiple segmented data of the same object, they can be stored in this folder in the form of segmented files. Figure 2 Taking n segmented data as an example in it, the segment numbers of these n segmented data can be represented by the numbers 1 to n. Figure 2 Taking the method of directly appending the segment number after the folder name as an example in it. Considering that in practical applications, the segmented file belongs to the file under the folder, the segmented file can also be directly named with the segmented upload identifier and the target segment number. For example Figure 2 The first segmented file in it can be directly named "multipart_1".

[0096] As can be seen from the above technical solution, in the case of obtaining a segmented upload request transmitted by the client, a folder is created, and the name of the folder is set according to the set folder naming method and the object name carried in the segmented upload request; by setting the folder, support can be provided for the storage of segmented data, and the data uploaded by different segmented upload requests can be distinguished by the folder name. Considering that the lengths of different segmented data are not fixed, in order to implement the storage of segmented data, segmented files for storing each segmented data can be created under the folder according to the segmented data and the segment number carried in the segmented upload request. Each segmented file is named according to the corresponding segment number, and the segmented data is recorded in the segmented file with the same name as its segment name. In order to implement the retrieval of segmented data, after all segmented data is successfully uploaded, an object file can be created, and metadata for characterizing the file type and the storage location of each segmented file is recorded in the object file. In this technical solution, by creating a folder that supports the storage of segmented data, the segmented data can be stored in the folder in the form of segmented files, and by creating an object file that records the storage location of the segmented files, the subsequent retrieval of segmented data can be supported. The problem that the S3 and NAS integrated interoperable storage system does not support segmented upload is solved, so that the integrated interoperable storage system can implement segmented upload of large objects, not only can accurately save and download data, but also can greatly improve the object upload performance, bringing a good user experience to customers, while enriching the functions of the distributed integrated interoperable storage and improving the market competitiveness of distributed storage products.

[0097] Figure 3 The flowchart of a method for downloading segmented data provided by an embodiment of this application, the method includes:

[0098] S301: In the case of obtaining a download request transmitted by the client, read a target object file that matches the object name carried in the download request.

[0099] When the client needs to obtain segmented data, it can transmit a download request to the server. In order to facilitate the server to know which data the client needs to obtain, the object name can be carried in the download request.

[0100] After the server obtains the download request, it can search for the corresponding object file according to the object name. There may be multiple object files created by the server. For the convenience of distinction, the object file that matches the object name carried in the download request can be called the target object file.

[0101] S302: Determine whether a segmented upload identifier is recorded in the target object file.

[0102] The target object file may be an ordinary file or an object file that records metadata. In practical applications, the type of the target object file can be identified by determining whether a segmented upload identifier is recorded in the target object file.

[0103] If a segmented upload identifier is recorded in the target object file, it indicates that the target object file corresponds to segmented data. At this time, in order to obtain all the segmented data corresponding to the object name, S303 can be executed.

[0104] S303: Obtain the corresponding target segmented files according to the storage locations of each segmented file recorded in the target object file.

[0105] Metadata for characterizing the storage locations of each segmented file is recorded in the target object file. Based on the metadata, the server can read the corresponding segmented files.

[0106] The server often stores the segmented files corresponding to multiple objects respectively. In the embodiments of the present application, for the convenience of distinction, the segmented files corresponding to the target object file can be called target segmented files.

[0107] S304: Concatenate each target segmented file according to its corresponding segment number to obtain a target data packet, and feedback the target data packet to the client.

[0108] Each target segmented file has its corresponding segment number. Therefore, based on the segment number, each target segmented file can be concatenated in order to obtain the complete data. The obtained complete data can be called the target data packet. At this time, the server can feedback the target data packet to the client.

[0109] Figure 4 This is a schematic diagram of a segmented data download provided by the embodiments of the present application. Figure 4 In it, "temp dir" represents the created folder, "object file" represents the object file, and the folder is created under the / tenant / bucket directory. Figure 4 The meanings of the symbols in it can be referred to Figure 2 , which will not be elaborated here. Figure 4 In it, n segmented data are taken as an example, and the segment numbers of these n segmented data can be represented by the numbers 1 to n. When the server receives a download request transmitted by the client, it can match the target object file according to the object name carried in the download request, and obtain the corresponding target segmented files according to the storage locations of each segmented file recorded in the target object file. Concatenate each target segmented file according to its corresponding segment number to obtain the target data packet data. Figure 4The part_1 data represents the segmented data with a segment number of 1, and the part_n data represents the segmented data with a segment number of n. Concatenating the segmented data with segment numbers from 1 to n can obtain the data.

[0110] Figure 5 The flowchart of a method for deleting segmented data provided by an embodiment of the present application, the method includes:

[0111] S501: When receiving a deletion request transmitted by a client, determine whether there is a target object file that matches the object name carried in the deletion request.

[0112] When the server receives a deletion request, it needs to first determine whether the corresponding data is stored on the server. Therefore, when receiving a deletion request, it can be determined whether there is a target object file that matches the object name carried in the deletion request.

[0113] If there is no target object file that matches the object name carried in the deletion request, it means that the corresponding data is not stored on the server, and the deletion operation may not be performed. To facilitate the client to understand this situation, the server can feedback a prompt message indicating that the data has been deleted to the client.

[0114] If there is a target object file that matches the object name carried in the deletion request, it means that the corresponding data is stored on the server, and at this time, S502 can be executed.

[0115] S502: Determine whether a segmented upload identifier is recorded in the target object file.

[0116] Since the storage methods of ordinary files and segmented files are different, ordinary files do not need to be segmented. For ordinary files, the object file is equivalent to an ordinary file. For segmented files, in addition to multiple segmented files created in the same folder, it also includes an object file that records metadata. Therefore, there are differences in the deletion methods of ordinary files and segmented files. For ordinary files, the deletion operation can be directly performed, and for segmented files, all segmented files that match the object name and the corresponding object file need to be deleted.

[0117] In the embodiment of the present application, it can be determined whether the target object file belongs to an ordinary file or an object file that records metadata by whether a segmented upload identifier is recorded in the target object file.

[0118] In a specific implementation, it is possible to determine whether a segmented upload identifier is recorded in the target object file. If the segmented upload identifier is not recorded in the target object file, it indicates that the target object file is an ordinary file, and in this case, S503 can be executed. If the segmented upload identifier is recorded in the target object file, it indicates that the target object file is an object file that records metadata, and in this case, S504 can be executed.

[0119] S503: Delete the target object file.

[0120] S504: According to the storage locations of the segmented files recorded in the target object file, delete each segmented file and its corresponding folder, and then delete the target object file.

[0121] For segmented files, it is necessary to delete all segmented files that match the object name to ensure that the data corresponding to the object name is completely deleted. Therefore, in practical applications, when the segmented upload identifier is recorded in the target object file, each segmented file and its corresponding folder can be deleted according to the storage locations of the segmented files recorded in the target object file, and then the target object file can be deleted.

[0122] Figure 6 FIG. is a schematic diagram of data segmented deletion provided by an embodiment of the present application. Figure 6 In the figure, "temp dir" represents the created folder, "object file" represents the object file, and the folder is created under the / tenant / bucket directory. Figure 6 The meanings of the symbols in the figure can be referred to Figure 2 and will not be elaborated here. Figure 6 In the figure, n segmented data are taken as an example, and the segment numbers of these n segmented data can be represented by the numbers 1 to n. When the server receives the deletion request transmitted by the client, it can match the target object file according to the object name carried in the deletion request, and according to the storage locations of the segmented files recorded in the target object file, all target segmented files can be deleted. At this time, the folder where the target segmented file is located is empty, and the folder can be deleted. After deleting the segmented file and its corresponding folder, the target object file is no longer useful, and at this time, the target object file can be deleted.

[0123] In the embodiment of the present application, by constructing a folder, the segmented data is stored in the form of segmented files under the folder, and the storage locations of the segmented files are recorded through the object file, so that the storage system can support a series of operations such as segmented upload, download, and deletion. The client can perform segmented upload concurrently, which can greatly improve the object upload performance, bring a good user experience to the customer, and improve the market competitiveness of the distributed storage product.

[0124] Figure 7Schematic structural diagram of a data access device provided by an embodiment of the present application, including a first creation unit 71, a setting unit 72, an establishment unit 73, a second creation unit 74, and a recording unit 75;

[0125] The first creation unit 71 is configured to create a folder when a segmented upload request transmitted by a client is obtained;

[0126] The setting unit 72 is configured to set the name of the folder according to a set folder naming method and the object name carried in the segmented upload request;

[0127] The establishment unit 73 is configured to establish segmented files for storing each segmented data under the folder according to the segmented data and the segment number carried in the segmented upload request, and each segmented file is named according to the corresponding segment number;

[0128] The second creation unit 74 is configured to create an object file after all the segmented data is successfully uploaded;

[0129] The recording unit 75 is configured to record metadata for characterizing the file type and the storage location of each segmented file in the object file.

[0130] Optionally, the setting unit is configured to splice a directory identifier, an object name, a segmented upload identifier, and an operation identifier corresponding to the segmented upload request as the name of the folder; wherein, the directory identifier is an identifier of the directory to which the folder belongs.

[0131] Optionally, the establishment unit is configured to splice a segmented upload identifier and a target segment number as the name of the target segmented file; wherein, the target segment number is any one of all the segment numbers; the target segmented file is the segmented file corresponding to the target segment number.

[0132] Optionally, the second creation unit includes a feedback subunit, a verification subunit, and a creation subunit;

[0133] The feedback subunit is configured to, every time a segmented data is obtained, feedback response information carrying an operation identifier and a segment number to the client;

[0134] The verification subunit is configured to, when a transmission completion request feedback by the client is received, use the verification information carried in the transmission completion request to verify all the segmented files included in the folder;

[0135] The creation subunit is configured to create an object file under the directory to which the folder belongs when all the segmented files included in the folder pass the verification.

[0136] Optionally, it further includes a display unit;

[0137] The display unit is used to display a prompt message of abnormal segment files when there are segment files that fail to pass the verification in the folder.

[0138] Optionally, it further includes a reading unit, a first judging unit, an acquiring unit, a splicing unit and a feedback unit;

[0139] A reading unit, configured to read a target object file matching an object name carried in the download request when a download request transmitted by a client is obtained;

[0140] A first determination unit, configured to determine whether a multipart upload identifier is recorded in the target object file;

[0141] an acquisition unit, configured to acquire a corresponding target segment file according to a storage position of each segment file recorded in the target object file when a segment upload identifier is recorded in the target object file;

[0142] A splicing unit, used for splicing target segment files according to their corresponding segment numbers to obtain a target data packet;

[0143] The feedback unit is used to feed back the target data packet to the client.

[0144] Optionally, it further includes a second judgment unit, a third judgment unit, a first deletion unit and a second deletion unit;

[0145] A second judgment unit is used to judge whether there is a target object file matching the object name carried in the deletion request when a deletion request transmitted by the client is obtained;

[0146] A third determination unit is used to determine whether a multipart upload identifier is recorded in the target object file if there is a target object file matching the object name carried in the deletion request;

[0147] A first deleting unit, configured to delete a target object file when no multipart upload identifier is recorded in the target object file;

[0148] The second deleting unit is used to delete each segment file and its folder according to the storage position of each segment file recorded in the target object file, and delete the target object file when the segment upload identifier is recorded in the target object file.

[0149] Figure 7 The description of the features in the corresponding embodiments can be found in Figure 1 , Figure 3 and Figure 5 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.

[0150] As can be seen from the above technical solution, in the case of obtaining a segmented upload request transmitted by the client, a folder is created, and the name of the folder is set according to the set folder naming method and the object name carried in the segmented upload request. By setting up the folder, it can provide support for the storage of segmented data, and the data uploaded by different segmented upload requests can be distinguished by the folder name. Considering that the lengths of different segmented data are not fixed, in order to implement the storage of segmented data, segmented files for storing each segmented data can be created under the folder according to the segmented data and segment numbers carried in the segmented upload request. Each segmented file is named according to the corresponding segment number, and the segmented data is recorded in the segmented file with the same name as its segment name. In order to implement the retrieval of segmented data, after all segmented data is successfully uploaded, an object file can be created, and metadata for characterizing the file type and the storage locations of each segmented file is recorded in the object file. In this technical solution, by creating a folder that supports the storage of segmented data, the segmented data can be stored in the form of segmented files under this folder, and by creating an object file that records the storage locations of segmented files, it can support the subsequent retrieval of segmented data. This solves the problem that the S3 and NAS integrated interoperable storage system does not support segmented upload, enabling the integrated interoperable storage system to implement segmented upload of large objects, not only accurately saving and downloading data, but also greatly improving the object upload performance, bringing a good user experience to customers, enriching the functions of the distributed integrated interoperable storage, and enhancing the market competitiveness of distributed storage products.

[0151] Figure 8 The following is a structural diagram of an electronic device provided by an embodiment of the present application, as Figure 8 shown, the electronic device includes: a memory 20 for storing a computer program;

[0152] a processor 21 for implementing the steps of the data access method in the above embodiment when executing the computer program.

[0153] The electronic device provided in this embodiment may include, but is not limited to, a smart phone, a tablet computer, a notebook computer, or a desktop computer, etc.

[0154] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 may be implemented in at least one of the following hardware forms: DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for the rendering and drawing of the content to be displayed on the display screen. In some embodiments, the processor 21 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0155] The memory 20 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 20 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201. After the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the data access method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may further include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, folder naming methods, etc.

[0156] In some embodiments, the electronic device may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0157] Those skilled in the art can understand that Figure 8 the structure shown in

[0158] It can be understood that if the data access method in the above embodiments is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), electrically erasable programmable ROMs, registers, hard disks, removable disks, CD-ROMs, magnetic disks, or optical discs.

[0159] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data access method as described above are implemented.

[0160] The above has introduced in detail a data access method, device, equipment, and computer-readable storage medium provided by the embodiments of the present application. The various embodiments in the specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description of the method part.

[0161] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0162] The above has introduced in detail a data access method, apparatus, device, and computer-readable storage medium provided by this application. Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention. It should be noted that for those of ordinary skill in the art of this technology, without departing from the principle of the present invention, several improvements and modifications can be made to this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data access method, characterized in that, The method is applied to a scenario where multiple protocols share a piece of data, including: When a segmented upload request transmitted by a client is obtained, a folder is created, and the name of the folder is set according to a set folder naming method and the object name carried in the segmented upload request; among them, each segmented upload request belonging to the same object will carry the same object name and operation identifier; when there are multiple operations on the same object, the operation identifier is used to distinguish each operation; the folder name is formed by concatenating in the order of directory identifier, object name, segmented upload identifier, and operation identifier, or randomly sorting for concatenation, or adding other parameters for concatenation; the creation of the folder is used to support the storage of segmented data, and it is created in a specified directory, and the name of the folder is used to distinguish the data uploaded by different segmented upload requests; According to the segmented data and segment number carried in the segmented upload request, segmented files for storing each segmented data are created under the folder, and each of the segmented files is named according to the corresponding segment number; among them, the name of the target segmented file is formed by concatenating in the order of segmented upload identifier and target segment number, or by concatenating in the order of target segment number and segmented upload identifier; After all segmented data are uploaded successfully, an object file is created; metadata for characterizing the file type and the storage locations of each of the segmented files is recorded in the object file; Among them, after uploading the segmented data, when the transmission is completed, all segmented files under the folder are verified, and after all verifications pass, an object file is created. The verification includes the correctness and integrity verification of the segmented files; It also includes obtaining the target segmented file according to the storage locations of the segmented files recorded when the target object file records the segmented upload identifier, and concatenating the target segmented file according to the corresponding segment number to obtain the target data packet, and then feeding it back to the client.

2. The data access method according to claim 1, wherein The directory identifier is the identifier of the directory to which the folder belongs.

3. The data access method according to claim 2, wherein The target segment number is any one of all the segment numbers; the target segmented file is the segmented file corresponding to the target segment number.

4. The data access method according to claim 2, characterized in that, The step of creating an object file after all segmented data are uploaded successfully includes: For each segmented data obtained, a response message carrying the operation identifier and segment number is fed back to the client; When a transmission completion request fed back by the client is received, all segmented files included in the folder are verified by using the verification information carried in the transmission completion request; When all segmented files included in the folder pass the verification, an object file is created under the directory to which the folder belongs.

5. The data access method according to claim 4, wherein After verifying all segmented files included in the folder by using the verification information carried in the transmission completion request, it also includes: When there is a segmented file that fails to pass the verification in the folder, a prompt message indicating that the segmented file is abnormal is displayed.

6. The data access method according to claim 1, wherein After recording metadata for characterizing the file type and the storage locations of each of the segmented files in the object file, it also includes: When the download request transmitted by the client is obtained, read the target object file that matches the object name carried in the download request; Determine whether a chunked upload identifier is recorded in the target object file; When a chunked upload identifier is recorded in the target object file, obtain the corresponding target chunked files according to the storage locations of the respective chunked files recorded in the target object file; Concatenate the respective target chunked files according to their corresponding chunk numbers to obtain a target data packet; Feed back the target data packet to the client.

7. The data access method according to claim 1, wherein After recording metadata in the object file for characterizing the file type and the storage locations of the respective chunked files, it further includes: When the deletion request transmitted by the client is obtained, determine whether there is a target object file that matches the object name carried in the deletion request; When there is a target object file that matches the object name carried in the deletion request, determine whether a chunked upload identifier is recorded in the target object file; When a chunked upload identifier is not recorded in the target object file, delete the target object file; When a chunked upload identifier is recorded in the target object file, delete each chunked file and its affiliated folder according to the storage locations of the respective chunked files recorded in the target object file, and delete the target object file.

8. A data access device, characterized in that, It includes a first creation unit, a setting unit, an establishment unit, a second creation unit, and a recording unit; the device is applicable to a scenario where multiple protocols share the same data; The first creation unit is configured to create a folder when a chunked upload request transmitted by the client is obtained; The setting unit is configured to set the name of the folder according to the set folder naming method and the object name carried in the chunked upload request; wherein, each chunked upload request belonging to the same object will carry the same object name and operation identifier; when there are multiple operations on the same object, the operation identifier is used to distinguish each operation; concatenate in the order of the directory identifier, object name, chunked upload identifier, and operation identifier, or randomly sort for concatenation, or add other parameters for concatenation as the name of the folder; the creation of the folder is used to support the storage of chunked data, and it is created in the specified directory, and the name of the folder is used to distinguish the data uploaded by different chunked upload requests; The establishment unit is configured to establish chunked files for storing the respective chunked data under the folder according to the chunked data and chunk numbers carried in the chunked upload request, and each of the chunked files is named according to the corresponding chunk number; wherein, concatenate in the order of the chunked upload identifier and the target chunk number, or concatenate in the order of the target chunk number and the chunked upload identifier as the name of the target chunked file; The second creation unit is configured to create an object file after all the chunked data is successfully uploaded; The recording unit is configured to record metadata in the object file for characterizing the file type and the storage locations of the respective chunked files; Among them, after uploading segmented data, all segmented files in the folder are verified when the transmission is completed. After the verification passes, an object file is created. The verification includes the correctness and integrity verification of the segmented files; It also includes obtaining a target segmented file according to the storage locations of the recorded segmented files when the target object file records a segmented upload identifier, splicing the target segmented file according to the corresponding segment numbers to obtain a target data packet, and feeding it back to the client.

9. An electronic device, characterized in that, It includes: A memory for storing computer programs; A processor for executing the computer program to implement the steps of the data access method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the processor, the steps of the data access method according to any one of claims 1 to 7 are implemented.

Images