Data management method and device, electronic device and storage medium

Abstract

This specification provides a data management method and apparatus, an electronic device, and a storage medium. The data management method includes: receiving a write request sent by a client to instruct writing data to be written into a target file; determining target data within the data to be written based on the end position of the data stored in the target file and the requested write position in the write request, wherein the target data is part or all of the data to be written; generating verification information based on a user identifier of the client and data information of the target data, and generating a sub-file to be written based on the verification information and the target data; and writing the sub-file to be written into the target file with the end position of the target file as the starting position.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of database technology, and in particular, to a data management method and apparatus, an electronic device, and a storage medium. Background Art

[0002] In the database field, appending (also known as appending) is a common data operation. It refers to adding new data to the end of an existing dataset rather than overwriting or modifying existing data. Appending can ensure data integrity and efficient writing. Object storage, as a data storage model without a directory structure, can store and manage data as objects, making it ideal for storing unstructured data such as images, videos, and documents.

[0003] In the database field, there is often a need to append data to object storage files. However, the existing methods for appending data to object storage files cannot guarantee data security, especially when appending data to object storage files in Write Once Read Many (WORM) mode, which is more likely to threaten data security. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a data management method and apparatus, an electronic device, and a storage medium.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a data management method is proposed, the method comprising:

[0007] receiving a write request sent by a client for instructing to write data to be written into a target file, and determining target data within the data to be written based on an end position of data stored in the target file and a requested write position in the write request, wherein the target data is part or all of the data to be written;

[0008] generating verification information based on the user identification of the client and the data information of the target data, and generating a sub-file to be written based on the verification information and the target data;

[0009] The end position of the target file is used as the starting position, and the sub-file to be written is written into the target file.

[0010] In a possible embodiment of the present specification, determining the target data in the data to be written based on the end position of the stored data in the target file and the requested write position in the write request includes:

[0011] If the offset of the requested write position is consistent with the offset of the end position of the target file, the data to be written is determined as the target data, wherein the offset is used to represent the offset relative to the starting position of the target file.

[0012] In a possible embodiment of the present specification, the write request includes the data length of the data to be written;

[0013] The determining the target data within the data to be written based on the end position of the target file and the requested write position in the write request includes:

[0014] If the offset of the requested write position is less than the offset of the end position of the target file, determining the offset of the end position of the data to be written based on the offset of the requested write position and the data length of the data to be written, wherein the offset is used to represent the offset relative to the start position of the target file;

[0015] If the offset of the end position of the data to be written is greater than the offset of the end position of the data stored in the target file, the data in the data to be written whose offset is greater than the offset of the end position of the target file is determined as the target data.

[0016] In a possible embodiment of the present specification, determining the offset data in the data to be written, whose offset is greater than the end position of the target file, as the target data includes:

[0017] Determine data in the data to be written whose offset is not greater than the offset of the end position of the target file, and perform consistency verification with the stored data corresponding to the same offset in the target file;

[0018] If the result of the consistency verification is passed, the data whose offset in the data to be written is greater than the offset of the end position of the target file is determined as the target data.

[0019] In a possible embodiment of the present specification, generating verification information based on the user identifier of the client and the data information of the target data includes:

[0020] The user identification of the client and the data information of the target data are input into a preset verification function, and the verification value output by the verification function is used as verification information.

[0021] In a possible embodiment of the present specification, generating the to-be-written sub-file based on the verification information and the target data includes:

[0022] An initial sub-file is generated based on the target data, and the verification information is added to the header of the initial sub-file to obtain a sub-file to be written.

[0023] In a possible embodiment of this specification, the method further includes:

[0024] The starting position of the target data is determined as the subfile name of the subfile to be written, and the length of the target data is determined as the data length of the subfile to be written.

[0025] In a possible embodiment of this specification, the method further includes:

[0026] receiving a read request sent by a client for instructing to read data to be read from a target file, and determining at least one target sub-file in the target file based on a data location range in the read request;

[0027] For each target subfile, generating verification information based on the user identifier of the client and data information of the data in the target subfile, and determining whether the target subfile is valid based on the generated verification information and the verification information in the target subfile;

[0028] Based on all valid target sub-files and the data location range in the read request, the data to be read is returned to the client.

[0029] According to a second aspect of one or more embodiments of this specification, a data management device is provided, the device comprising:

[0030] a target data module, configured to receive a write request sent by a client for instructing to write data to be written into a target file, and determine target data within the data to be written based on an end position of data stored in the target file and a requested write position in the write request, wherein the starting position of the target data is the end position of data stored in the target file;

[0031] a subfile module, configured to generate verification information based on the user identification of the client and the data information of the target data, and to generate a subfile to be written based on the verification information and the target data;

[0032] The writing module is used to write the sub-file to be written into the target file.

[0033] According to a third aspect of one or more embodiments of this specification, a computer program product is provided, comprising a computer program / instruction, which implements the steps of the method described in the first aspect when executed by a processor.

[0034] According to a fourth aspect of one or more embodiments of this specification, an electronic device is provided, including:

[0035] processor;

[0036] a memory for storing processor-executable instructions;

[0037] The processor implements the method described in the first aspect by running the executable instructions.

[0038] According to a fifth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect are implemented.

[0039] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0040] The data management method provided in the embodiments of this specification receives a write request sent by a client to instruct writing the data to be written into a target file, determines the target data in the data to be written based on the end position of the data stored in the target file and the requested write position in the write request; generates verification information based on the user identification of the client and the data information of the target data, and generates a sub-file to be written based on the verification information and the target data; and writes the sub-file to be written into the target file with the end position of the target file as the starting position. The method writes the target data in the data to be written in the form of a sub-file, so there is no need to modify or overwrite the data already written in the target file, and an effect of adding data to the stored data, that is, an effect of a data append write operation, can be achieved; and the target data is written with the end position of the target file as the starting position, so there is no conflict between the target data and the data already written in the target file, thereby ensuring data consistency; more importantly, the method adds verification information related to the user identification of the client and the target data in the file to be written, so that the validity of each sub-file written can be verified. For example, when querying, the verification information in the sub-file can be used to determine whether the sub-file is valid, thereby improving the security of data management and preventing malicious sub-files from damaging data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flowchart of a data management method provided by an exemplary embodiment.

[0042] Figure 2 It is a schematic diagram of the directory structure of a target file provided by an exemplary embodiment.

[0043] Figure 3FIG. 1 is a schematic diagram of data to be written in a situation provided by an exemplary embodiment.

[0044] Figure 4 FIG. 1 is a schematic diagram of data to be written in another situation provided by an exemplary embodiment.

[0045] Figure 5 FIG. 1 is a schematic diagram of data to be written in another situation provided by an exemplary embodiment.

[0046] Figure 6 FIG. 1 is a schematic diagram of data to be written in another situation provided by an exemplary embodiment.

[0047] Figure 7 is a flow chart of a data reading process provided by an exemplary embodiment.

[0048] Figure 8 is a schematic diagram of a data reading process provided by an exemplary embodiment.

[0049] Figure 9 It is a schematic diagram of a sub-file provided by an exemplary embodiment without adding verification information.

[0050] Figure 10 This is a schematic diagram of writing a forged sub-file into a target file without verification information added to the sub-file provided by an exemplary embodiment.

[0051] Figure 11 This is a schematic diagram of writing a forged sub-file into a target file for adding verification information to a sub-file provided by an exemplary embodiment.

[0052] Figure 12 It is a structural diagram of a device provided by an exemplary embodiment.

[0053] Figure 13 is a block diagram of a data management device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0054] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The implementations described in the following exemplary embodiments are not intended to represent all implementations consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of one or more embodiments of this specification, as detailed in the appended claims.

[0055] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the method may include more or fewer steps than those described in this specification. In addition, a single step described in this specification may be broken down into multiple steps for description in other embodiments, and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0056] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0057] First, it should be noted that compared to the patent application with publication number CN117130995A filed by the applicant on August 4, 2023, the present disclosure further improves its technical solution and solves the technical problems existing in its technical solution. The specific effects will be described in detail below in conjunction with the specific solution content.

[0058] In the related art, object storage, as a data storage model without a directory structure, can store and manage data in the form of objects, which is very suitable for storing unstructured data such as images, videos, and documents. When storing data through the object storage model, in order to meet the data storage compliance requirements, avoid data tampering, and meet the long-term retention and archiving needs of data, the object storage WORM mode is often used. As a data protection strategy in the object storage model, the object storage WORM mode requires that once an object is successfully created, it is not allowed to be deleted or overwritten within a period of time. Append write operations, as a data writing method that appends data to the end of the storage object, will inevitably involve modifications to the stored data. Therefore, append write operations are generally not allowed in the object storage WORM mode. Even if append writes are simulated in some ways, data security and data consistency cannot be guaranteed.

[0059] Based on the above technical problems mentioned in the background technology, at least one embodiment of this specification provides a data management method, which can realize continuous writing of data in object storage files that do not support append write operations, such as object storage files in WORM mode, and can ensure data consistency and security.

[0060] The method can be executed by a computing device, which can be a terminal device, such as a desktop computer, a portable computer, a tablet computer, a laptop computer, a smart phone, etc., or the computing device can be a server, such as a single server, multiple servers, a server cluster, a cloud computing platform, etc. This specification does not limit the device type of the computing device.

[0061] It should be noted that when a computing device receives a data write request, it can use the data writing method provided in this specification to write the data to be written after the stored data in the object storage file, so as to achieve a continuous data writing effect similar to an append write operation.

[0062] Optionally, object storage files can be used to store various types of unstructured data, such as image data, video data, audio data, document data, report data, log data, time series data, etc. Therefore, the data to be written involved in this specification may be image data, video data, audio data, document data, report data, log data, time series data, etc. This specification does not limit the specific type of data to be written.

[0063] The above is only an illustrative description of the application scenarios of this specification and does not constitute a limitation on the application scenarios of this specification. In more possible implementation methods, the solutions provided in this specification can be applied to more scenarios that require data writing. This specification does not limit the specific application scenarios.

[0064] After introducing the application scenarios of this specification, the specific implementation process of this specification is introduced next.

[0065] Please refer to the attached Figure 1 , which exemplarily shows a flow chart of the data management method, including steps S101 to S103.

[0066] In step S101, a write request sent by a client is received to instruct writing the data to be written into a target file, and target data is determined in the data to be written based on the end position of the data stored in the target file and the requested write position in the write request, wherein the target data is part or all of the data to be written.

[0067] The target file may be an object storage file for storing image data, video data, audio data, document data, report data, log data, time series data, etc., and the data to be written may be video data, audio data, document data, report data, log data, time series data, etc. Optionally, the target file may be other types of files, and the data to be written may be other types of data. This specification does not limit the specific types of the target file and the data to be written.

[0068] It should be noted that the write request can carry the data to be written, the file identifier of the target file to which the data is to be written (such as the file name, file ID, etc.), and the requested write position of the data to be written in the target file (that is, an indication of the position in the target file from which the data to be written is to be written), so that after receiving the write request, the computing device can directly obtain the file identifier of the target file and the requested write position of the data to be written in the target file based on the write request, and thus can determine the target file from multiple data files maintained by the computing device based on the obtained file identifier, and obtain the end position of the stored data in the target file.

[0069] For example, the requested write position has an offset, which represents its offset relative to the starting point of the target file. The end position of the target file is the end position of the data stored therein, and this end position has an offset, which represents its offset relative to the starting point of the target file. Bytes are typically used as the unit of these offsets.

[0070] This step can split the data to be written based on the requested write position and the end position of the data stored in the target file, so as to obtain the data to be written with the end position of the stored data as the starting position (that is, the data whose offset is greater than the offset of the end position of the stored data) as the target data to be actually written to the target file, thereby ensuring that the data written this time does not conflict with the data already written in the target file.

[0071] In step S102, verification information is generated based on the user identification of the client and the data information of the target data, and a sub-file to be written is generated based on the verification information and the target data.

[0072] Exemplarily, the user identification of the client and the data information of the target data may be input into a preset verification function, and the verification value output by the verification function may be used as the verification information.

[0073] For example, the user identifier can be the identifier of the user logged in on the client that sends the write request. After the user logs in based on the user identifier, the client can generate a write request in response to the received operation instruction (that is, the instruction entered by the user through the operation of the client) and send it to the device that executes the method. Therefore, the write request carries the user identifier.

[0074] For example, the data information may be the data path, data length, file data (ie, data content), etc. of the target data.

[0075] For example, the verification function may be a hash function, such as an md5 function, etc. Then the verification value output by the verification function is a hash value.

[0076] Exemplarily, an initial sub-file may be generated based on the target data, and the verification information may be added to the header of the initial sub-file to obtain the sub-file to be written.

[0077] For example, the verification information occupies a fixed number of bytes, such as 8 bytes of verification information at the header of each file to be written. This allows the verification information to be read from the first 8 bytes of each sub-file within the target file when reading it. Furthermore, when reading data within the target file based on an offset from the start of the target file, the data position can be determined without interference from the header of each sub-file, thereby accurately locating the data to be read.

[0078] It should be understood that when the verification information is added to the header of the initial sub-file, the version number of the sub-file can also be added synchronously. In this case, the number of bytes occupied by the header increases, such as 16 bytes.

[0079] For example, the starting position of the target data may be determined as the name of the subfile to be written into the subfile, and the length of the target data may be determined as the data length of the subfile to be written into the subfile.

[0080] In step S103, the sub-file to be written is written into the target file with the end position of the target file as the starting position.

[0081] It should be understood that after writing the sub-file to be written, the end position of the stored data in the target file may be updated.

[0082] By continuously writing data in the target file in the form of sub-files, each new data to be written will be stored in the target file in the form of a sub-file, so that the target file can include multiple sub-files. The data stored in adjacent sub-files is continuous, and a sub-file will not be modified or overwritten after it is written, which meets the requirements of the WORM mode.

[0083] In addition, it should be noted that, precisely because the data written each time will be stored in the target file in the form of a sub-file, the target file can be organized into a virtual folder, which can include multiple sub-files, and each time data is written, a new sub-file will be generated in the virtual folder based on the data to be written this time.

[0084] The data management method provided in the embodiments of this specification receives a write request sent by a client to instruct writing the data to be written into a target file, determines the target data in the data to be written based on the end position of the data stored in the target file and the requested write position in the write request; generates verification information based on the user identification of the client and the data information of the target data, and generates a sub-file to be written based on the verification information and the target data; and writes the sub-file to be written into the target file with the end position of the target file as the starting position. The method writes the target data in the data to be written in the form of a sub-file, so there is no need to modify or overwrite the data already written in the target file, and an effect of adding data to the stored data, that is, an effect of a data append write operation, can be achieved; and the target data is written with the end position of the target file as the starting position, so there is no conflict between the target data and the data already written in the target file, thereby ensuring data consistency; more importantly, the method adds verification information related to the user identification of the client and the target data in the file to be written, so that the validity of each sub-file written can be verified. For example, when querying, the verification information in the sub-file can be used to determine whether the sub-file is valid, thereby improving the security of data management and preventing malicious sub-files from damaging data security.

[0085] In some embodiments of the present disclosure, determining the target data in the data to be written based on the end position of the stored data in the target file and the requested write position in the write request in step S101 may be performed as follows:

[0086] First, the positional relationship between the offset of the requested write position and the offset of the end position of the target file is determined. The positional relationship between the two can have three possible results: the first is that the offset of the requested write position is greater than the offset of the end position of the target file; the second is that the offset of the requested write position is consistent with the offset of the end position of the target file; and the third is that the offset of the requested write position is less than the offset of the end position of the target file.

[0087] Next, based on the positional relationship between the offset of the requested write position and the offset of the end position of the target file, the target data is determined within the data to be written. That is, when the positional relationship between the offset of the requested write position and the offset of the end position of the target file is one of the three possible results, the operation is performed in the following different manners, respectively.

[0088] In a possible implementation, if the offset of the requested write position is greater than the offset of the end position of the target file, there is no need to obtain the target data from the data to be written, that is, the data to be written does not contain the target data.

[0089] It should be noted that if the offset of the requested write position is greater than the offset of the end position of the stored data, it indicates that the data writing process is non-continuous writing, which is an illegal operation and there is no need to write data. Therefore, there is no need to obtain the target data to generate the file to be written.

[0090] For ease of understanding, a specific example is used below to illustrate the situation where the requested write position is located after the end position of the stored data.

[0091] As described in the above embodiment, the target file may include multiple sub-files. Therefore, the target file may be understood as a folder, and the multiple sub-files included in the target file may be organized together according to the directory structure. Figure 2 , Figure 2 This is a schematic diagram of a directory structure of a target file provided by an exemplary embodiment. Figure 2 As shown, the file name of the target file can be start_offset, and the target file can include 5 sub-files, and the file names of these 5 sub-files are 0, 100, 500, 750 and 980 respectively. Assume that the sub-files with file names 0, 100 and 500 have been written with data, the data length of the sub-file with file name 0 is 100, the data length of the sub-file with file name 100 is 400, and the data length of the sub-file with file name 500 is 250. Since the present specification can write data in the form of sub-files each time, the data length of each sub-file in the target file is also the length of the data stored in each sub-file, that is, in the example Figure 2 In the target file shown, the length of the stored data (denoted as written_len) is 750, or in other words, the end position of the stored data is 750 bytes.

[0092] If the offset of the requested write position of the data to be written is 800 and the length of the data to be written is 200, then the requested write position is after the end position of the stored data, which is an illegal non-continuous write operation. Figure 3 , Figure 3This is a schematic diagram of a data writing process provided by an exemplary embodiment. Figure 3 That is, it shows the data writing process when the offset of the requested write position is greater than the offset of the end position of the target file, such as Figure 3 As shown, subfile 0 with a data length of 100, subfile 100 with a data length of 400, and subfile 500 with a data length of 250 are stored in the target file in sequence. The length of the stored data in the target file is written_len=750. According to the data writing rules, this data writing should start writing data from the position of offset=750, but the requested write position of the data to be written (that is, newappend) is offset=800, and the length of the data to be written is len=200. Offset>written_len. This data writing operation is obviously a non-continuous write operation, so an error can be directly reported without writing data.

[0093] In another possible implementation, if the offset of the requested write position is consistent with the offset of the end position of the target file, the data to be written is determined as the target data.

[0094] It should be noted that if the offset of the requested write position is consistent with the offset of the end position of the target file, it indicates that the data writing process is continuous writing and the position where the data is to be written is empty. Therefore, the full amount of data in the data to be written can be directly used as the target data, that is, the full amount of data in the data to be written can be used as the data actually to be written this time.

[0095] Still in Figure 2 For example, if the offset of the requested write position of the data to be written is not 750 and the length of the data to be written is 200, the offset of the requested write position is consistent with the offset of the end position of the stored data. In this case, it is a continuous write operation starting from the end position of the stored data. Figure 4 , Figure 4 is another schematic diagram of a data writing process provided by an exemplary embodiment. Figure 4 That is, it shows the data writing process when the offset of the requested write position is consistent with the offset of the end position of the stored data, such as Figure 4As shown, subfile 0 with a data length of 100, subfile 100 with a data length of 400, and subfile 500 with a data length of 250 are stored in the target file in sequence. The length of the stored data in the target file is written_len=750. According to the data writing rule, this data writing should start writing data from the position of offset=750, and the requested write position of the data to be written (that is, newappend) is also offset=750. The length of the data to be written is len=200, offset=written_len. This data writing operation is a continuous write operation starting from the data stored position. The full amount of data in the data to be written can be directly used as the target data to be written, and it can be done as shown below. Figure 2 In the directory structure shown, name the new subfolder 750.

[0096] In another possible implementation, if the offset of the requested write position is less than the offset of the end position of the data already stored in the target file, the offset of the end position of the data to be written is determined based on the offset of the requested write position and the data length of the data to be written. Furthermore, if the offset of the end position of the data to be written is greater than the offset of the end position of the target file, the data in the data to be written whose offset is greater than the offset of the end position of the target file is determined as the target data. The write request also includes the data length (len) of the data to be written, which may also be expressed in bytes.

[0097] It should be noted that the above embodiments mainly introduce the situations where the requested write position is consistent with the end position of the stored data, and the requested write position is after the end position of the stored data. For these two situations, the end position of the data to be written must also be after the end position of the stored data. Therefore, there is no need to consider the positional relationship between the end position of the data to be written and the end position of the stored data. However, for the situation where the requested write position is before the end position of the stored data, the positional relationship between the end position of the data to be written and the end position of the stored data cannot be determined. Therefore, when the requested write position is before the end position of the stored data, it is necessary to further subdivide multiple situations for consideration.

[0098] In one possible case, if the offset of the end position of the data to be written is less than or equal to the offset of the end position of the stored data, or, there is no need to obtain the target data from the data to be written.

[0099] It should be noted that in this case, if the end position of the data to be written is before the end position of the stored data or the end position of the data to be written is the same as the end position of the stored data, it indicates that all positions where data is to be written this time already store data. And since it is the object storage WORM mode currently, it is not allowed to modify or overwrite the written data, so there is no need to obtain the target data.

[0100] Still taking the data writing in the target file as shown in Figure 2 as an example, if the offset of the requested write position of the data to be written is 750 and the length of the data to be written is 200, then the offset of the requested write position is the same as the offset of the end position of the stored data. At this time, it is a continuous write operation starting from the end position of the stored data. Refer to Figure 5 , Figure 5 which is another schematic diagram of the data writing process provided by an exemplary embodiment, Figure 5 that is, it shows the data writing process in the case where the offset of the requested write position is less than the offset of the end position of the stored data and the offset of the end position of the data to be written is less than the offset of the end position of the stored data. As shown in Figure 5 , sub-file 0 with a data length of 100, sub-file 100 with a data length of 400, and sub-file 500 with a data length of 250 are stored in the target file in sequence. The length of the stored data in the target file is written_len = 750. According to the data writing rule, this data writing should start writing data from the position of offset = 750. However, the offset of the requested write position of the data to be written this time (that is, new append) is offset = 400, and the length of the data to be written is len = 200. Then it can be determined that the end write position is 600. offset < written_len and offset + len < written_len. This data writing operation needs to modify or overwrite the stored data, and this data writing method is not allowed in the object storage WORM mode, so there is no need to obtain the target data.

[0101] The above Figure 5 only takes the case where the end position of the data to be written is before the end position of the stored data as an example for illustration. The case where the end position of the data to be written is the same as the end position of the stored data is the same in principle and will not be elaborated here.

[0102] In another possible case, if the offset of the end position of the data to be written is greater than the offset of the end position of the stored data, then the data in the data to be written whose offset is greater than the offset of the end position of the target file can be determined as the target data.

[0103] For example, determine the data in the data to be written whose offset is not greater than the offset of the end position of the target file, and perform consistency verification with the stored data corresponding to the same offset in the target file; if the result of the consistency verification is passed, then determine the data in the data to be written whose offset is greater than the offset of the end position of the target file as the target data.

[0104] Among them, the offset of the data to be written whose offset is less than the offset of the end position of the target file is from offset to written_len, and the length is written_len-offset, and the stored data corresponding to the same offset in the target file is the data with an offset from offset to written_len in the stored data, and the length is also written_len-offset; the data with an offset greater than the end position of the target file in the data to be written is from written_len to offset+len, and the length is offset+len-written_len.

[0105] Alternatively, if the data to be written, whose offset is less than the end position of the target file, differs from the data already stored in the target file at the same offset, the consistency verification fails and there is no need to retrieve the target data. It should be noted that this situation indicates that if data needs to be written, the stored data must be modified or overwritten, which is not permitted in the object storage WORM mode, so there is no need to retrieve the target data.

[0106] Optionally, if the data in the data to be written whose offset is less than the end position of the target file is consistent with the stored data corresponding to the same offset in the target file, the result of the consistency verification is passed, and the data in the data to be written whose offset is greater than the end position of the target file is determined as the target data. It should be noted that in this case, there is no need to modify or overwrite the stored data, and there is no need to write the data in the data to be written that is located before the end position of the data stored in the target file. Instead, the data in the data to be written that is located after the end position of the data stored in the target file can be directly written to the target file, because the data in the data to be written that is located before the end position of the data stored in the target file has been stored in the target file.

[0107] Still in Figure 2Taking data writing in the target file shown as an example, if the offset of the requested writing position of the data to be written is 700 and the length of the data to be written is 200, the offset of the requested writing position is less than the offset of the end position of the stored data. At this time, data writing needs to be performed before the end position of the stored data. Refer to Figure 6 , Figure 6 which is another schematic diagram of the data writing process provided by an exemplary embodiment. Figure 6 That is, it shows the data writing process when the offset of the requested writing position is less than the offset of the end position of the stored data and the offset of the end writing position of the data to be written is greater than the offset of the end position of the stored data. As Figure 6 shown, sub-file 0 with a data length of 100, sub-file 100 with a data length of 400, and sub-file 500 with a data length of 250 are sequentially stored in the target file. The length of the stored data in the target file is written_len = 750. According to the data writing rule, this data writing should start writing data from the position of offset = 750. However, the offset of the requested writing position of the data to be written this time (that is, new append) is offset = 700, and the length of the data to be written is len = 200. Then, it can be determined that the end writing position is 900. offset < written_len and offset + len > written_len. The positions where data writing needs to be performed during this data writing operation can be divided into two parts: 700 to 750 and 750 to 900. Among them, there is already stored data in the part from 700 to 750. If data needs to be written in this part of the position, the stored data needs to be modified or overwritten, and this data writing method is not allowed in the object storage WORM mode. Therefore, this part of the data does not need to be considered when generating the file to be written; for the part from 750 to 900, this part in the target database is empty and data can be directly written. Therefore, the part from 750 to 900 in the data to be written can be obtained as the target data.

[0108] In the above embodiment, the data to be written is split according to the offset of the requested writing position of the data to be written this time and the offset of the end position of the target file, so as to use the data part after the end position of the stored data in the data to be written as the target data. For the data part at the position where data has been stored and needs to be written, there is no need to repeat the writing, thereby reducing the data processing volume in the data writing process and improving the data writing speed and data writing efficiency.

[0109] In some embodiments of the present disclosure, the method further includes the data reading process as Figure 7 shown, including steps S701 to step S703.

[0110] In step S701, a read request sent by a client for instructing to read data to be read from a target file is received, and at least one target sub-file is determined in the target file based on a data location range in the read request.

[0111] The data position range in the read request includes a read start position and a read end position. The read start position can be represented by an offset in bytes, and the read end position can be represented by an offset in bytes.

[0112] It should be noted that the target file can include multiple subfiles, and the data in different subfiles is written in different batches. In other words, each written data will be stored as a subfile. For example, after receiving a read request, the subfiles in the target file can be listed. The list result includes the file names of all subfiles and the data length of each subfile. Then, according to the list result of all subfiles, the subfile at the starting point of the read can be found first, and then the subfile at the end of the read can be traversed backward until the subfile at the end of the read is found. These two subfiles and all subfiles in between are the target subfiles.

[0113] The above process of determining the target subfile by traversal is only an exemplary implementation. In more possible implementations, other positioning methods can be used to determine the target subfile. For example, a binary search method can be used to determine the target subfile.

[0114] In step S702, for each target subfile, verification information is generated based on the user identification of the client and the data information of the data in the target subfile, and whether the target subfile is valid is determined based on the generated verification information and the verification information in the target subfile.

[0115] Exemplarily, this step can generate verification information in the same manner as when writing data into the target file. For example, the user identification of the client and the data information of the target data can be input into a preset verification function, and the verification value output by the verification function is used as the verification information. For example, the user identification can be the identification of the user logged in on the client that sends the read request. After the user logs in based on the user identification, the client can generate a read request in response to the received operation instruction (i.e., the instruction input by the user through the operation of the client) and send it to the device that executes the method. Therefore, the read request carries the user identification. For example, the data information can be the data path, data length, file data (i.e., data content), etc. of the target data. For example, the verification function can be a hash function, such as the md5 function, etc. Then the verification value output by the verification function is a hash value.

[0116] If the verification information generated in this step is consistent with the verification information in the target subfile, it is determined whether the target subfile is valid; if the verification information generated in this step is inconsistent with the verification information in the target subfile, it is determined whether the target subfile is invalid.

[0117] In step S703, the data to be read is returned to the client based on all valid target sub-files and the data location range in the read request.

[0118] By obtaining valid target sub-files, the required data portion is read from each valid target sub-file, and then the read data portions are combined to obtain the complete data to be read.

[0119] For ease of understanding, the data reading process provided in this specification is described below using a specific example.

[0120] See also Figure 8 , Figure 8 This is a flowchart of a data reading process provided by an exemplary embodiment. Figure 8 As shown, the target data file includes 4 sub-files, among which the file name of the first sub-file is 0 and the data length (or file length) is 100, the file name of the second sub-file is 100 and the data length is 400, the file name of the third sub-file is 500 and the data length is 250, and the file name of the fourth sub-file is 750 and the data length is 150.

[0121] If the data reading instruction is read(offset=300,len=500), that is, to read data of length 500 starting from data storage location 300, the subfile corresponding to the position of offset=300 will be found first, that is, the subfile with file name 100, and then the traversal will continue backward until the subfile corresponding to the position of offset+len=800, that is, the subfile with file name 750. The target subfiles are the subfile with file name 100, the subfile with file name 500, and the subfile with file name 750.

[0122] Then, based on the user identification of the client sending the read request and the data information of each target subfile, verification information of each target subfile is generated; and then the generated verification information is compared with the verification information in the header of the target subfile to see whether they are consistent.

[0123] After comparison, the three target sub-files are all valid target sub-files, so the data to be read can be divided into the following three parts:

[0124] File name: 100, read(offset=200, len=200), that is, from sub-file 100, get the data starting from data storage position 200 and with a data length of 200;

[0125] File name: 500, read(offset=0,len=250), that is, from sub-file 500, get the data starting from data storage position 0 and with a data length of 250;

[0126] File name: 750, read(offset=0,len=50), that is, get the data starting from data storage position 0 and with a data length of 50 from subfile 750;

[0127] By combining these three parts of data, you can get the complete data to be read.

[0128] Compared with the patent application with publication number CN117130995A submitted by the applicant on August 4, 2023, the data management method provided in this specification sets verification information in the header of a sub-file each time it is written, so that when the user reads the data, it can avoid being affected by erroneous data or even illegal malicious data, thereby improving data security, especially when appending to an object storage file in WORM mode.

[0129] For ease of understanding, the above effects are described below with a specific example.

[0130] Please refer to the attached Figure 9 , which exemplarily shows the result of writing data without adding verification information to the sub-file header. Figure 9 Four subfiles have been written into the target file, namely, subfile with file name 0 and length 100, subfile with file name 100 and length 400, subfile with file name 500 and length 250, and subfile with file name 750 and length 150. The length of the target file is 900.

[0131] Please refer to the attached Figure 10 If an attacker forges a sub-file named 900 and writes it into the target file, the user will mistakenly believe that the length of the target file is 1000 when reading the target file, thereby reading the data in the sub-file forged by the attacker; and if the previous data is deleted or updated in this forged data, it will affect the data correctness and cause a data security incident.

[0132] Please refer to the attached Figure 11If the target file is written according to the data provided in this manual, verification information will be added to the headers of the four sub-files in the target file; if an attacker forges a sub-file named 900 and writes it into the target file, when the user reads the target file, the forged sub-file will be eliminated because it cannot pass the verification of the verification information, thereby preventing the user from reading incorrect data and improving data security.

[0133] Figure 12 This is a schematic structural diagram of a device provided by an exemplary embodiment. Figure 12 At the hardware level, the device includes a processor 1202, an internal bus 1204, a network interface 1206, a memory 1208, and a non-volatile memory 1210. Of course, it may also include hardware required for other tasks. One or more embodiments of this specification can be implemented based on software, such as the processor 1202 reading the corresponding computer program from the non-volatile memory 1210 into the memory 1208 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0134] Please refer to Figure 13 , the data management device can be used for Figure 12 The data management device may include:

[0135] The target data module 1301 is configured to receive a write request sent by a client to instruct writing data to be written into a target file, and determine target data within the data to be written based on the end position of the data stored in the target file and the requested write position in the write request, wherein the start position of the target data is the end position of the data stored in the target file;

[0136] The sub-file module 1302 is configured to generate verification information based on the user identification of the client and the data information of the target data, and generate a sub-file to be written based on the verification information and the target data;

[0137] The writing module 1303 is configured to write the sub-file to be written into the target file.

[0138] In a possible embodiment of the present specification, the target data module is configured to, when determining target data within the data to be written based on an end position of the stored data in the target file and a requested write position in the write request,:

[0139] If the requested write position is the end position of the data stored in the target file, the data to be written is determined as the target data.

[0140] In a possible embodiment of the present specification, the write request includes the data length of the data to be written;

[0141] The target data module is configured to, when determining target data within the data to be written based on an end position of the stored data in the target file and a requested write position in the write request,:

[0142] If the requested write position is before the end position of the data stored in the target file, determining the end position of the data to be written based on the requested write position and the data length of the data to be written;

[0143] If the end position of the data to be written is after the end position of the data stored in the target file, the data to be written that is after the end position of the data stored in the target file is determined as the target data.

[0144] In a possible embodiment of the present specification, when the target data module is configured to determine data in the data to be written that is located after the end position of the data stored in the target file as the target data, it is configured to:

[0145] If the data in the data to be written that is located before the end position of the data stored in the target file is the same as the data at the same position in the target file, the data in the data to be written that is located after the end position of the data stored in the target file is determined as the target data.

[0146] In a possible embodiment of the present specification, when the sub-file module is used to generate verification information based on the user identification of the client and the data information of the target data, it is used to:

[0147] The user identification of the client and the data information of the target data are input into a preset verification function, and the verification value output by the verification function is used as verification information.

[0148] In a possible embodiment of the present specification, when the sub-file module is used to generate the sub-file to be written based on the verification information and the target data, it is used to:

[0149] An initial sub-file is generated based on the target data, and the verification information is added to the header of the initial sub-file to obtain a sub-file to be written.

[0150] In a possible embodiment of the present specification, the apparatus further includes a naming module, configured to:

[0151] The starting position of the target data is determined as the subfile name of the subfile to be written, and the length of the target data is determined as the data length of the subfile to be written.

[0152] In a possible embodiment of this specification, the device further includes a reading module, configured to:

[0153] receiving a read request sent by a client for instructing to read data to be read from a target file, and determining at least one target sub-file in the target file based on a data location range in the read request;

[0154] For each target subfile, generating verification information based on the user identifier of the client and data information of the data in the target subfile, and determining whether the target subfile is valid based on the generated verification information and the verification information in the target subfile;

[0155] Based on all valid target sub-files and the data location range in the read request, the data to be read is returned to the client.

[0156] One or more embodiments of this specification also provide a computer program product, comprising a computer program / instruction, which implements the steps of the method provided in the first aspect when executed by a processor.

[0157] One or more embodiments of this specification also provide a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0158] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0159] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0161] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0163] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "an," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0165] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0166] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when..." or "in response to determining."

[0167] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included in the scope of protection of one or more embodiments of this specification.

Claims

1. A data management method, comprising: receiving a write request sent by a client for instructing to write data to be written into a target file, and determining target data within the data to be written based on an end position of the target file and a requested write position in the write request, wherein the target data is part or all of the data to be written; Generating verification information based on the user identification of the client and the data information of the target data, and generating a subfile to be written based on the verification information and the target data; the verification information is obtained by inputting the user identification of the client and the data information of the target data into a preset verification function; the data information includes: data path, data length, and data content of the target data; The end position of the target file is used as the starting position, and the sub-file to be written is written into the target file.

2. The data management method according to claim 1, wherein determining the target data within the data to be written based on the end position of the target file and the requested write position in the write request comprises: If the offset of the requested write position is consistent with the offset of the end position of the target file, the data to be written is determined as the target data, wherein the offset is used to represent the offset relative to the starting position of the target file.

3. The data management method according to claim 1, wherein the write request includes the data length of the data to be written; The determining the target data within the data to be written based on the end position of the target file and the requested write position in the write request includes: If the offset of the requested write position is less than the offset of the end position of the target file, determining the offset of the end position of the data to be written based on the offset of the requested write position and the data length of the data to be written, wherein the offset is used to represent the offset relative to the start position of the target file; If the offset of the end position of the data to be written is greater than the offset of the end position of the target file, the data in the data to be written whose offset is greater than the offset of the end position of the target file is determined as the target data.

4. The data management method according to claim 3, wherein determining the offset data of the data to be written that has an offset greater than the end position of the target file as the target data comprises: Determine data in the data to be written whose offset is not greater than the offset of the end position of the target file, and perform consistency verification with the stored data corresponding to the same offset in the target file; If the result of the consistency verification is passed, the data whose offset in the data to be written is greater than the offset of the end position of the target file is determined as the target data.

5. The data management method according to claim 1, wherein generating the sub-file to be written based on the verification information and the target data comprises: An initial sub-file is generated based on the target data, and the verification information is added to the header of the initial sub-file to obtain a sub-file to be written.

6. The data management method according to claim 5, further comprising: The starting position of the target data is determined as the subfile name of the subfile to be written, and the length of the target data is determined as the data length of the subfile to be written.

7. The data management method according to claim 1, further comprising: receiving a read request sent by a client for instructing to read data to be read from a target file, and determining at least one target sub-file in the target file based on a data location range in the read request; For each target subfile, generating verification information based on the user identifier of the client and data information of the data in the target subfile, and determining whether the target subfile is valid based on the generated verification information and the verification information in the target subfile; Based on all valid target sub-files and the data location range in the read request, the data to be read is returned to the client.

8. A data management device, comprising: a target data module, configured to receive a write request sent by a client for instructing to write data to be written into a target file, and determine target data within the data to be written based on an end position of data stored in the target file and a requested write position in the write request, wherein the target data is part or all of the data to be written; a subfile module, configured to generate verification information based on the user identification of the client and the data information of the target data, and to generate a subfile to be written based on the verification information and the target data; the verification information is obtained by inputting the user identification of the client and the data information of the target data into a preset verification function; the data information includes: the data path, data length, and data content of the target data; The writing module is used to write the sub-file to be written into the target file with the end position of the target file as the starting position.

9. The data management device according to claim 8, further comprising a reading module configured to: receiving a read request sent by a client for instructing to read data to be read from a target file, and determining at least one target sub-file in the target file based on a data location range in the read request; For each target subfile, generating verification information based on the user identifier of the client and data information of the data in the target subfile, and determining whether the target subfile is valid based on the generated verification information and the verification information in the target subfile; Based on all valid target sub-files and the data location range in the read request, the data to be read is returned to the client.

10. A computer program product comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

11. An electronic device comprising: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 7 by running the executable instructions.

12. A computer-readable storage medium having computer instructions stored thereon, wherein when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images