Data storage method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data storage method, device, electronic equipment and storage medium. The data storage method comprises: in response to a file storage request for a first target data file, obtaining the first target data file to be stored in a target storage system, the target storage system being configured with table metadata, the table metadata indicating storage branch information corresponding to the target storage system, the storage branch information comprising a main branch and at least one experimental branch, the main branch being used to mount data files that have been used for model training and the training results of which meet preset requirements, and the experimental branch being used to mount data files that have not been used for model training; based on the table metadata, determining a first target experimental branch corresponding to the file storage request, and newly creating a first target snapshot corresponding to the first target experimental branch; and based on the first target snapshot, storing the first target data file to a corresponding storage location in the target storage system. The embodiments of the present application can improve the data storage efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of data storage technology, and more specifically, to a data storage method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, machine learning has an ever-increasing demand for training data. To facilitate the management of large amounts of data, storage systems are typically used to store the data. For example, user behavior data during application usage is stored for later use.

[0003] Current data storage systems typically use copy-on-write to add new data to the storage system. However, this method requires reading the data already stored in the storage system, merging it with the new data, and then rewriting it back into the storage system, resulting in significant data read and write overhead and thus affecting data storage efficiency. Summary of the Invention

[0004] This disclosure provides at least one data storage method, apparatus, electronic device, and storage medium, which can improve data storage efficiency.

[0005] This disclosure provides a data storage method, including:

[0006] In response to a file storage request for a first target data file, the first target data file to be stored in the target storage system is obtained. The first target data file includes at least one feature data. The target storage system is configured with table metadata, which is used to indicate the storage branch information corresponding to the target storage system. The storage branch information includes a trunk branch and at least one experimental branch. The trunk branch is used to mount data files that have been used for model training and whose training results meet preset requirements. The experimental branch is used to mount data files that have not yet been used for model training. The file storage request carries the first target branch information.

[0007] Based on the table metadata, a first target experimental branch matching the first target branch information carried in the file storage request is determined from the at least one experimental branch, and a first target snapshot corresponding to the first target experimental branch is created. The first target snapshot is used to indicate the target storage path of the first target data file in the target storage system.

[0008] According to the storage path indicated by the first target snapshot, the first target data file is stored in the corresponding storage location in the target storage system.

[0009] In this embodiment, the storage architecture of the target storage system is implemented through a main branch and at least one experimental branch. The main branch is used to mount data files that have been used for model training and whose training results meet preset requirements, while the experimental branch is used to mount data files that have not yet been used for model training. When adding a new data file to the target storage system, it can be added through one of the experimental branches. Compared with the copy-on-write storage method used in related technologies, the new data file is isolated from the data files corresponding to other branches in the target storage system. There is no need to copy and read data in other branches. This reduces the overhead of data processing and thus helps to improve the efficiency of data storage.

[0010] In one possible implementation, the first target snapshot corresponds to target file details, which include a target storage path allocated for the first target data file in the target storage system; storing the first target data file to the corresponding storage location in the target storage system according to the storage path indicated by the first target snapshot includes:

[0011] According to the target storage path included in the target file details corresponding to the first target snapshot, the first target data file is stored in the storage location corresponding to the target storage path.

[0012] In this embodiment of the disclosure, since the target file details include the target storage path allocated to the first target data file in the target storage system, the accuracy of storing the first target data file can be improved by following the target storage path included in the target file details.

[0013] In one possible implementation, the method further includes:

[0014] Based on the first target data file, the target model is trained to obtain training results, and it is determined whether the training results meet the preset requirements.

[0015] If the training results meet the preset requirements, a second target snapshot corresponding to the main branch is created, and a correspondence indication relationship is established between the second target snapshot and the first target data file and the existing data file corresponding to the main branch.

[0016] In this embodiment of the disclosure, when the training results meet the preset requirements, a correspondence indication relationship is established between the second target snapshot and the first target data file and the existing data files corresponding to the trunk branch. In this way, it can be ensured that the data files corresponding to the trunk branch are all conducive to model training, thereby improving the quality of the data files corresponding to the trunk branch.

[0017] In one possible implementation, establishing the correspondence between the second target snapshot and the first target data file and the existing data files corresponding to the trunk branch includes:

[0018] Determine whether there are contradictory feature data between the first target data file and the existing data files corresponding to the main branch;

[0019] If there are no conflicting feature data between the first target data file and the existing data file corresponding to the trunk branch, establish a correspondence between the second target snapshot and the first target data file and the existing data file corresponding to the trunk branch.

[0020] In this embodiment, if there are no conflicting feature data between the first target data file and the existing data files corresponding to the trunk branch, a correspondence indication relationship is established between the second target snapshot and the first target data file and the existing data files corresponding to the trunk branch. This avoids conflicting feature data in the data files corresponding to the trunk branch, thereby improving the quality of data storage. Furthermore, during subsequent read-time merging, conflicts in data merging are avoided, preventing them from affecting data reading efficiency.

[0021] In one possible implementation, if the feature data in the first target data file is data obtained by modifying the feature data in the data file corresponding to the main branch based on a preset modification logic, the method further includes:

[0022] If the data file corresponding to the main branch is changed, the first target data file after the change is obtained based on the latest changed data file corresponding to the main branch and the preset change logic;

[0023] Create a third snapshot corresponding to the first target experimental branch, and establish a correspondence between the third snapshot and the modified first target data file.

[0024] In this embodiment of the disclosure, if the feature data in the first target data file is obtained by modifying the feature data in the data file corresponding to the main branch based on a preset modification logic, and the data file corresponding to the main branch is modified, the data file corresponding to the first target experimental branch can be updated simultaneously. This ensures that the feature data executed in the data file corresponding to the first target experimental branch is up-to-date. In this way, when conducting feature data research based on the first target experimental branch, the timeliness of the feature data can be ensured, thereby improving the accuracy of the feature research.

[0025] In one possible implementation, the method further includes:

[0026] For each snapshot, determine whether the retention time of each snapshot exceeds the preset time;

[0027] If the retention time of the snapshot exceeds the preset time and the snapshot is not connected to any branch, the snapshot will be deleted.

[0028] In this embodiment of the disclosure, if the retention time of a snapshot exceeds a preset time and the snapshot is not connected by any branch, it indicates that the snapshot is not used. Therefore, the snapshot can be deleted from the table metadata. This can improve the simplicity of the table metadata, thereby saving search time when performing branch search based on the table metadata and improving the efficiency of branch determination.

[0029] In one possible implementation, the method further includes:

[0030] For each branch, if there are file details on that branch that do not correspond to any of the snapshots, delete those file details that do not correspond to any of the snapshots from the table metadata; and / or,

[0031] For each branch, if there is a data file on the branch that does not have a corresponding indication relationship with any of the snapshots, the data file that does not have a corresponding indication relationship is deleted.

[0032] In this embodiment of the disclosure, file details that do not have corresponding relationships in the table metadata can also be deleted. This simplifies the content of the table metadata, thereby saving search time and improving the efficiency of branch determination when performing branch searches based on the table metadata. Deleting data files that do not have corresponding indication relationships can increase the effective storage space of the target storage system.

[0033] In one possible implementation, the method further includes:

[0034] In response to a file read request, based on the second target branch information carried in the file read request, a second target experimental branch is determined from the table metadata, and at least one second target data file that matches the second target branch information carried in the file read request is determined from the data file corresponding to the second target experimental branch.

[0035] The at least one second target data file is merged during read to obtain a merged data file, and feature data is read based on the merged data file.

[0036] In this embodiment of the disclosure, when reading a data file, it can be read through experimental branches. This can achieve read isolation of feature data, that is, unnecessary data does not need to be read. This can reduce the overhead of data reading and improve the efficiency of data reading.

[0037] In one possible implementation, the feature data in the data file also includes identification information; the read-time merging of the at least one second target data file to obtain a merged data file includes:

[0038] Based on the identification information of each feature data in the at least one second target data file, feature data with the same identification information are merged to obtain the merged data file.

[0039] In this embodiment of the disclosure, the feature data are merged based on the identification information of each feature data, thereby improving the accuracy of data merging.

[0040] This disclosure provides a data storage device, including:

[0041] The data acquisition module is used to respond to a file storage request for a first target data file, acquire the first target data file to be stored in the target storage system, the first target data file including at least one feature data; the target storage system is configured with table data, the table data is used to indicate the storage branch information corresponding to the target storage system, the storage branch information includes a main branch and at least one experimental branch, the main branch is used to mount data files that have been used for model training and whose training results meet preset requirements, the experimental branch is used to mount data files that have not yet been used for model training, and the file storage request carries the first target branch information;

[0042] The branch determination module is used to determine, based on the table metadata, a first target experimental branch that matches the first target branch information carried in the file storage request from the at least one experimental branch, and to create a first target snapshot corresponding to the first target experimental branch. The first target snapshot is used to indicate the target storage path of the first target data file in the target storage system.

[0043] The data storage module is used to store the first target data file in the corresponding storage location of the target storage system according to the storage path indicated by the first target snapshot.

[0044] In one possible implementation, the first target snapshot corresponds to target file details, and the target file details include a target storage path allocated for the first target data file in the target storage system; the data storage module is specifically used for:

[0045] According to the target storage path included in the target file details corresponding to the first target snapshot, the first target data file is stored in the storage location corresponding to the target storage path.

[0046] In one possible implementation, the data storage device further includes a relationship determination module, the relationship determination module being used to:

[0047] Based on the first target data file, the target model is trained to obtain training results, and it is determined whether the training results meet the preset requirements.

[0048] If the training results meet the preset requirements, a second target snapshot corresponding to the main branch is created, and a correspondence indication relationship is established between the second target snapshot and the first target data file and the existing data file corresponding to the main branch.

[0049] In one possible implementation, the relationship determination module is specifically used for:

[0050] Determine whether there are contradictory feature data between the first target data file and the existing data files corresponding to the main branch;

[0051] If there are no conflicting feature data between the first target data file and the existing data file corresponding to the trunk branch, establish a correspondence between the second target snapshot and the first target data file and the existing data file corresponding to the trunk branch.

[0052] In one possible implementation, the relationship determination module is further configured to:

[0053] The feature data in the first target data file is obtained by changing the feature data in the data file corresponding to the trunk branch based on a preset change logic. In the case that the data file corresponding to the trunk branch is changed, the changed first target data file is obtained based on the latest changed data file corresponding to the trunk branch and the preset change logic.

[0054] Create a third snapshot corresponding to the first target experimental branch, and establish a correspondence between the third snapshot and the modified first target data file.

[0055] In one possible implementation, the data storage device further includes an information deletion module, the information deletion module being used for:

[0056] For each snapshot, determine whether the retention time of each snapshot exceeds the preset time;

[0057] If the retention time of the snapshot exceeds the preset time and the snapshot is not connected to any branch, the snapshot will be deleted.

[0058] In one possible implementation, the information deletion module is further configured to:

[0059] For each branch, if there are file details on that branch that do not correspond to any of the snapshots, delete those file details that do not correspond to any of the snapshots from the table metadata; and / or,

[0060] For each branch, if there is a data file on the branch that does not have a corresponding indication relationship with any of the snapshots, the data file that does not have a corresponding indication relationship is deleted.

[0061] In one possible implementation, the data storage device further includes a data reading module.

[0062] The branch determination module is also used for:

[0063] In response to a file read request, based on the second target branch information carried in the file read request, a second target experimental branch is determined from the table metadata, and at least one second target data file that matches the second target branch information carried in the file read request is determined from the data file corresponding to the second target experimental branch.

[0064] The data reading module is used for:

[0065] The at least one second target data file is merged during read to obtain a merged data file, and feature data is read based on the merged data file.

[0066] This disclosure provides an electronic device, including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the data storage method described in any of the above embodiments is executed.

[0067] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the data storage method described in any of the above embodiments.

[0068] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0069] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0070] Figure 1 A flowchart of a data storage method provided by some embodiments of this disclosure is shown;

[0071] Figure 2 A schematic diagram of a storage architecture for a target storage system provided by some embodiments of the present disclosure is shown;

[0072] Figure 3 A flowchart is shown illustrating a method for establishing a correspondence between a first target data file and a trunk branch, as provided in some embodiments of this disclosure.

[0073] Figure 4 This illustration shows a schematic diagram of creating a second target snapshot corresponding to a trunk branch, provided by some embodiments of this disclosure;

[0074] Figure 5 A schematic diagram is shown illustrating a new snapshot corresponding to an experimental branch, provided by some embodiments of this disclosure;

[0075] Figure 6 A flowchart of a data reading method provided by some embodiments of this disclosure is shown;

[0076] Figure 7 The diagram shows a structural schematic of a data storage device provided in some embodiments of this disclosure;

[0077] Figure 8 This diagram illustrates the structure of another data storage device provided in some embodiments of the present disclosure;

[0078] Figure 9 A schematic diagram of an electronic device provided in some embodiments of the present disclosure is shown. Detailed Implementation

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0080] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0081] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0082] Research has found that current data storage systems typically use copy-on-write to add new data to the storage system. However, this method requires reading the data already stored in the storage system, merging it with the new data, and then rewriting it back into the storage system, resulting in high data read and write overhead and thus low data storage efficiency.

[0083] Based on the above research, this disclosure provides a data storage method. First, in response to a file storage request for a first target data file, a first target data file to be stored in a target storage system is obtained. The target data file includes at least one feature data. The target storage system is configured with table metadata, which indicates storage branch information corresponding to the target storage system. The storage branch information includes a trunk branch and at least one experimental branch. The trunk branch is used to mount data files that have been used for model training and whose training results meet preset requirements. The experimental branch is used to mount data files that have not yet been used for model training. The file storage request carries target branch information. Then, based on the table metadata, a first target experimental branch corresponding to the file storage request is determined from the at least one experimental branch, and a first target snapshot corresponding to the first target experimental branch is created. The first target snapshot indicates the target storage path of the first target data file in the target storage system. Finally, based on the first target snapshot, the first target data file is stored in the corresponding storage location of the target storage system. In this way, the first target data file can be written to the system based on the experimental branch without affecting the data files mounted on the trunk branch. Compared with the copy-on-write method used in related technologies, there is no need to operate on the data files mounted on the trunk branch, thereby improving the efficiency of data storage.

[0084] To facilitate understanding of this embodiment, the executing entity of the data storage method provided in this disclosure will first be described in detail. The executing entity of the data storage method provided in this disclosure is an electronic device. In this embodiment, the electronic device is a server, which can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In other embodiments, the electronic device can also be a terminal device. This terminal device can be a mobile device, user terminal, handheld device, computing device, or wearable device, etc. Furthermore, this data storage method can also be implemented by a processor calling computer-readable instructions stored in memory.

[0085] The data storage method provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. See also Figure 1 The diagram shown is a flowchart of a data storage method provided in an embodiment of this disclosure. The data storage method includes the following steps S101 to S102:

[0086] S101, in response to a file storage request for a first target data file, obtain the first target data file to be stored in the target storage system. The first target data file includes at least one feature data. The target storage system is configured with table metadata, which is used to indicate the storage branch information corresponding to the target storage system. The storage branch information includes a trunk branch and at least one experimental branch. The trunk branch is used to mount data files that have been used for model training and whose training results meet preset requirements. The experimental branch is used to mount data files that have not yet been used for model training. The file storage request carries the first target branch information.

[0087] First, the storage architecture of the target storage system will be introduced. Specifically, the target storage system relies on a metadata architecture to store data files; please refer to [link to relevant documentation]. Figure 2 This is a schematic diagram of the storage architecture of a target storage system provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, the target storage system is configured with table metadata, which includes storage branch information corresponding to the target storage system. This storage branch information includes a main branch and at least one experimental branch (or research branch), for example... Figure 2 The experimental branches 1 and 2 shown are as follows: the main branch is used to mount data files that have been used for model training and whose training results meet the preset requirements (e.g., data files 1, 2, and 3), while the experimental branches are used to mount data files that have not yet been used for model training (e.g., data file 5). It should be understood that the target model can be trained based on the data files to obtain training results. If the training results meet the preset requirements, the data files can be mounted to the main branch. For some data files (e.g., newly added files), they can be mounted to the experimental branches.

[0088] In this embodiment of the disclosure, the file storage request carries first target branch information, which may be the identification information of one of the experimental branches (for example, the identification information of experimental branch 1 may be A1, the identification information of experimental branch 2 may be A2, or it may be other format identification information, which is not limited), so as to determine the first target experimental branch corresponding to storing the first target data file in subsequent steps.

[0089] Optionally, the first target data file may include user behavior data during the use of the target application. This behavior data includes sample data and / or sample label data. The sample data and sample label data are used to train the target model. The trained model can be used to predict the user's target usage behavior and, based on this target usage behavior, display content matching the target usage behavior to the user. In other embodiments, the data files stored in the target storage system may also be user data that needs to be retained based on General Data Protection Regulation (GDPR) compliance requirements.

[0090] It should be noted that if the model performance is improved after training with the newly added sample data and sample label data, the newly added sample data and sample label data will be retained; if the model performance is reduced, the newly added sample data and sample label data will be deleted, thereby saving corresponding storage space.

[0091] For example, the target application can be a short video application, a news discovery application, or a shopping application, etc., without specific limitations. The sample data refers to data used to characterize user behavior, and the sample tag data refers to data used to characterize the results of user behavior. The content of the sample data and the content of the sample tag data differ for different types of applications.

[0092] For example, for news applications, the sample data could be data such as the content and duration of news browsing, and correspondingly, the sample tag data could be data on whether a news item was saved, liked, or disliked. For shopping applications, the sample data could be consumption data on whether a user searched for or watched a certain type of shopping content, and correspondingly, the sample tag data could be data on whether the user placed an order or added a product to their shopping cart. For short video applications, the sample data could be data on the duration or frequency of a user watching a certain type or a specific video, and the sample tag data could be data on whether the user was interested in the video.

[0093] It should be noted that the sample data typically occurs earlier than the sample label data. Furthermore, the examples of sample data and sample label data above are merely illustrative; in other embodiments, the sample data and sample label data can be determined based on actual application requirements for different types of applications. Additionally, the classification of the target applications described above is also just an example; some applications may correspond to several different categories. For instance, some short video applications also have shopping functions and can be classified as shopping applications.

[0094] In some implementations, the multiple feature data included in the first target data file have a preset correlation relationship. For example, the multiple feature data may be usage behavior data of different users within the same time range.

[0095] See Figure 2 It is understood that each branch corresponds to at least one snapshot, and each snapshot corresponds to at least one data file. For example, snapshot 1 of the trunk branch corresponds to file 1, file 2, and file 3. Each snapshot also corresponds to a file detail, which includes the storage path of each data file corresponding to that snapshot. In some embodiments, the file detail may also include file attribute information of each data file. The file attribute information may include file type, file statistics (e.g., the number of feature data included in the data file or the type of feature data included in the data file), and storage time, etc., which are not limited here.

[0096] S102, based on the table metadata, determine a first target experimental branch from the at least one experimental branch that matches the first target branch information carried in the file storage request, and create a first target snapshot corresponding to the first target experimental branch. The first target snapshot is used to indicate the target storage path of the first target data file in the target storage system.

[0097] It is understood that since the file storage request carries the first target branch information, the first target experimental branch corresponding to the first target branch information can be determined based on the storage branch information in the table metadata. For example, if the file storage request carries the target branch information as experimental branch 1, then experimental branch 1 in the target storage system is determined as the first target experimental branch, wherein the first target branch information carried in the file storage request is determined by the user.

[0098] For example, a user can input first target branch information (e.g., experimental branch 1) in a preset file storage interface, and then generate a file storage request based on the first target branch information, so that the file storage request carries the first target branch information.

[0099] Since the first target data file needs to be stored in the target storage system, a first target snapshot can be created after the first target branch is determined. Since the first target snapshot is used to indicate the target storage path of the first target data file in the target storage system, the storage path corresponding to the first target data file can be determined.

[0100] S103, according to the storage path indicated by the first target snapshot, store the first target data file in the corresponding storage location of the target storage system.

[0101] Since the first target snapshot is used to indicate the target storage path of the first target data file in the target storage system, the first target data file can be stored in the storage location corresponding to the target storage path in the target storage system based on the first target snapshot.

[0102] Since each snapshot corresponds to a file detail, and the file detail includes the target storage path allocated to the first target data file, in some implementations, the target data file can be stored in a storage location corresponding to the target storage path based on the target file detail corresponding to the first target snapshot.

[0103] In this embodiment of the disclosure, since the storage architecture of the target storage system is implemented through a main branch and at least one experimental branch, when adding a new data file to the target storage system, it can be added through one of the experimental branches. Compared with the copy-on-write storage method used in related technologies, the new data file is isolated from the data files corresponding to other branches in the target storage system, and no operation is required on other data. In this way, the overhead of data processing can be reduced, which is conducive to improving the efficiency of data storage.

[0104] Optionally, for each snapshot, if the data file corresponding to that snapshot is changed (e.g., a data file is added or deleted), a new snapshot will be generated, and this new snapshot will correspond to the changed data file. The following is based on... Figure 2 The storage architecture shown is explained in detail for different change scenarios (adding or deleting data files).

[0105] (1) Taking the addition of a new data file as an example, the process of snapshot change is introduced.

[0106] If the first target branch information carried in the file storage request is experimental branch 2, a first target snapshot (e.g., snapshot 3) corresponding to experimental branch 2 can be created. Snapshot 3 indicates the target storage path in the target storage system. After the target storage path is determined, the first target data file can be stored in the storage location corresponding to the target storage path. Thus, when writing the first target data file, the first target data file can be isolated from other data files in the storage system.

[0107] (2) Taking the deletion of data files as an example, the process of snapshot modification is introduced.

[0108] For example, please see again Figure 2If a user needs to delete some feature data in data file 3 corresponding to experimental branch 1, according to the architecture in the diagram, data file 3 also has a corresponding indication relationship with snapshot 1 corresponding to the trunk branch. Therefore, data file 3 cannot be deleted directly. Instead, a new data file 6 can be created, and the data other than the deleted feature data in data file 3 can be stored in data file 6. A file detail 5 can also be created, which is the file detail of data file 6. At the same time, a snapshot 4 corresponding to experimental branch 1 can be added, and a correspondence relationship can be established between snapshot 4 and file detail 3 and file detail 5. In this way, some feature data in data file 3 can be deleted without affecting the data files corresponding to other branches, thereby achieving data isolation between experimental branches and trunk branches during the data writing process.

[0109] In some implementations, for each experimental branch, if the data file mounted on that branch has already been used for model training and the training results meet preset requirements, a corresponding instruction relationship can be established between that data file and the main branch. For details, please refer to [link to relevant documentation]. Figure 3 The diagram shows a flowchart of a method for establishing a correspondence between a first target data file and a main branch, according to an embodiment of this disclosure. The method includes the following steps S301 to S302:

[0110] S301, Based on the first target data file, train the target model to obtain training results, and determine whether the training results meet the preset requirements.

[0111] Specifically, feature data from the first target data file can be input into the target model to train it, obtaining the training results. The training results can then be used to determine if they meet preset requirements. Optionally, the training results may include model performance metrics. By comparing these metrics with preset performance metrics, it can be determined whether the training results meet the preset requirements. In other implementations, other methods can be used for judgment, such as allowing users to manually review the training results to determine if they meet the preset requirements. This is not a limitation.

[0112] S302, if the training result meets the preset requirements, create a second target snapshot corresponding to the trunk branch, and establish a correspondence between the second target snapshot and the first target data file and the existing data file corresponding to the trunk branch.

[0113] For example, please also see Figure 2 and Figure 4 , Figure 4This is a schematic diagram of creating a second target snapshot corresponding to a main branch, provided by an embodiment of the present disclosure. The main branch includes snapshot 4 and snapshot 1. If the training result corresponding to the data file corresponding to snapshot 3 of experimental branch 2 meets the preset requirements, a second target snapshot (i.e., snapshot 5) corresponding to the main branch can be created. A correspondence indication relationship is established between snapshot 5 and the data file (data file 5) corresponding to snapshot 3 of experimental branch 2 and the existing data files (e.g., data files 1, 2, and 3) corresponding to the main branch. In this way, by changing the correspondence indication relationship between snapshots and data files, without the need for data copying, it is beneficial to reduce the data processing overhead.

[0114] Optionally, when establishing the correspondence between the second target snapshot and the first target data file and the existing data files corresponding to the trunk branch, it can be first determined whether there is contradictory feature data between the first target data file and the existing data files corresponding to the trunk branch. If there is no contradictory feature data between the first target data file and the existing data files corresponding to the trunk branch, the correspondence between the second target snapshot and the first target data file and the existing data files corresponding to the trunk branch can be established.

[0115] For example, if the first target data file and the existing data file corresponding to the trunk branch have both modified the same feature data but the modifications are different, then it is considered that there is a contradiction between the first target data file and the existing data file corresponding to the trunk branch, and subsequent steps cannot be performed. If the first target data file and the existing data file corresponding to the trunk branch have added different feature data, then it can be considered that there is no contradiction, and subsequent steps can be performed.

[0116] It should be noted that if multiple experimental branches make different modifications to the same feature data, for example, it is considered that there is a contradiction between the data files corresponding to these multiple experimental branches, and subsequent steps cannot be performed. If the modifications made by multiple experimental branches do not affect each other, it is considered that there is no contradiction. In this case, a new snapshot corresponding to the main branch can be created, and a correspondence between the new snapshot and the data files under the multiple experimental branches and the existing data files corresponding to the main branch can be established.

[0117] In some implementations, the feature data in the first target data file can be data obtained by performing modification operations (e.g., calculation operations) on the feature data in the data file corresponding to the trunk branch according to a preset modification logic. When the data file corresponding to the trunk branch is modified (e.g., new data is written), the modified first target data file is obtained based on the latest modified data file corresponding to the trunk branch and the preset modification logic. At this time, since the first target data file has been modified, it is necessary to create a third snapshot corresponding to the first target experimental branch and establish a corresponding indication relationship between the third snapshot and the modified first target data file. In this way, feature surveys can be conducted based on the latest data file corresponding to the experimental branch.

[0118] For example, please see Figure 5 The diagram shown is a schematic representation of a new snapshot corresponding to an experimental branch, provided in an embodiment of this disclosure. Figure 5 As shown, the snapshots corresponding to the trunk branch include snapshot 4, snapshot 1, snapshot 5, and snapshot 6. Experimental branch 2 corresponds to snapshot 3, and the data file corresponding to snapshot 3 is obtained by logically modifying the data file corresponding to snapshot 1. As snapshot 1 to snapshot 6 corresponding to the trunk branch changes (the change from snapshot 1 to snapshot 6 means that the data file corresponding to the snapshot changes), logical changes can be made based on the new data file corresponding to the trunk branch to obtain a new data file corresponding to experimental branch 2. A new snapshot 7 is created on the experimental branch, and a correspondence between snapshot 7 and the new data file corresponding to experimental branch 2 is established. In this way, it can be ensured that the data file corresponding to experimental branch 2 is the latest data file.

[0119] In this embodiment of the disclosure, the target storage system also supports the ability to perform periodic cleanup. If the data file corresponding to an experimental branch is a data file of an experiment failure, the experimental branch can be deleted from the table metadata. For example, please refer again to... Figure 2 If the data file 5 corresponding to experimental branch 2 is a data file of experimental failure, then experimental branch 2 can be deleted from the table metadata. In this way, the data can be deleted with low data overhead and will not affect other data.

[0120] Optionally, for each snapshot, it can be determined whether the retention time of each snapshot exceeds a preset time. If the retention time of the snapshot exceeds the preset time and the snapshot does not correspond to any branch, the snapshot can be deleted from the table metadata.

[0121] Alternatively, for each branch, if there exists a file detail on that branch that does not correspond to any of the snapshots, then that file detail can be deleted from the table metadata. Similarly, for each data file, if there is no corresponding data file for any of the snapshots corresponding to each branch, then that data file should be deleted.

[0122] It should be understood that after data storage, when training the model, it is necessary to read the training data required for training from the target storage system. Therefore, see [link to relevant documentation]. Figure 6 The diagram shows a flowchart of a data reading method provided in an embodiment of this disclosure. The reading method includes the following steps S601 to S602:

[0123] S601, in response to a file read request, based on the second target branch information carried in the file read request, a second target experimental branch is determined from the table metadata, and at least one second target data file that matches the second target branch information carried in the file read request is determined from the data file corresponding to the second target experimental branch.

[0124] The second target branch information carried in the file read request can be identification information used to indicate one of the branches. For example, if the identification information of experimental branch 1 is ID1, then the second target branch information can be ID1. In this way, experimental branch 1 can be determined from the table metadata based on the identification information ID1.

[0125] It is understood that the target storage system is based on a table-based metadata architecture. Therefore, when reading data, it is also necessary to read data according to the architecture. For example, please refer again. Figure 2 If the file read request indicates access to the data file corresponding to Experiment Branch 1, then when reading the data file, the data reading can be completed based on the following: Experiment Branch 1 -> Snapshot 2 -> File Details 1, 2, 3 -> Data Files 1, 2, 3, 4. From... Figure 2 As can be seen, since the branches are isolated from each other, when reading data files 1, 2, and 3 corresponding to the main branch, they can be read directly without copying them to the storage space corresponding to experimental branch 1. This saves on data file copying overhead. Furthermore, since the experimental branches (e.g., experimental branch 1 and experimental branch 2) are isolated from each other, when reading the data file corresponding to experimental branch 1, it is not necessary to read the data file corresponding to experimental branch 2. This achieves data reading isolation.

[0126] S603, the at least one second target data file is merged during read to obtain a merged data file, and feature data is read based on the merged data file.

[0127] Optionally, when merging multiple second target data files during read operations, a multi-way merge algorithm can be used for merging and reading.

[0128] For example, for data files 1, 2, 3, and 4, a multi-way merge algorithm can be used to merge and read the data files. In this way, during data reading, it is not necessary to load all files in the target storage system into memory at the same time, thus avoiding performance degradation caused by data overflowing to the disk.

[0129] In some implementations, the at least one second target data file can be sorted based on the primary key of each second target data file, and a multi-way merge method can be used to read data from the sorted at least one second target data file.

[0130] Specifically, during the process of reading data from at least one sorted second target data file using a multi-way merge method, it is necessary to align and merge the various feature data. In this embodiment, feature data with the same identification information can be merged according to the identification information of each feature data in the multiple second target data files to obtain the merged data file. The identification information can be a row primary key or other identification information that can uniquely identify each feature data, except for the row primary key. No limitation is made here.

[0131] Those skilled in the art will understand that, in the methods described above in the specific embodiments, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic. Furthermore, the various embodiments can be combined with each other to form different technical solutions.

[0132] Based on the same technical concept, this disclosure also provides a data storage device corresponding to the data storage method. Since the principle of the device in this disclosure for solving the problem is similar to the data storage method described above in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0133] Reference Figure 7 The diagram shown is a schematic representation of a data storage device 700 provided in an embodiment of this disclosure. The data storage device includes:

[0134] The data acquisition module 701 is used to acquire the first target data file to be stored in the target storage system in response to a file storage request for the first target data file. The first target data file includes at least one feature data. The target storage system is configured with table data, which is used to indicate the storage branch information corresponding to the target storage system. The storage branch information includes a main branch and at least one experimental branch. The main branch is used to mount data files that have been used for model training and whose training results meet preset requirements. The experimental branch is used to mount data files that have not yet been used for model training. The file storage request carries the first target branch information.

[0135] The branch determination module 702 is used to determine, based on the table metadata, a first target experimental branch that matches the first target branch information carried in the file storage request from the at least one experimental branch, and to create a first target snapshot corresponding to the first target experimental branch. The first target snapshot is used to indicate the target storage path of the first target data file in the target storage system.

[0136] The data storage module 703 is used to store the first target data file to the corresponding storage location in the target storage system according to the storage path indicated by the first target snapshot.

[0137] In one possible implementation, the first target snapshot corresponds to target file details, the target file details including the target storage path allocated for the first target data file in the target storage system; the data storage module 703 is specifically used for:

[0138] According to the target storage path included in the target file details corresponding to the first target snapshot, the first target data file is stored in the storage location corresponding to the target storage path.

[0139] Please see Figure 8 This is a schematic diagram of another data storage device provided in an embodiment of the present disclosure. The data storage device further includes a relationship determination module 704, which is used for:

[0140] Based on the first target data file, the target model is trained to obtain training results, and it is determined whether the training results meet the preset requirements.

[0141] If the training results meet the preset requirements, a second target snapshot corresponding to the main branch is created, and a correspondence indication relationship is established between the second target snapshot and the first target data file and the existing data file corresponding to the main branch.

[0142] In one possible implementation, the relationship determination module 704 is specifically used for:

[0143] Determine whether there are contradictory feature data between the first target data file and the existing data files corresponding to the main branch;

[0144] If there are no conflicting feature data between the first target data file and the existing data file corresponding to the trunk branch, establish a correspondence between the second target snapshot and the first target data file and the existing data file corresponding to the trunk branch.

[0145] In one possible implementation, the relationship determination module 704 is further configured to:

[0146] The feature data in the first target data file is obtained by changing the feature data in the data file corresponding to the trunk branch based on a preset change logic. In the case that the data file corresponding to the trunk branch is changed, the changed first target data file is obtained based on the latest changed data file corresponding to the trunk branch and the preset change logic.

[0147] Create a third snapshot corresponding to the first target experimental branch, and establish a correspondence between the third snapshot and the modified first target data file.

[0148] In one possible implementation, the data storage device further includes an information deletion module 705, which is used for:

[0149] For each snapshot, determine whether the retention time of each snapshot exceeds the preset time;

[0150] If the retention time of the snapshot exceeds the preset time and the snapshot is not connected to any branch, the snapshot will be deleted.

[0151] In one possible implementation, the information deletion module 705 is further configured to:

[0152] For each branch, if there are file details on that branch that do not correspond to any of the snapshots, delete those file details that do not correspond to any of the snapshots from the table metadata; and / or,

[0153] For each branch, if there is a data file on the branch that does not have a corresponding indication relationship with any of the snapshots, the data file that does not have a corresponding indication relationship is deleted.

[0154] In one possible implementation, the data storage device further includes a data reading module 706.

[0155] The branch determination module 702 is also used for:

[0156] In response to a file read request, based on the second target branch information carried in the file read request, a second target experimental branch is determined from the table metadata, and at least one second target data file that matches the second target branch information carried in the file read request is determined from the data file corresponding to the second target experimental branch.

[0157] The data reading module 706 is used for:

[0158] The at least one second target data file is merged during read to obtain a merged data file, and feature data is read based on the merged data file.

[0159] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0160] Based on the same technical concept, this disclosure also provides an electronic device. (See also...) Figure 9 The diagram shows the structure of an electronic device 900 provided in this embodiment of the present disclosure, including a processor 901, a memory 902, and a bus 903. The memory 902 stores execution instructions and includes a main memory 9021 and an external memory 9022. The main memory 9021, also called internal memory, is used to temporarily store computational data in the processor 901, as well as data exchanged with external memory 9022 such as a hard disk. The processor 901 exchanges data with the external memory 9022 through the main memory 9021.

[0161] In this embodiment, the memory 902 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 901. That is, when the electronic device 900 is running, the processor 901 communicates with the memory 902 through the bus 903, so that the processor 901 executes the application code stored in the memory 902, and then executes the method described in any of the foregoing embodiments.

[0162] The memory 902 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0163] Processor 901 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0164] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 900. In other embodiments of this application, the electronic device 900 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0165] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data storage method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0166] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the data storage method in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0167] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0168] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0169] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0170] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0171] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0172] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data storage method, characterized in that, The method includes: In response to a file storage request for a first target data file, the first target data file to be stored in the target storage system is obtained. The first target data file includes at least one feature data. The target storage system is configured with table metadata, which is used to indicate the storage branch information corresponding to the target storage system. The storage branch information includes a trunk branch and at least one experimental branch. The trunk branch is used to mount data files that have been used for model training and whose training results meet preset requirements. The at least one experimental branch is used to mount data files that have not yet been used for model training. The trunk branch and the at least one experimental branch are isolated from each other. The file storage request carries the first target branch information. Based on the table metadata, a first target experimental branch matching the first target branch information carried in the file storage request is determined from the at least one experimental branch, and a first target snapshot corresponding to the first target experimental branch is created. The first target snapshot is used to indicate the target storage path of the first target data file in the target storage system. According to the storage path indicated by the first target snapshot, the first target data file is stored in the corresponding storage location of the target storage system, wherein the first target data file is isolated from the data files corresponding to other experimental branches and the trunk branch in the target storage system. The method further includes: Based on the first target data file, the target model is trained to obtain training results, and it is determined whether the training results meet the preset requirements. If the training results meet the preset requirements, a second target snapshot corresponding to the main branch is created, and a correspondence indication relationship is established between the second target snapshot and the first target data file and the existing data file corresponding to the main branch.

2. The method according to claim 1, characterized in that, The first target snapshot corresponds to target file details, which include the target storage path allocated for the first target data file in the target storage system; storing the first target data file to the corresponding storage location in the target storage system according to the storage path indicated by the first target snapshot includes: According to the target storage path included in the target file details corresponding to the first target snapshot, the target data file is stored in the storage location corresponding to the target storage path.

3. The method according to claim 1 or 2, characterized in that, The step of establishing the correspondence between the second target snapshot and the first target data file and the existing data files corresponding to the trunk branch includes: Determine whether there are contradictory feature data between the first target data file and the existing data files corresponding to the main branch; If there are no conflicting feature data between the first target data file and the existing data file corresponding to the trunk branch, establish a correspondence between the second target snapshot and the first target data file and the existing data file corresponding to the trunk branch.

4. The method according to claim 1, characterized in that, When the feature data in the first target data file is data obtained by modifying the feature data in the data file corresponding to the main branch based on a preset modification logic, the method further includes: If the data file corresponding to the main branch is changed, the first target data file after the change is obtained based on the latest changed data file corresponding to the main branch and the preset change logic; Create a new third target snapshot corresponding to the first target experimental branch, and establish a corresponding indication relationship between the third target snapshot and the modified first target data file.

5. The method according to claim 1, characterized in that, The method further includes: For each snapshot, determine whether the retention time of each snapshot exceeds the preset time; If the retention time of the snapshot exceeds the preset time and the snapshot is not connected to any branch, the snapshot will be deleted.

6. The method according to claim 2, characterized in that, The method further includes: For each branch, if there are file details on that branch that do not correspond to any of the snapshots, delete those file details that do not correspond to any of the snapshots from the table metadata; and / or, For each branch, if there is a data file on the branch that does not have a corresponding indication relationship with any of the snapshots, the data file that does not have a corresponding indication relationship is deleted.

7. The method according to claim 1 or 2, characterized in that, The method further includes: In response to a file read request, based on the second target branch information carried in the file read request, a second target experimental branch is determined from the table metadata, and at least one second target data file that matches the second target branch information carried in the file read request is determined from the data file corresponding to the second target experimental branch. The at least one second target data file is merged during read to obtain a merged data file, and feature data is read based on the merged data file.

8. The method according to claim 7, characterized in that, The feature data in the data file also has identification information; the read-time merging of the at least one second target data file to obtain a merged data file includes: Based on the identification information of each feature data in the at least one second target data file, feature data with the same identification information are merged to obtain the merged data file.

9. A data storage device, characterized in that, include: A data acquisition module is used to respond to a file storage request for a first target data file, acquire the first target data file to be stored in the target storage system, the first target data file including at least one feature data; the target storage system is configured with table data, the table data is used to indicate the storage branch information corresponding to the target storage system, the storage branch information includes a trunk branch and at least one experimental branch, the trunk branch is used to mount data files that have been used for model training and whose training results meet preset requirements, the at least one experimental branch is used to mount data files that have not yet been used for model training, the trunk branch and the at least one experimental branch are isolated from each other, and the file storage request carries the first target branch information; The branch determination module is used to determine, based on the table metadata, a first target experimental branch that matches the first target branch information carried in the file storage request from the at least one experimental branch, and to create a first target snapshot corresponding to the first target experimental branch. The first target snapshot is used to indicate the target storage path of the first target data file in the target storage system. The data storage module is used to store the first target data file in the corresponding storage location of the target storage system according to the storage path indicated by the first target snapshot, wherein the first target data file is isolated from the data files corresponding to other experimental branches and the trunk branch in the target storage system; The relationship determination module is used for: Based on the first target data file, the target model is trained to obtain training results, and it is determined whether the training results meet the preset requirements. If the training results meet the preset requirements, a second target snapshot corresponding to the main branch is created, and a correspondence indication relationship is established between the second target snapshot and the first target data file and the existing data file corresponding to the main branch.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the data storage method as described in any one of claims 1-8 is performed.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the data storage method as described in any one of claims 1-8.

Images