Data management method and device, equipment and storage medium

Abstract

The invention provides a data management method and device, equipment and a storage medium, and the data management method comprises the steps: responding to a data management request for a target cloud storage module, and obtaining a target data cache volume corresponding to the data management request in the target cloud storage module; performing anomaly detection on the target data cache volume according to the target detection parameter of the target data cache volume and a preset detection database to obtain a cache volume detection result; and performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result. According to the technical scheme, the data security of a distributed cloud storage mode can be improved, the loss probability of the cloud storage data volume is reduced, and the operation stability of cloud storage is improved.

Description

Technical Field

[0001] This application relates to the field of computer technologies, and in particular, to a data management method, apparatus, device, and storage medium. Background Art

[0002] Currently, with the rapid development of Internet technologies and cloud storage technologies, more and more users use cloud storage for model training or other business operations. For example, users can use distributed caching tools for business operations such as providing data for machine learning training. However, existing cloud storage tools such as distributed caching tools have defects such as complex usage processes, inability to detect whether stored files are damaged and error information, and occupation of a large amount of storage space when in use, which easily leads to data loss or anomalies and cannot meet the cloud storage scenario requirements of users. Summary of the Invention

[0003] Embodiments of this application provide a data management method, apparatus, device, and storage medium, aiming to solve the technical problem that data loss or anomalies are likely to occur when performing business tasks in a cloud storage scenario in the prior art.

[0004] On the one hand, embodiments of this application provide a data management method, and the data management method includes the following steps:

[0005] In response to a data management request for a target cloud storage module, obtain a target data cache volume corresponding to the data management request in the target cloud storage module;

[0006] Perform anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result;

[0007] Perform data management on the target data cache volume according to the cache volume detection result to obtain a data management result.

[0008] In a possible implementation manner of this application, before obtaining the target data cache volume corresponding to the data management request in the target cloud storage module, it further includes:

[0009] Obtain an initial data volume to be configured, and data volume mounting parameters of the initial data volume;

[0010] Configure the initial data volume according to the data volume mounting parameters and a preset initial configuration file to obtain a mounted data cache volume;

[0011] The obtaining of the target data cache volume corresponding to the data management request in the target cloud storage module includes:

[0012] Read the target data cache volume in the mounted data cache volume according to the cache volume identifier in the data management request.

[0013] In a possible implementation manner of the present application, before performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result, it further includes:

[0014] Obtain the cache volume backup policy corresponding to the target data cache volume;

[0015] Use the cache volume backup policy to back up the target data cache volume to obtain cache volume backup data;

[0016] The performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result includes:

[0017] If it is determined that the cache volume detection result is an abnormal cache volume detection result, use the cache volume backup data to perform a fault recovery on the target data cache volume to obtain a data management result.

[0018] In a possible implementation manner of the present application, the configuring the initial data volume according to the data volume mounting parameter and a preset initial configuration file to obtain a mounted data cache volume includes:

[0019] Obtain the data volume mounting identifier and the data volume mounting point in the cache mounting parameter, and generate a data volume mounting string according to the data volume mounting identifier and the data volume mounting point;

[0020] Update the initial data volume variable of the initial configuration file according to the data volume mounting string to obtain a target configuration file;

[0021] Perform a mounting configuration on the initial data volume according to the target configuration file to obtain a mounted data cache volume.

[0022] In a possible implementation manner of the present application, the performing an abnormal detection on the target data cache volume according to the target detection parameter of the target data cache volume and a preset detection database to obtain a cache volume detection result includes:

[0023] Respond to an abnormal detection request for the target data cache volume, and generate an abnormal detection parameter corresponding to the abnormal detection request;

[0024] Determine the target detection file in the target data cache volume according to the abnormal detection parameter and the preset detection database;

[0025] Perform a hash operation on the target detection file to obtain the target detection parameter of the target detection file;

[0026] Perform anomaly detection on the target data cache volume according to the target detection parameter and the anomaly detection parameter to obtain a cache volume detection result.

[0027] In a possible implementation manner of the present application, before determining the target detection file in the target data cache volume according to the anomaly detection parameter and a preset detection database, the method further includes:

[0028] Obtain the cache file data in the target data cache volume, perform an operation on the cache file data to obtain a cache file operation value corresponding to the cache file data;

[0029] Associate and store the cache file operation value and the cache file data in a preset database to obtain an initial detection database;

[0030] Add the initial detection database to a local cache module according to a data addition instruction in a preset cache version management module to obtain a detection database corresponding to the target data cache volume.

[0031] In a possible implementation manner of the present application, the performing anomaly detection on the target data cache volume according to the target detection parameter and the anomaly detection parameter to obtain a cache volume detection result includes:

[0032] If the target detection parameter is the same as the anomaly detection parameter, determine that the target detection file is a normal detection file;

[0033] If the target detection parameter is different from the anomaly detection parameter, determine that the target detection file is an anomaly detection file;

[0034] If the anomaly detection file exists in the target data cache volume, determine that the cache volume detection result of the target data cache volume is an abnormal cache volume detection result.

[0035] In a possible implementation manner of the present application, the performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result includes:

[0036] Read the cache volume detection result and determine that the cache volume detection result is an abnormal cache volume detection result;

[0037] Obtain cache volume backup data of the target data cache volume and a data upload instruction in a cache version management module;

[0038] Use the data upload instruction to write the cache volume backup data into the target data cache volume to obtain a data management result.

[0039] On the other hand, the present application provides a data management device, which includes:

[0040] A cache volume acquisition module, configured to respond to a data management request for a target cloud storage module, and acquire a target data cache volume corresponding to the data management request in the target cloud storage module;

[0041] A cache volume detection module, configured to perform anomaly detection on the target data cache volume according to target detection parameters of the target data cache volume and a preset detection database, and obtain a cache volume detection result;

[0042] A data management module, configured to perform data management on the target data cache volume according to the cache volume detection result, and obtain a data management result.

[0043] On the other hand, the present application further provides a data management device, which includes:

[0044] One or more processors;

[0045] A memory; and

[0046] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement the steps of the data management method.

[0047] On the other hand, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the data management method.

[0048] In this application, by responding to a data management request for a target cloud storage module, the target data cache volume corresponding to the data management request in the target cloud storage module is obtained; the target data cache volume is subjected to anomaly detection according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result; and data management is performed on the target data cache volume according to the cache volume detection result to obtain a data management result. After receiving a data management request corresponding to a mounted target data cache volume in the target cloud storage module of distributed storage, the target detection parameters corresponding to the specified stored file data in the target data cache volume and the preset detection database are obtained to perform anomaly detection on the target data cache volume, so as to evaluate whether the stored files in the target data cache volume are damaged or have other anomalies, obtain a cache volume detection result, and perform data management on the target data cache volume according to the cache volume detection result to perform operations such as fault recovery on the abnormal stored files in the target data cache volume to obtain a data management result. This is to improve the data security of the distributed cloud storage method, reduce the probability of loss of cloud storage data volumes, and improve the operating stability of cloud storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative efforts.

[0050] Figure 1 It is a schematic diagram of the scenario of the data management method in the embodiment of the present application;

[0051] Figure 2 It is a schematic flowchart of an embodiment of the data management method in the embodiment of the present application;

[0052] Figure 3 It is a schematic flowchart of an embodiment of creating a target data cache volume in the data management method provided in the embodiment of the present application;

[0053] Figure 4 It is a schematic flowchart of an embodiment of performing anomaly detection on a target data cache volume to obtain a cache volume detection result in the data management method provided in the embodiment of the present application;

[0054] Figure 5 It is a schematic structural diagram of an embodiment of the data management device provided in the embodiment of the present application;

[0055] Figure 6 It is a schematic structural diagram of an embodiment of the data management device provided in the embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present invention.

[0057] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation of the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of the described features. In the description of the present invention, "a plurality" means two or more, unless otherwise specifically defined.

[0058] In the present application, the term "exemplary" is used to mean "serving as an example, illustration, or description". Any embodiment described as "exemplary" in the present application is not necessarily to be construed as more preferred or more advantageous than other embodiments. In order for any person skilled in the art to implement and use the present invention, the following description is given. In the following description, details are set forth for the purpose of explanation. It should be understood that those skilled in the art can recognize that the present invention can be implemented without the use of these specific details. In other instances, well-known structures and processes are not described in detail to avoid unnecessary details from obscuring the description of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and features disclosed in the present application.

[0059] Currently, with the rapid development of Internet technology and cloud storage technology, more and more users use cloud storage for model training or other business operations. For example, users can use distributed cache tools for business operations such as machine learning training data supply. However, existing cloud storage tools such as distributed cache tools have defects such as complex usage processes, inability to detect whether stored files are damaged and error information, and occupation of a large amount of storage space when in use, which easily leads to data loss or anomalies and cannot meet the cloud storage scenario requirements of users.

[0060] Based on this, the present application proposes a data management method, apparatus, device, and computer-readable storage medium to solve the technical problem that data loss or anomalies are likely to occur when performing business tasks in a cloud storage scenario in the prior art.

[0061] The data management method in the embodiments of the present invention is applied to a data management apparatus, the data management apparatus is disposed in a data management device, and one or more processors, a memory, and one or more application programs are provided in the data management device. One or more application programs are stored in the memory and are configured to be executed by the processor to implement the data management method; wherein, the data management device may be an intelligent terminal, such as a mobile phone, a tablet computer, a network device, and an intelligent computer, etc.; optionally, the data management device may also be a server, or a service cluster composed of multiple servers.

[0062] As Figure 1 shown, Figure 1 is a schematic diagram of the scenario of the data management method in the embodiments of the present application. In the embodiments of the present invention, the data management scenario includes a data management device 100 (the data management device 100 integrates a data management apparatus) and a target cloud storage module 200. A computer-readable storage medium corresponding to the data management method runs in the data management device 100 to execute the steps of the data management method. The target cloud storage module 200 is a distributed cloud storage terminal for data cloud storage interaction with the data management device 100. Optionally, in a specific embodiment, the target cloud storage module may be a JuiceFS distributed storage server.

[0063] It can be understood that Figure 1 the data management device in the data management method scenario shown, or the apparatus included in the data management device does not constitute a limitation to the embodiments of the present invention, that is, the number of data management devices, the types of devices, or the number of apparatuses included in each device, the types of apparatuses in the data management method scenario do not affect the overall implementation of the technical solution in the embodiments of the present invention, and can all be regarded as equivalent replacements or derivatives of the technical solution required to be protected in the embodiments of the present invention.

[0064] The data management device 100 in the embodiments of the present invention is mainly used for: responding to a data management request for the target cloud storage module, obtaining a target data cache volume corresponding to the data management request in the target cloud storage module; performing anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result; performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result.

[0065] The data management device 100 in the embodiments of the present invention may be an independent data management device, such as intelligent terminals like mobile phones, tablet computers, network devices, servers, and intelligent computers, or may be a data management network or data management cluster composed of multiple data management devices.

[0066] The embodiments of the present application provide a data management method, device, equipment, and computer-readable storage medium, which will be described in detail below respectively.

[0067] Those skilled in the art can understand that Figure 1 the application environment shown is only one of the application scenarios related to the solution of the present application, and does not constitute a limitation on the application scenario of the solution of the present application. Other application environments may also include more or fewer data management devices than Figure 1 shown, or data management network connection relationships. For example Figure 1 only one data management device is shown in. It can be understood that the scenario of this data management method may also include one or more data management devices, which are not specifically limited here; the data management device 100 may also include a memory for storing detection data and other data.

[0068] It should be noted that Figure 1 the schematic diagram of the scenario of the data management method shown is only an example. The scenario of the data management method described in the embodiments of the present invention is for more clearly explaining the technical solution of the embodiments of the present invention, and does not constitute a limitation on the technical solution provided by the embodiments of the present invention.

[0069] Based on the above scenario of the data management method, various embodiments of the data management method disclosed by the present invention are proposed.

[0070] As Figure 2 shown Figure 2 is a schematic flowchart of an embodiment of the data management method in the embodiments of the present application. The data management method includes the following steps 201 to step 203:

[0071] 201. In response to a data management request for a target cloud storage module, obtain a target data cache volume corresponding to the data management request in the target cloud storage module;

[0072] The data management method in this embodiment is applied to a data management device. The type and quantity of the data management device are not specifically limited. That is, the data management device may be one or more intelligent terminals or servers. In a specific embodiment, the data management device is an intelligent computer.

[0073] Among them, the target cloud storage module is communicatively connected to the data management device, capable of mounting the corresponding data cache volume to the local of the data management device, receiving the cache files accessed and uploaded by the data management device, and transmitting the cache files to the cloud storage server through the cloud storage client in a distributed cloud storage system. Among them, the target cloud storage module may include a cloud storage client and a cloud storage server. Optionally, in a specific embodiment, the target cloud storage module may be a distributed cloud storage system such as JuiceFS.

[0074] Among them, the target data cache volume is a cloud storage data cache volume that has been mounted and accessed locally by the data management device in the target cloud storage module and awaits data management.

[0075] Specifically, during operation, the data management device can respond to a data management request associated with the target data cache volume mounted in the target cloud storage module. Among them, the data management request is an operation event for performing storage fault analysis on the target data cache volume in the target cloud storage module during data operation and maintenance to determine whether the cached data in the target data cache volume is abnormally faulty. Optionally, the triggering method of the data management request is not specifically limited herein. That is, the data management request can be triggered actively by a user. For example, the user actively triggers a data management request for the target data cache volume by clicking a data management button corresponding to the target data cache volume in the target cloud storage module in the data management device. In addition, the data management request can also be triggered automatically by the data management device. For example, a data management process is pre-set in the data management device, and a data management request for the target data cache volume is automatically triggered when specific management conditions are met.

[0076] Specifically, after receiving a data management request associated with the target data cache volume, the data management device obtains the target data cache volume corresponding to the data management request, and performs data volume health detection and data volume health status evaluation on the target data cache volume in subsequent steps to determine whether the cached data in the target data cache volume has failed. That is, after obtaining the data management request, the data management device parses the data management request to obtain the cache volume identifier carried by the data management request, and locates the target data cache volume corresponding to the cache volume identifier among the mounted data cache volumes that have been mounted to the data management device according to the cache volume identifier.

[0077] Optionally, in a specific embodiment, before receiving a data management request associated with the target data cache volume, the data management device also configures the cloud storage client in the target cloud storage module, and then quickly creates a cache volume through the configured cloud storage client and mounts it to the data management device. That is, the data management device calls the cache volume creation interface and passes in the data volume mounting parameters corresponding to the initial data volume for generating the mounted data cache volume to a preset orchestration application to generate a corresponding configuration file, and configures the initial data volume based on the configuration file to obtain the mounted data cache volume, and creates a data volume access interface and a service exposure port corresponding to the mounted data cache volume, and detects whether the service exposure port is available. If the service port is available, it is determined that the mounting configuration of the mounted data cache volume is completed.

[0078] 202. Perform anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result;

[0079] Specifically, after determining the target data cache volume corresponding to the data management request, the data management device also performs anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result.

[0080] Wherein, the target detection parameter is a detection parameter characterizing the data integrity of the specified cache file data in the target data cache volume. Optionally, the target detection parameter can be a detection parameter obtained by performing specified data calculation on the specified cache data when synchronizing the specified cache data to the target data cache volume or when modifying the specified cache data in the target data cache volume.

[0081] Optionally, in a specific embodiment, the data management device can calculate the target detection parameter when writing data to the target data cache volume. That is, after generating and mounting the target data cache volume, the data management device also pre-synchronizes the specified cache file data to the target data cache volume, and before synchronizing the specified cache file data to the target data cache volume, performs a hash operation on each specified cache file data to obtain the file data hash value corresponding to each specified cache file data, and determines the file data hash value as the target detection parameter of the specified cache file data. Optionally, in other embodiments, the target detection parameter can also be other detection parameters that can characterize the integrity of the cache file data except the file data hash value. This embodiment does not make specific limitations.

[0082] Specifically, the data management device also pre-creates a detection database associated with the target data cache volume, and stores the target detection parameters corresponding to each cache file data in the target data cache volume into the detection database. That is, the data management device stores the file path corresponding to the cache file data as the data key and the target detection parameter as the data value into the detection database. Optionally, the detection database can be a Redis database or a local cache database.

[0083] Optionally, in another specific embodiment, when the data management device modifies the cache data in the target data cache volume, it can synchronously modify the target detection parameter of the cache data to obtain the modified target detection parameter. That is, when the data management device modifies the cache file data in the target cache data volume, it first obtains the target detection parameter associated with the cache file data in the detection database, and locks the target detection parameter using a distributed lock. After locking the target detection parameter, the data management device modifies the cache file data to obtain the modified file data, calculates the modified detection parameter corresponding to the modified file data, uses the modified detection parameter as the target detection parameter of the modified file data, and releases the distributed lock.

[0084] Optionally, in other embodiments, the target detection parameter can also be the folder detection parameter obtained by the data management device through data calculation on the cache folder in the target data cache volume.

[0085] Specifically, after determining the target detection parameters in each target data cache volume, the data management device can also respond to an exception detection request for the target data cache volume, locate the target detection file to be detected and the corresponding target detection parameter according to the exception detection request and the detection database, and then perform a health detection on the target data cache volume according to the target detection parameter and the exception detection parameter corresponding to the exception detection request, so as to judge the data health status of the target data cache volume, and output the corresponding cache volume detection result according to the data health status. Among them, the data health status is a detection status indicating whether there are abnormal faults such as data corruption in the target detection files in the target data cache volume.

[0086] Optionally, the data management device can also perform a connection status detection on the target data cache volume. That is, the data management device regularly calls the cache volume interface of the target data cache volume according to a preset detection frequency to detect whether the cache volume interface of the target data cache volume can be correctly accessed, so as to judge whether the cache node of the target data cache volume is connected, determine the target data cache volume corresponding to the unconnected cache node as an abnormal data cache volume, and output the corresponding alarm information and cache volume detection result.

[0087] 203. Perform data management on the target data cache volume according to the cache volume detection result to obtain a data management result.

[0088] Specifically, the data management device can also perform data management on the target data cache volume according to the cache volume detection result to obtain a data management result. That is, the data management device can perform backup and recovery on the anomaly detection files in the target data cache volume according to the cache volume detection result, so as to ensure the validity of the cache file data in the target data cache volume and guarantee the stability of the business execution running based on the target data cache volume.

[0089] Specifically, the data management device has also pre-set a timed backup process. According to this timed backup process, it reads the cache file data in the target data cache volume and the cache volume backup policy corresponding to the target data cache volume. Among them, the cache volume backup policy is a backup policy representing the corresponding data backup operation for the target data cache volume. Optionally, the cache volume backup policy includes a first backup policy and a second backup policy. The data management device can perform cache version backup on the cache file data according to the first backup policy and / or the second backup policy to obtain cache volume backup data. Among them, the cache volume backup data includes first cache volume backup data and second cache volume backup data, and uploads the first cache volume backup data and the second cache volume backup data to the backup file database. Among them, the first backup policy is a backup policy for performing metadata backup on the target data cache volume. The second backup policy is a backup policy for backing up the target data cache volume by using the cache version management module. Optionally, the backup file database can be a NAS (Network Attached Storage) database.

[0090] Optionally, in a specific embodiment, the data management device can use the first backup policy to read the cache metadata recording the basic data information in the cache file data of the target data cache volume, and generate first cache volume backup data according to the full cache metadata, and store the first cache volume backup data in the backup file database. Among them, the first cache volume backup data is the cache version metadata corresponding to the target data cache volume.

[0091] Optionally, in another specific embodiment, the data management device can use the second backup policy to read the cache file data in the target data cache volume, calculate the difference detection parameters between each cache file data and the historical cache file data, and then determine the incremental difference data whose difference detection parameters are not equal to the preset difference threshold, and perform backup on the incremental difference data to obtain second cache volume backup data, and upload the second cache volume backup data to the backup file database. Among them, the second cache volume backup data is the incremental backup data corresponding to the target data cache volume.

[0092] Specifically, after the data management device reads the cache volume detection result corresponding to the target data cache volume and determines that the cache volume detection result is an abnormal cache volume detection result, it also uses the cache volume backup data to perform a metadata backup recovery operation or a version backup recovery operation on the target data cache volume to obtain a data management result.

[0093] Optionally, in a specific embodiment, after the data management device determines that the cache volume detection result is an abnormal cache volume detection result, it reads the latest version of the cache version metadata associated with the target data cache volume in the backup file database as the cache volume backup data, and uses the cache version metadata to perform a metadata backup restoration on the target data cache volume to obtain a data management result.

[0094] Optionally, in another specific embodiment, after the data management device determines that the cache volume detection result is an abnormal cache volume detection result, it also uses the cache version management module to access the backup file database, obtains the cache volume backup data corresponding to the target data cache volume in the backup file database, and obtains the data upload instruction corresponding to the cache version management module, and uses the data upload instruction to backup and write the cache volume backup data into the target data volume to obtain a data management result.

[0095] In this embodiment, the data management device responds to a data management request for a target cloud storage module, obtains the target data cache volume corresponding to the data management request in the target cloud storage module; performs an anomaly detection on the target data cache volume according to the target detection parameter of the target data cache volume and a preset detection database to obtain a cache volume detection result; performs data management on the target data cache volume according to the cache volume detection result to obtain a data management result. It realizes that after receiving a data management request corresponding to a target data cache volume mounted in a target cloud storage module of a distributed storage, obtaining the target detection parameter corresponding to the specified storage file data in the target data cache volume and a preset detection database to perform an anomaly detection on the target data cache volume, so as to evaluate whether the stored files in the target data cache volume are damaged or other abnormal situations, obtain a cache volume detection result, and perform data management on the target data cache volume according to the cache volume detection result to perform operations such as fault recovery on the abnormal stored files in the target data cache volume to obtain a data management result. To improve the data security of the distributed cloud storage method, reduce the probability of loss of cloud storage data volumes, and improve the operation stability of cloud storage.

[0096] As Figure 3 shown, Figure 3 FIG. is a schematic flowchart of an embodiment of creating a target data cache volume in the data management method provided by the embodiment of the present application. Specifically, in this embodiment, the data management method further includes steps 301 to 302:

[0097] 301. Obtain the initial data volume to be configured and the data volume mounting parameters of the initial data volume;

[0098] 302. Configure the initial data volume according to the data volume mounting parameters and a preset initial configuration file to obtain a mounted data cache volume.

[0099] Based on the above embodiments, in this embodiment, before the data management device receives a data management request associated with a target data cache volume, it also configures the cloud storage client in the target cloud storage module, and then quickly creates a cache volume through the configured cloud storage client and mounts it to the data management device, thereby improving the cache volume generation efficiency and simplifying the cache volume generation complexity.

[0100] Specifically, during operation, the data management device can obtain the initial data volume to be configured and receive the data volume mounting parameters corresponding to the initial data volume. Among them, the data volume mounting parameters include a data volume mounting identifier and a data volume mounting point. Among them, the data volume mounting identifier is the cache volume name identifier after the initial data volume is mounted. The data volume mounting point is the mounting path corresponding to the initial data volume.

[0101] Specifically, the data management device also obtains the initial configuration file corresponding to the cache volume mounting request and the initial data variables in the initial configuration file. Among them, the initial configuration file is a configuration template file for mounting and configuring the initial data volume. The initial data variable is a configuration template variable to be mounted and configured in the initial data volume. Optionally, in a specific embodiment, the initial configuration file can be a yaml splicing file. Optionally, in a specific embodiment, the file code of an embodiment of the initial configuration file is as follows:

[0102]

[0103] Among them, in this specific embodiment, the initial data variables of the initial configuration file are #{juicefs.vol ume.spec|4} and #{juicefs.vo l ume.spec|10} respectively. Among them, the (|4) and (|10) are the number of spaces to be added before each line when representing replacement variables.

[0104] Specifically, after the data management device obtains the data volume mounting parameters, the initial configuration file, and the initial data variables, the data management device also updates the initial data volume by using the data volume mounting parameters and the initial data set variables to obtain a mounted data cache volume. That is, the data management device obtains the data mounting identifier and the data volume mounting point in the cache-mounted data volume, and generates a data volume mounting string according to the data volume mounting identifier and the data volume mounting point. That is, the data management device concatenates the data volume mounting identifier and the data volume mounting point into a data volume mounting string.

[0105] Optionally, before generating the data volume mounting string, the data management device also determines whether there are duplicate data volume mounting identifiers or duplicate data volume mounting points in the data volume mounting identifier and the data volume mounting point. If there are no duplicate data volume mounting identifiers and duplicate data volume mounting points, the data management device generates a data volume mounting string according to the data volume mounting identifier and the data volume mounting point.

[0106] Specifically, after the data management device obtains the data volume mounting string, the data management device updates the initial data volume variable of the initial configuration file according to the data volume mounting string to obtain a target configuration file. That is, the data management device replaces the initial data volume variable in the initial configuration file by using the data volume mounting string to obtain a target configuration file containing the target data volume variable.

[0107] Optionally, in a specific embodiment, the data mounting identifier is date_dvc, and the data volume mounting point is / date / dvc. The file code of an embodiment of the target configuration file is as follows:

[0108]

[0109] Specifically, after generating the target configuration file, the data management device also performs a mounting configuration on the initial data volume according to the target configuration file to obtain a mounted data cache volume.

[0110] Specifically, after generating the mounted data cache volume, the data management device also responds to a data management request, obtains the cache volume identifier in the data management request, and reads a target data cache volume in the mounted data cache volume whose data volume identifier is the same as the cache volume identifier according to the cache volume identifier.

[0111] In this embodiment, the data management device obtains the initial data volume corresponding to the cache volume mounting request and the data volume mounting parameters of the initial data volume by responding to the cache volume mounting request; obtains the initial configuration file corresponding to the cache volume mounting request and the initial data volume variable of the initial configuration file; updates the initial data volume according to the data volume mounting parameters and the initial data volume variable to obtain a mounted data cache volume; and reads the target data cache volume in the mounted data cache volume according to the cache volume identifier in the data management request. It realizes the creation of a data cache volume through the orchestration of the cache creation process, improves the cache volume creation efficiency of distributed cloud storage, and simplifies the complexity of cache volume creation.

[0112] As Figure 4 shown, Figure 4 FIG. is a schematic flowchart of an embodiment for performing anomaly detection on a target data cache volume in the data management method provided by an embodiment of the present application to obtain a cache volume detection result. Specifically, in this embodiment, the data management method further includes steps 401 to 404:

[0113] 401. Respond to an anomaly detection request for the target data cache volume and generate anomaly detection parameters corresponding to the anomaly detection request;

[0114] 402. Determine the target detection file in the target data cache volume according to the anomaly detection parameters and a preset detection database;

[0115] 403. Perform a hash operation on the target detection file to obtain target detection parameters of the target detection file;

[0116] 404: Perform anomaly detection on the target data cache volume according to the target detection parameters and the anomaly detection parameters to obtain a cache volume detection result.

[0117] Based on the above embodiment, in this embodiment, the data management device can respond to an anomaly detection request for the target data cache volume, locate the target detection file to be detected and the target detection parameters corresponding to the target detection file according to the anomaly detection request and the detection database, and then perform a health check on the target data cache volume according to the target detection parameters and the anomaly detection parameters corresponding to the anomaly detection request, so as to judge the data health status of the target data cache volume, and output a corresponding cache volume detection result according to the data health status. Among them, the data health status is a detection status indicating whether there are abnormal faults such as data corruption in the target detection file in the target data cache volume.

[0118] Specifically, during operation, the data management device can receive an anomaly detection request for a target data cache volume. Here, the anomaly detection request is an operation event that drives the data management device to perform a health check on the specified target data cache volume.

[0119] Specifically, after receiving the anomaly detection request, the data management device also uses a random sampling module to generate several anomaly detection parameters corresponding to the anomaly detection request, and performs parallel detection on the target data cache volume according to the anomaly detection parameters to determine the data health status corresponding to the target data cache volume. Here, the anomaly detection parameters are sampling detection parameters randomly generated by the random sampling module and associated with the target data cache volume. Optionally, in a specific embodiment, the anomaly detection parameter can be a hash value.

[0120] Specifically, after obtaining the anomaly detection parameters, the data management device determines the target detection file of the target data cache volume according to the anomaly detection parameters and a preset detection database. That is, the data management device accesses the detection database, inputs the anomaly detection parameters into the detection database, and uses the anomaly detection parameters to locate the target detection file to be detected for the target data cache volume in the detection database. Here, the target detection file is the cache file data whose file hash value is the same as the anomaly detection parameter.

[0121] Specifically, after determining the target detection file to be detected, the data management device performs a hash operation on the target detection file stored in the target data cache volume to obtain the target detection hash value of the target detection file, uses the target detection hash value as the target detection parameter of the target detection file, and compares the target detection parameter with the anomaly detection parameter to determine whether the target detection file in the target data cache volume is abnormal, and obtains the cache volume detection result.

[0122] Optionally, in a specific embodiment, before using the detection database to determine the target detection file of the target data cache volume, the data management device also generates a detection database corresponding to the target data cache volume. That is, the data management device obtains the cache file data in the target data cache volume, performs an operation on the cache file data to obtain the cache file operation value corresponding to the cache file data. The cache file operation value and the cache file data are stored in association in a preset database to obtain an initial detection database, and the initial detection database is added to the local cache module according to the data addition instruction in the preset cache version management module to obtain the detection database corresponding to the target data cache volume. Here, the preset cache version management module is a DVC (Data Version Control) module, and the data addition instruction is a dvc add instruction.

[0123] Specifically, after obtaining the target detection file, the data management device also performs a hash operation on the obtained target detection file to obtain the current target detection parameters of the target detection file, and performs an anomaly detection on the target cache volume based on the newly obtained target detection parameters and the anomaly detection parameters to obtain a cache volume detection result. That is, the data management device calculates the current target detection parameters of the target detection file, and compares the target detection parameters with the anomaly detection parameters to determine whether the data integrity of the target detection file has changed, and then outputs the corresponding cache volume detection result.

[0124] Optionally, if the target detection parameter is the same as the anomaly detection parameter, the data management device determines that the current data of the target detection file is the same as the data when it was written to or modified in the target data cache volume, and determines that the target detection file is a normal detection file.

[0125] Optionally, if the target detection parameter is different from the anomaly detection parameter, the data management device determines that the data of the target detection file has changed, and determines that the target detection file is an anomaly detection file with data corruption.

[0126] Optionally, if the data management device determines that there is such an anomaly detection file in the target data cache volume after detecting each target detection file, it determines that the cache volume detection result of the target data cache volume is an abnormal cache volume detection result.

[0127] Specifically, after the data management device determines an abnormal cache volume detection result caused by data corruption in the target data cache volume, it can accurately back up and recover the abnormal detection files in the target data cache volume, thereby ensuring the validity of the cache file data in the target data cache volume and guaranteeing the stability of the business execution based on the target data cache volume.

[0128] In this embodiment, the data management device generates anomaly detection parameters corresponding to the anomaly detection request by responding to the anomaly detection request for the target data cache volume; determines the target detection file in the target data cache volume according to the anomaly detection parameters and a preset detection database; performs a hash operation on the target detection file to obtain the target detection parameters of the target detection file; performs an anomaly detection on the target data cache volume according to the target detection parameters and the anomaly detection parameters to obtain a cache volume detection result. It realizes accurate identification of whether there are anomalies such as data corruption in the cache file data in the target data cache volume, thereby improving the accuracy and timeliness of data detection and data repair, ensuring the validity of the cache file data in the target data cache volume, and guaranteeing the stability of the business execution based on the target data cache volume.

[0129] To better implement the data management method in the embodiments of the present application, based on the data management method, an embodiment of the present application also provides a data management device, as Figure 5 shown Figure 5 is a schematic structural diagram of an embodiment of the data management device provided by the embodiments of the present application. Specifically, the data management device 500 includes:

[0130] A cache volume acquisition module 501, configured to respond to a data management request for a target cloud storage module, and acquire a target data cache volume corresponding to the data management request in the target cloud storage module;

[0131] A cache volume detection module 502, configured to perform anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result;

[0132] A data management module 503, configured to perform data management on the target data cache volume according to the cache volume detection result to obtain a data management result.

[0133] In a possible implementation manner of this embodiment, before the data management device acquires the target data cache volume corresponding to the data management request in the target cloud storage module, it further includes:

[0134] Acquire an initial data volume to be configured, and the data volume mounting parameters of the initial data volume;

[0135] Configure the initial data volume according to the data volume mounting parameters and a preset initial configuration file to obtain a mounted data cache volume;

[0136] The acquiring the target data cache volume corresponding to the data management request in the target cloud storage module includes:

[0137] Read the target data cache volume in the mounted data cache volume according to the cache volume identifier in the data management request.

[0138] In a possible implementation manner of this embodiment, before the data management device performs data management on the target data cache volume according to the cache volume detection result to obtain a data management result, it further includes:

[0139] Acquire the cache volume backup policy corresponding to the target data cache volume;

[0140] Use the cache volume backup policy to back up the target data cache volume to obtain cache volume backup data;

[0141] The performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result includes:

[0142] If it is determined that the cache volume detection result is an abnormal cache volume detection result, the target data cache volume is recovered from a fault using the cache volume backup data to obtain a data management result.

[0143] In a possible implementation manner of this embodiment, the data management device configures the initial data volume according to the data volume mounting parameter and a preset initial configuration file to obtain a mounted data cache volume, including:

[0144] Obtain the data volume mounting identifier and the data volume mounting point in the cache mounting parameter, and generate a data volume mounting string according to the data volume mounting identifier and the data volume mounting point;

[0145] Update the initial data volume variable of the initial configuration file according to the data volume mounting string to obtain a target configuration file;

[0146] Mount and configure the initial data volume according to the target configuration file to obtain a mounted data cache volume.

[0147] In a possible implementation manner of this embodiment, the data management device performs an abnormality detection on the target data cache volume according to the target detection parameter of the target data cache volume and a preset detection database to obtain a cache volume detection result, including:

[0148] Respond to an abnormality detection request for the target data cache volume, and generate an abnormality detection parameter corresponding to the abnormality detection request;

[0149] Determine a target detection file in the target data cache volume according to the abnormality detection parameter and the preset detection database;

[0150] Perform a hashing operation on the target detection file to obtain a target detection parameter of the target detection file;

[0151] Perform an abnormality detection on the target data cache volume according to the target detection parameter and the abnormality detection parameter to obtain a cache volume detection result.

[0152] In a possible implementation manner of this embodiment, before the data management device determines a target detection file in the target data cache volume according to the abnormality detection parameter and the preset detection database, it further includes:

[0153] Obtain the cache file data in the target data cache volume, perform an operation on the cache file data to obtain a cache file operation value corresponding to the cache file data;

[0154] Associate and store the cache file operation value and the cache file data in a preset database to obtain an initial detection database;

[0155] Add the initial detection database to the local cache module according to the data addition instruction in the preset cache version management module to obtain the detection database corresponding to the target data cache volume.

[0156] In a possible implementation manner of this embodiment, the data management device performs anomaly detection on the target data cache volume according to the target detection parameter and the anomaly detection parameter to obtain a cache volume detection result, including:

[0157] If the target detection parameter is the same as the anomaly detection parameter, determine that the target detection file is a normal detection file;

[0158] If the target detection parameter is different from the anomaly detection parameter, determine that the target detection file is an anomaly detection file;

[0159] If the anomaly detection file exists in the target data cache volume, determine that the cache volume detection result of the target data cache volume is an abnormal cache volume detection result.

[0160] In a possible implementation manner of this embodiment, the data management device performs data management on the target data cache volume according to the cache volume detection result to obtain a data management result, including:

[0161] Read the cache volume detection result and determine that the cache volume detection result is an abnormal cache volume detection result;

[0162] Obtain the cache volume backup data of the target data cache volume and the data upload instruction in the cache version management module;

[0163] Use the data upload instruction to write the cache volume backup data into the target data cache volume to obtain a data management result.

[0164] In this embodiment, the data management device obtains a target data cache volume corresponding to the data management request in the target cloud storage module by responding to the data management request for the target cloud storage module; performs anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result; and performs data management on the target data cache volume according to the cache volume detection result to obtain a data management result. After receiving a data management request corresponding to a mounted target data cache volume in the target cloud storage module of the distributed storage, the target detection parameters corresponding to the specified storage file data in the target data cache volume and the preset detection database are obtained to perform anomaly detection on the target data cache volume, so as to evaluate whether the storage files in the target data cache volume are damaged or have other anomalies, obtain a cache volume detection result, and perform data management on the target data cache volume according to the cache volume detection result to perform operations such as fault recovery on the abnormal storage files in the target data cache volume to obtain a data management result. This is to improve the data security of the distributed cloud storage method, reduce the probability of loss of cloud storage data volumes, and improve the operating stability of cloud storage.

[0165] An embodiment of the present invention further provides a data management device, as Figure 6 shown, Figure 6 is a schematic structural diagram of an embodiment of the data management device provided in the embodiments of the present application.

[0166] The data management device integrates any one of the data management devices provided in the embodiments of the present invention. The data management device includes:

[0167] One or more processors;

[0168] A memory; and

[0169] One or more applications, where the one or more applications are stored in the memory and are configured to be executed by the processor to perform the steps in the data management method described in any one of the embodiments of the above data management method embodiment.

[0170] Specifically: The data management device may include a processor 601 with one or more processing cores, a memory 602 of one or more computer-readable storage media, a power supply 603, an input unit 604, and other components. Those skilled in the art can understand that Figure 6 the structure of the data management device shown in

[0171] The processor 601 is the control center of the data management device, connecting various parts of the entire data management device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 602, and by calling the data stored in the memory 602, it performs various functions of the data management device and processes data, thereby monitoring the data management device as a whole. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor. Among them, the application processor mainly processes the operating system, user interface, application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor may not be integrated into the processor 601 either.

[0172] The memory 602 can be used to store software programs and modules. The processor 601 executes various functional applications and data processing by running the software programs and modules stored in the memory 602. The memory 602 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, image playback function, etc.); the data storage area can store the data created according to the use of the data management device. In addition, the memory 602 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 602 can also include a memory controller to provide the processor 601 with access to the memory 602.

[0173] The data management device further includes a power supply 603 for powering each component. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 603 can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0174] The data management device may further include an input unit 604, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0175] Although not shown, the data management device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 601 in the data management device will load the executable files corresponding to the processes of one or more application programs into the memory 602 according to the following instructions, and the processor 601 will run the application programs stored in the memory 602 to realize various functions as follows:

[0176] In response to a data management request for a target cloud storage module, obtain a target data cache volume corresponding to the data management request in the target cloud storage module;

[0177] Perform anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result;

[0178] Perform data management on the target data cache volume according to the cache volume detection result to obtain a data management result.

[0179] For this reason, an embodiment of the present invention provides a computer-readable storage medium, which may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disc, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any of the data management methods provided by the embodiments of the present invention. For example, when the computer program is loaded by the processor, the following steps may be executed:

[0180] In response to a data management request for a target cloud storage module, obtain a target data cache volume corresponding to the data management request in the target cloud storage module;

[0181] Perform anomaly detection on the target data cache volume according to the target detection parameters of the target data cache volume and a preset detection database to obtain a cache volume detection result;

[0182] Perform data management on the target data cache volume according to the cache volume detection result to obtain a data management result.

[0183] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For parts not described in detail in a certain embodiment, reference may be made to the detailed descriptions of other embodiments above, and details will not be repeated here.

[0184] In specific implementation, the above-mentioned respective units or structures may be implemented as independent entities, or may be combined arbitrarily to be implemented as the same or several entities. For the specific implementation of the above-mentioned respective units or structures, reference may be made to the method embodiments above, and details will not be repeated here.

[0185] For the specific implementation of each of the above operations, reference may be made to the foregoing embodiments, and details will not be repeated here.

[0186] The above has introduced in detail a data management method provided by an embodiment of the present application. Specific embodiments are used herein to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A data management method, characterized in that: The data management method comprises: In response to a data management request for a target cloud storage module, obtaining a target data cache volume in the target cloud storage module corresponding to the data management request; Performing anomaly detection on the target data cache volume according to the target detection parameter of the target data cache volume and a preset detection database to obtain a cache volume detection result; Data management is performed on the target data cache volume according to the cache volume detection result to obtain a data management result.

2. The data management method according to claim 1, characterized in that: Before obtaining the target data cache volume corresponding to the data management request in the target cloud storage module, the method further includes: Obtaining an initial data volume to be configured and data volume mounting parameters of the initial data volume; The initial data volume is configured according to the data volume mounting parameters and a preset initial configuration file to obtain a mounted data cache volume; The obtaining a target data cache volume corresponding to the data management request in the target cloud storage module includes: The target data cache volume in the mounted data cache volume is read according to the cache volume identifier in the data management request.

3. The data management method according to claim 1, characterized in that: Before performing data management on the target data cache volume according to the cache volume detection result and obtaining the data management result, the method further includes: Obtaining a cache volume backup policy corresponding to the target data cache volume; Backing up the target data cache volume by using the cache volume backup strategy to obtain cache volume backup data; The performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result includes: If it is determined that the cache volume detection result is an abnormal cache volume detection result, the cache volume backup data is used to perform fault recovery on the target data cache volume to obtain a data management result.

4. The data management method according to claim 2, characterized in that: The configuring the initial data volume according to the data volume mounting parameters and a preset initial configuration file to obtain a mounted data cache volume includes: Obtaining a data volume mount identifier and a data volume mount point in the cache mount parameter, and generating a data volume mount character string according to the data volume mount identifier and the data volume mount point; The initial data volume variable of the initial configuration file is updated according to the data volume mount character string to obtain a target configuration file; The initial data volume is mounted and configured according to the target configuration file to obtain a mounted data cache volume.

5. The data management method according to claim 1, characterized in that: The performing abnormality detection on the target data cache volume according to the target detection parameter of the target data cache volume and a preset detection database to obtain a cache volume detection result includes: In response to the anomaly detection request for the target data cache volume, generate anomaly detection parameters corresponding to the anomaly detection request; Determining a target detection file in the target data cache volume according to the anomaly detection parameter and a preset detection database; Performing a hash operation on the target detection file to obtain target detection parameters of the target detection file; Anomaly detection is performed on the target data cache volume according to the target detection parameter and the anomaly detection parameter to obtain a cache volume detection result.

6. The data management method according to claim 5, characterized in that: Before determining the target detection file in the target data cache volume according to the anomaly detection parameter and a preset detection database, the method further includes: Acquire cache file data in the target data cache volume, perform operations on the cache file data, and obtain cache file operation values ​​corresponding to the cache file data; The cache file calculation value and the cache file data are associated and stored in a preset database to obtain an initial detection database; The initial detection database is added to the local cache module according to the data adding instruction in the preset cache version management module to obtain the detection database corresponding to the target data cache volume.

7. The data management method according to claim 5, characterized in that: The performing anomaly detection on the target data cache volume according to the target detection parameter and the anomaly detection parameter to obtain a cache volume detection result includes: If the target detection parameter is the same as the abnormal detection parameter, determining that the target detection file is a normal detection file; If the target detection parameter and the anomaly detection parameter are different, determining that the target detection file is an anomaly detection file; If the abnormality detection file exists in the target data cache volume, it is determined that the cache volume detection result of the target data cache volume is an abnormal cache volume detection result.

8. The data management method according to any one of claims 1 to 7, characterized in that: The performing data management on the target data cache volume according to the cache volume detection result to obtain a data management result includes: Reading the cache volume detection result, and determining that the cache volume detection result is an abnormal cache volume detection result; Acquire cache volume backup data of the target data cache volume and data upload instructions in a cache version management module; The cache volume backup data is written into the target data cache volume using the data upload instruction to obtain a data management result.

9. A data management device, characterized in that: The data management device comprises: A cache volume acquisition module is configured to respond to a data management request for a target cloud storage module and acquire a target data cache volume corresponding to the data management request in the target cloud storage module; a cache volume detection module configured to perform anomaly detection on the target data cache volume according to a target detection parameter of the target data cache volume and a preset detection database to obtain a cache volume detection result; The data management module is configured to perform data management on the target data cache volume according to the cache volume detection result to obtain a data management result.

10. A data management device, characterized in that: The data management device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and are configured to be executed by the processor to implement the steps of the data management method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps of the data management method according to any one of claims 1 to 8.

Images