Data management method, device, computer equipment and computer readable storage medium

Abstract

The present application relates to a data management method, device, computer equipment and computer-readable storage medium. The method includes: in response to an experiment creation instruction for a deep learning model, obtaining permission verification information corresponding to the experiment creation instruction; the experiment creation instruction is used to create a training experiment for the deep learning model; in the case of passing the verification based on the permission verification information, obtaining the experiment configuration information and data set configuration information corresponding to the experiment creation instruction; the experiment configuration information is used to configure the training parameters of the deep learning model in the experiment database; the data set configuration information is used to determine the training data set of the deep learning model from the experiment database; in the process of training the deep learning model using the training data set and training parameters, determining the training log of the deep learning model; displaying the experimental analysis results of the deep learning model, the experimental analysis results are determined based on the training log. The use of this application can improve the management efficiency of model training data.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data management method, apparatus, computer equipment, and computer-readable storage medium. Background Art

[0002] Deep learning model technology is an important branch of artificial intelligence and an advanced form of machine learning. Deep learning model technology is based on neural networks and automatically extracts features from data and makes predictions or classifications by simulating the learning mechanism of the human brain. In the context of the rapid development of deep learning technology, model management and model training data management are conducive to ensuring the quality and reliability of the model.

[0003] The management of model training data in traditional technologies is vulnerable to attacks, leading to security issues. Furthermore, due to the huge amount of models and model training data, it cannot adapt to the management needs in the production environment, which is not conducive to improving the management efficiency of model training data. Summary of the invention

[0004] Based on this, it is necessary to provide a data management method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical problems, which can improve the management efficiency of model training data.

[0005] In a first aspect, the present application provides a data management method, comprising:

[0006] In response to an experiment creation instruction for a deep learning model, obtaining permission verification information corresponding to the experiment creation instruction; the experiment creation instruction is used to create a training experiment for the deep learning model;

[0007] When the permission verification information passes the verification, the experiment configuration information and the data set configuration information corresponding to the experiment creation instruction are obtained; the experiment configuration information is used to configure the training parameters of the deep learning model in the experiment database; the data set configuration information is used to determine the training data set of the deep learning model from the experiment database;

[0008] In the process of training the deep learning model using the training data set and the training parameters, determining a training log of the deep learning model;

[0009] Displays the experimental analysis results of the deep learning model, which are determined based on the training logs.

[0010] In a second aspect, the present application provides a data management device, comprising:

[0011] A first obtaining module is used to obtain permission verification information corresponding to an experiment creation instruction for a deep learning model in response to the experiment creation instruction; the experiment creation instruction is used to create a training experiment for the deep learning model;

[0012] The second acquisition module is used to obtain the experiment configuration information and data set configuration information corresponding to the experiment creation instruction when the permission verification information passes the verification; the experiment configuration information is used to configure the model training parameters of the deep learning model in the experiment database; the data set configuration information is used to determine the training data set of the deep learning model from the experiment database;

[0013] A determination module, for determining a training log of the deep learning model in the process of training the deep learning model using the training data set and the training parameters;

[0014] The display module is used to display the experimental analysis results of the deep learning model. The experimental analysis results are determined based on the training log.

[0015] In a third aspect, the present application provides a computer device, the computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method when executing the computer program.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above method when executed by a processor.

[0017] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and the computer program implements the steps in the above method when executed by a processor.

[0018] The above-mentioned data management method, apparatus, computer equipment, computer-readable storage medium and computer program product, by responding to the experiment creation instruction, determine the experiment configuration information and the data set configuration information on the premise that the permission verification information corresponding to the experiment creation instruction passes the verification, so as to accurately and timely verify whether the permission corresponding to the experiment creation instruction meets the requirements before creating the training experiment for the deep learning model, and ensure the compliance of the user operation; in the process of training the deep learning model using the training data set and training parameters represented by the experiment configuration information and the data set configuration information, the training log of the deep learning model is determined in time, and the experimental analysis results determined according to the training log are displayed, so as to monitor the training of the deep learning model throughout the life cycle, and can comprehensively and systematically manage the configuration information, training logs and analysis results involved in the training experiment of the deep learning model, optimize the management process of the configuration information, training logs and analysis results, etc., so as to improve the management efficiency of the model training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An application environment diagram of a data management method provided in an embodiment of the present application;

[0020] Figure 2 A flowchart of a data management method provided in an embodiment of the present application;

[0021] Figure 3 An architectural diagram of a data management service provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a process for using a data management service provided in an embodiment of the present application;

[0023] Figure 5 An architectural diagram of a data management system provided in an embodiment of the present application;

[0024] Figure 6 A structural block diagram of a data management device provided in an embodiment of the present application;

[0025] Figure 7 An internal structure diagram of a computer device provided in an embodiment of the present application;

[0026] Figure 8 An internal structure diagram of another computer device provided in an embodiment of the present application;

[0027] Fig. 9 An internal structure diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] The data management method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through a communication network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0030] like Figure 2 As shown, the embodiment of the present application provides a data management method, which is applied to Figure 1 The terminal 102 or the server 104 in the example is used for explanation. It is understandable that the computer device may include at least one of the terminal and the server. The method comprises the following steps:

[0031] S202: In response to an experiment creation instruction for a deep learning model, obtain permission verification information corresponding to the experiment creation instruction.

[0032] Among them, the experiment creation instruction can be used to create a training experiment for a deep learning model.

[0033] Among them, the permission verification information can represent the permission level of the object that triggers the experiment creation instruction. In actual applications, after the user logs in to the user account in the data management system, when the user needs to create a training experiment for a neural network (such as a deep learning model), the user can trigger the experiment creation instruction for the neural network through the terminal. At this time, the permission verification information corresponding to the experiment creation instruction for the neural network may include the permissions pre-assigned to the user account by the data management system.

[0034] In some embodiments, the user can manage the training experiments of the deep learning model and the data involved in the training process through the data management system. When the user needs to create a training experiment for the deep learning model, the user can log in to the user account in the data management system and trigger the experiment creation instruction for the deep learning model through the terminal. In actual applications, the data management system pre-assigns permissions to the user account / user, and the permissions can be used as permission verification information. When the user's permissions allow the user to create a training experiment, the terminal can create a training experiment. In a specific implementation, the terminal can respond to the experiment creation instruction for the deep learning model, and determine the permission verification information corresponding to the experiment creation instruction based on the permissions pre-assigned to the user account by the data management system.

[0035] S204: When the authority verification information passes the verification, the experiment configuration information and the data set configuration information corresponding to the experiment creation instruction are obtained.

[0036] Among them, the experiment configuration information can be used to configure the training parameters of the deep learning model in the experiment database. In practical applications, the experiment configuration information may include but is not limited to the experiment name, experiment description, experiment creation time, experiment indicators, etc.

[0037] Among them, the dataset configuration information can be used to determine the training dataset of the deep learning model from the experimental database. In practical applications, the dataset configuration information may include but is not limited to the dataset name, dataset type, dataset size, dataset storage location, etc.

[0038] In some embodiments, after determining the permission verification information corresponding to the experiment creation instruction, the terminal can verify the permission verification information with the preset experiment creation permission requirements to determine the verification result of the permission verification information, for example: the preset experiment creation permission requirements indicate that the user's permission level is higher than or equal to the preset permission level, and the experiment is allowed to be created. If the permission level represented by the permission verification information corresponding to the experiment creation instruction is higher than or equal to the preset permission level, the verification result of the permission verification information corresponding to the experiment creation instruction indicates that the permission verification information has passed the verification; if the permission level represented by the permission verification information corresponding to the experiment creation instruction is lower than the preset permission level, the verification result of the permission verification information corresponding to the experiment creation instruction indicates that the permission verification information verification has failed / failed. In the case where the permission verification information fails / failed to verify, the terminal can display a prompt message to the user to prompt the user that the user does not have the experiment creation permission. In the case where the permission verification information passes the verification, the user is allowed to create a training experiment. At this time, the terminal can determine the experiment configuration information and data set configuration information corresponding to the experiment creation instruction based on the user operation. For example, the terminal can set the experiment configuration information and data set configuration information corresponding to the experiment creation instruction in response to the edit instruction triggered by the user.

[0039] S206. In the process of training the deep learning model using the training data set and the training parameters, determine the training log of the deep learning model.

[0040] Among them, the training log can refer to a file or data set that records various information and indicators of the deep learning model during the training process.

[0041] In some embodiments, after determining the experimental configuration information and the data set configuration information, the terminal can configure the training parameters of the deep learning model in the experimental database according to the experimental configuration information, and determine the training data set of the deep learning model from the experimental database according to the data set configuration information. The terminal can then input the training data set of the deep learning model into the deep learning model, and train the deep learning model according to the training parameters of the deep learning model. In the process of training the deep learning model using the training data set and training parameters, the terminal can determine the training log of the deep learning model to monitor the process of the deep learning model.

[0042] S208. Display the experimental analysis results of the deep learning model, where the experimental analysis results are determined based on the training log.

[0043] Among them, the experimental analysis results can characterize the experimental conditions of the training experiment of the deep learning model. In practical applications, the experimental analysis results may include experimental anomalies, experimental results, etc.

[0044] In some embodiments, after determining the training log of the deep learning model, the terminal can analyze the training log of the deep learning model, record whether there are experimental anomalies during the training process, the location, time and type of the experimental anomalies, and the experimental results, and integrate the experimental anomalies, experimental results, etc. to obtain the experimental analysis results of the deep learning model.

[0045] It can be seen that in the embodiment of the present application, by responding to the experiment creation instruction, on the premise that the permission verification information corresponding to the experiment creation instruction passes the verification, the experiment configuration information and the data set configuration information are determined, so that before creating the training experiment for the deep learning model, it is accurately and timely verified whether the permission corresponding to the experiment creation instruction meets the requirements, so as to ensure the compliance of the user operation; in the process of training the deep learning model using the training data set and training parameters represented by the experiment configuration information and the data set configuration information, the training log of the deep learning model is determined in time, and the experimental analysis results determined according to the training log are displayed, so as to monitor the training of the deep learning model throughout its life cycle, and can comprehensively and systematically manage the configuration information, training logs and analysis results involved in the training experiment of the deep learning model, optimize the management process of configuration information, training logs and analysis results, etc., so as to improve the management efficiency of model training data.

[0046] In some embodiments, obtaining experimental configuration information and data set configuration information corresponding to the experiment creation instruction includes: obtaining the experimental configuration information and data set configuration information from a cache database corresponding to the preset cache dictionary when the experimental configuration information and data set configuration information corresponding to the experiment creation instruction are found in the preset cache dictionary; or obtaining the experimental configuration information and data set configuration information from the experimental database when the experimental configuration information and data set configuration information corresponding to the experiment creation instruction cannot be found in the preset cache dictionary.

[0047] The preset cache dictionary may be used to search for cache data. In practical applications, the preset cache dictionary may represent a mapping relationship between cache data and data entries corresponding to the cache data.

[0048] The cache database corresponding to the preset cache dictionary may refer to a database used to store cache data corresponding to data entries in the preset cache dictionary.

[0049] Among them, the experimental database can be a pre-built database used to manage information such as experiment name, experiment description, experiment creation time, experiment indicators, data set name, data set type, data set size, data set storage location, etc. In actual applications, the experimental database can include but is not limited to a database built based on Sqlite files.

[0050] In some embodiments, in order to improve the data loading speed, the terminal can store the data that is frequently accessed and does not change much as cache data in the cache database. When it is necessary to find the experimental configuration information and data set configuration information corresponding to the experiment creation instruction, the terminal can compare the experimental configuration information and data set configuration information corresponding to the experiment creation instruction with each data entry in the preset cache dictionary one by one. If the preset cache dictionary contains data entries that match the experimental configuration information and data set configuration information corresponding to the experiment creation instruction, the terminal can determine that the experimental configuration information and data set configuration information corresponding to the experiment creation instruction are found from the preset cache dictionary. At this time, the terminal can obtain the experimental configuration information and data set configuration information in the cache database corresponding to the preset cache dictionary. If the preset cache dictionary does not contain data entries that match the experimental configuration information and data set configuration information corresponding to the experiment creation instruction, the terminal can determine that the experimental configuration information and data set configuration information corresponding to the experiment creation instruction cannot be found from the preset cache dictionary. At this time, the terminal can obtain the experimental configuration information and data set configuration information from the experimental database. By reusing the stored data, unnecessary data transmission and processing can be reduced, thereby optimizing the utilization of system resources, reducing data loading time and improving system response speed, and improving user experience.

[0051] It can be seen that in this embodiment, a cache mechanism can be implemented based on a cache dictionary and a cache database, which can improve the efficiency of obtaining experimental configuration information and data set configuration information and the speed of data loading, reduce latency, and reduce the number of accesses to the database or computing resources, thereby improving the management efficiency of model training data.

[0052] In some embodiments, the above method also includes: based on the number of accesses and change frequencies of each data in the experimental database, screening out data whose number of accesses meets the preset number requirements and whose change frequencies meet the preset frequency requirements as cache data; storing the cache data in the cache database; and generating a preset cache dictionary based on each cache data in the cache database.

[0053] The number of accesses may refer to the total number of times a specific resource is accessed within a certain time range.

[0054] The change frequency may refer to the number of times a specific resource is updated, modified or deleted within a certain time range.

[0055] The preset number requirement may refer to information indicating whether the data is frequently accessed. In practical applications, an access number greater than or equal to a preset number threshold may indicate that the data is frequently accessed.

[0056] The preset frequency requirement may refer to information indicating whether the data changes frequently. In practical applications, a change frequency less than or equal to a preset frequency threshold may indicate that the data does not change much.

[0057] The cache data may refer to data stored in a cache database.

[0058] In some embodiments, the terminal can filter out the data that needs to be cached from the experimental database according to whether the data is frequently accessed and the degree of change of the data and cache it to the cache database. In a specific implementation, the terminal can first determine the number of accesses and the frequency of change of each data in the experimental database within a preset time period. For each data in the experimental database, the terminal can compare the number of accesses to the data with the preset number threshold and the frequency of change of the data with the preset frequency threshold, so as to determine whether the number of accesses to the data meets the preset number requirement and whether the frequency of change of the data meets the preset frequency requirement. If the number of accesses to a certain data meets the preset number requirement and the frequency of change meets the preset frequency requirement, the terminal can store the data as cache data in the cache database. The terminal can generate a preset cache dictionary based on each cache data in the cache database, thereby realizing data caching, so as to query and determine the cache data based on the preset cache dictionary.

[0059] It can be seen that in this embodiment, data is cached based on the number of data accesses and change frequencies, and a cache database can be accurately constructed and a cache dictionary can be determined based on the number of data accesses and change frequencies, so as to achieve accurate cache data query and improve the accuracy of querying cache data.

[0060] In some embodiments, the above method also includes: in response to a data creation instruction for a deep learning model, obtaining an initial data set corresponding to the data creation instruction; obtaining a labeling file for each training sample in the initial data set, and determining a sample label for each training sample based on the labeling file; in response to a filtering instruction for each training sample, filtering out a target sample from each training sample based on the label information corresponding to the filtering instruction and the sample label of each training sample; and generating a data set different from the initial data set based on the target sample.

[0061] The data creation instruction may refer to an instruction / command for creating a training data set in an experimental database.

[0062] The initial data set may refer to a data set selected by the data creation instruction in the experimental database. In practical applications, the initial data set may include a number of training samples and a labeling file corresponding to each training sample.

[0063] The sample labels of the training samples may include the data distribution of the training samples recorded in the annotation file (such as the type of defect labels, the number of each defect, the number of images, etc.).

[0064] The screening instruction may refer to an instruction / command for screening out specific training samples from training samples using sample labels.

[0065] The label information corresponding to the screening instruction may refer to a label used to screen training samples.

[0066] The target sample may refer to a training sample whose sample label matches (eg, is consistent with) the label information corresponding to the screening instruction.

[0067] In some embodiments, when a user creates a training data set in an experimental database, the user can trigger a data creation instruction for a deep learning model by operating a terminal, and the terminal responds to the data creation instruction for the deep learning model to obtain an initial data set corresponding to the data creation instruction. In practical applications, the data creation instruction can be used to determine at least one data set in the experimental database as an initial data set, and the initial data set can be used as a training data set after data screening. When the user needs to perform data screening on the data in the initial data set, the user can trigger a screening instruction for the initial data set by operating a terminal, and the terminal responds to the screening instruction for each training sample, first obtains the annotation file of each training sample in the initial data set, and determines the sample label of each training sample according to the annotation file, and then compares the label information corresponding to the screening instruction with the sample label of each training sample one by one, and screens out the training samples whose sample labels match (such as consistent) the label information corresponding to the screening instruction from each training sample as the target sample, and then the terminal can generate a data set different from the initial data set according to the target sample, and the data set different from the initial data set can be used as the training data set.

[0068] It can be seen that in this embodiment, the data in the initial data set is accurately screened based on the sample labels to generate a training data set. Based on the user's data screening needs, specific data can be accurately screened from the existing data set in the experimental database to create a new data set, so as to perform personalized training for different training experiments and optimize the management efficiency of training data in the experimental database.

[0069] In some embodiments, the above method also includes: in response to a data upload instruction for the experimental database, obtaining a data set to be uploaded corresponding to the data upload instruction; if the hash value of the data set to be uploaded does not match the hash value of each data set in the experimental database, storing the data set to be uploaded in the experimental database.

[0070] The data upload instruction may refer to an instruction / command for transferring data / data set to the experimental database.

[0071] The data set to be uploaded may refer to a set of data that needs to be transmitted to the experimental database.

[0072] The hash value may refer to a character string of fixed length obtained by calculation through a hash function or the like. In practical applications, the hash value may include but is not limited to an MD5 hash value.

[0073] In some embodiments, when a user needs to upload specific data / data set to an experimental database, the user can trigger a data upload instruction for the experimental database by operating the terminal. The terminal can respond to the data upload instruction for the experimental database, first determine the data set to be uploaded corresponding to the data upload instruction, and then determine the hash value of the data set to be uploaded and the hash value of each data set in the experimental database. In actual applications, the terminal can determine the hash value of the data set to be uploaded based on the MD5 algorithm, and the terminal can send a data upload request to the experimental database, and receive the hash value of each data set in the experimental database returned by the experimental database in response to the data upload request. In a specific implementation, the hash value of each data set in the experimental database can be determined in advance based on the MD5 algorithm and stored in a preset data list file. Then the terminal can compare the hash value of the data set to be uploaded with the hash value of each data set in the experimental database one by one to determine whether the hash value of the data set to be uploaded matches (such as consistent) with the hash value of each data set in the experimental database. If the hash value of the data set to be uploaded does not match the hash value of each data set in the experimental database, the terminal can determine that the data set to be uploaded does not exist in the experimental database, and the terminal can store the data set to be uploaded in the experimental database. If the hash value of the data set to be uploaded matches the hash value of any data set in the experimental database, the terminal can determine that the data set to be uploaded already exists in the experimental database, and the terminal can refuse to transfer the data set to the experimental database. By identifying and deleting duplicate data in the data set, saving storage space and reducing redundant data, it can ensure that there are no duplicate records or data items in the data set, thereby improving the consistency and accuracy of the data. Deduplication processing during data upload can reduce the amount of data that needs to be transmitted, thereby saving network bandwidth resources. By optimizing the data storage and query process, it helps enterprises better cope with big data challenges.

[0074] It can be seen that in this embodiment, the hash value of the data set to be uploaded and the hash values ​​of each data set in the experimental database are used to determine whether to upload the data set to be uploaded, so that data deduplication can be performed in time during the data set uploading stage, thereby avoiding redundancy of data in the experimental database, avoiding reduced processing efficiency and a large waste of storage resources caused by duplicate data, thereby improving the management efficiency of data in the experimental database.

[0075] In some embodiments, the above method also includes: in response to an access instruction to the experimental database, obtaining a pass code corresponding to the access instruction; if the pass code passes the verification, executing the target operation corresponding to the access instruction; the target operation includes a data search operation and a data modification operation.

[0076] The access instruction may refer to an instruction / command for executing a target operation on the experimental database. In practical applications, the target operation may include but is not limited to a data search operation and a data modification operation.

[0077] In some embodiments, when a user accesses the experimental database of the data management system, it is necessary to determine whether the user's access is a legitimate access. In actual applications, the data management system is pre-set with a key, which can be a string of 128 characters including uppercase and lowercase letters, special characters, and numbers. The data management system can first determine the user name corresponding to the access instruction, and then generate a pass code based on the key and the user name. The user needs to perform communication code verification / check every time he accesses. The data management system can send the pass code corresponding to the access instruction to the terminal for the user to view. After the terminal obtains the pass code corresponding to the access instruction, the user can enter the pass code through the terminal, and the terminal can send the communication code entered by the user to the data management system. If the pass code sent by the data management system to the terminal is consistent with the pass code sent by the terminal received by the data management system, the terminal can perform target operations such as data search operations and data modification operations corresponding to the access instruction to realize the user's data search, data modification and other operations on the experimental database. In a specific implementation, if the number of pass code verification failures within a preset time period is greater than or equal to a preset threshold, the user terminal will be prohibited from accessing the data management system and the experimental database. For example, the terminal can combine the username of the web page request with the fixed encrypted string in the Flask application, and encrypt it using the hash algorithm MD5 to generate a new access permission passcode. Every time a web page is accessed, the user needs to carry the passcode. If the verification fails, access is denied. If access is denied more than 5 times in one minute, the user's IP address will be prohibited from accessing the service. Encryption of access operations based on keys can ensure that data transmission between users and servers is secure, and prevent sensitive information (such as user authentication information, session status, etc.) from being intercepted or tampered with during transmission. It effectively resists man-in-the-middle attacks and ensures the integrity and confidentiality of data during transmission.

[0078] It can be seen that in this embodiment, timely and accurate verification of access instructions to the experimental database based on the pass code can avoid illegal access, prevent the experimental database from being attacked, and improve the security of the experimental database.

[0079] In some embodiments, the above method also includes: determining the experimental information based on the experimental name information, experimental description information, creation time information, experimental indicator information and experimental status information of the training experiment; determining the data set information based on the data set name information, data set type information, data set size information, storage path information and data set description information of the training data set; building an experimental database based on the experimental information and data set information, and setting the permission verification information of the users of the experimental database.

[0080] The experiment name information may refer to information representing the name of the training experiment.

[0081] The experiment description information may refer to information used to describe the training experiment.

[0082] The creation time information may refer to information representing the creation time of the training experiment.

[0083] The experimental indicator information may refer to information representing the performance indicators of the training experiment.

[0084] The experiment status information may refer to information representing the status / progress of the training experiment.

[0085] The experimental information may refer to a table recording the experiment name, experiment description, creation time, experiment index, and experiment status. In practical applications, the experimental information may be used as an experimental table.

[0086] The data set name information may refer to information representing the name of the data set.

[0087] The data set type information may refer to information representing the type of the data set.

[0088] The data set size information may refer to information representing the size of the data set. In practical applications, the unit of the data set size may include but is not limited to MB, GB, etc.

[0089] The storage path information may refer to information representing the storage path of the data set in the experimental database.

[0090] The data set description information may refer to information used to describe the data set.

[0091] The data set information may include a table recording the data set name, data set type, data set size, storage path and data set description. In practical applications, the data set information may be used as a data set table.

[0092] In some embodiments, the terminal can pre-build an experiment database based on a Sqlite file. In actual applications, the experiment database can contain several Sqlite data tables, which can include experiment tables, data set tables, and user tables. The table name of the experiment table (Experiments) is Experiment, and the fields of the experiment table can include ExperimentID, Name, Description, CreateTime, Metrics, Status, and Dataset. In the specific implementation, ExperimentID is an integer type field, which serves as the primary key of the table and is automatically incremented. ExperimentID is used to uniquely identify each experiment record in the table. Name is a text type field used to store the name of the experiment. This field is not allowed to be empty. Description is a text type field used to store a detailed description of the experiment. Description is optional. CreateTime is a timestamp type field used to record the creation time of the experiment. By default, CreateTime will be set to the current timestamp. Metrics is a text field used to store the performance indicators of the experiment. Metrics is optional. Status is a text field used to store the status of the experiment, including waiting, training, evaluating, completed, error, etc. Status cannot be empty. Dataset is a text field used to store the dataset used in the experiment. Dataset cannot be empty.

[0093] The table name of the dataset table (Datasets) is Dataset. The fields of the dataset table can include DatasetID, Name, Type, Size, StoragePath, and Description. DatasetID is an integer field, which is the primary key of the table and automatically increments. DatasetID is used to uniquely identify each dataset record in the table. Name is a text field, which is used to store the name of the dataset. Name cannot be empty. Type is a text field, which is used to describe the type of the dataset, such as CSV, JSON, Image, etc. Type is optional. Size is an integer field, which is used to store the size of the dataset. The unit can be MB, GB, etc. Size is optional. StoragePath is a text field, which is used to store the storage path of the dataset. StoragePath is optional. Description is a text field, which is used to store a detailed description of the dataset. The Description field is optional.

[0094] The table name of the user table (Users) can be Users. The fields of the user table may include: UserID, Username, Password, CreationDate, and Role. Among them, UserID is an integer type field, which serves as the primary key of the table and is usually set to automatically increment. UserID is used to uniquely identify each user record in the table. Username is a text type field used to store the user's login name or username. Username is usually unique, no duplication is allowed, and it is not allowed to be empty. Password is a text or encrypted string type field used to store the user's password. For security reasons, the password is usually stored encrypted rather than in plain text. CreationDate is a timestamp type field used to record the creation time of the user account. By default, CreationDate will be set to the current timestamp. Role is a text or enumeration type field used to represent the user's role in the system, such as "administrator", "labeler", "algorithmist", etc. Role is optional, but it is very important in systems that need to distinguish user permissions. The terminal can set the permission verification information of the user of the experimental database according to the role. The user table is the core component of most user-based applications and stores information related to user identity and permissions.

[0095] In the specific implementation, the terminal can manage the user's permissions by assigning roles. Each role has specific permissions, and the user obtains the corresponding permissions by assigning roles. Permissions are uniformly assigned in the management system: for example, different access permissions are assigned according to user roles and responsibilities, and the administrator supports all permission operations. Algorithm personnel ensure the security and compliance of data. Their main responsibilities are to be responsible for the overall management and maintenance of the entire experiment or project, and to be responsible for the training and tuning of the model, including data preprocessing, model configuration, and training process monitoring. The specific permissions of algorithm personnel may include permission allocation, system configuration, auditing and monitoring, resource allocation, model training, model tuning, viewing training logs, and limited data set management. Permission allocation: The access rights of each role or user can be set in detail, including access and operation permissions to resources such as data sets, models, and training logs. System configuration: The system configuration can be modified, such as adjusting system parameters, setting security policies, and managing storage resources. Auditing and monitoring: The operation logs of all users can be viewed, the system's operating status can be monitored, and potential security issues or performance bottlenecks can be discovered and handled in a timely manner. Resource allocation: According to the experimental requirements, computing resources and storage resources can be reasonably allocated to ensure the smooth progress of the experiment. Model training: can access and operate training data sets, configure model parameters, start the training process, and monitor various indicators during the training process. Model tuning: according to the training results, tune the model, including adjusting the model structure, optimizing hyperparameters, etc. View training logs: can view detailed logs of the training process, including key indicators such as loss value and accuracy. Limited data set management: with the authorization of the administrator, limited management of the training set can be performed, such as data screening, labeling result review, etc. The main responsibility of the labeler is to be responsible for labeling the data set, such as image classification, target detection, semantic segmentation, etc. The specific permissions of the labeler may include data upload, data viewing, and data download and modification. Among them, data upload: can access and upload the data set that has been sorted, set the description, label and other information of the data set. Data viewing: can view the completion status of the data upload, and after completing the data upload, can submit the upload to the algorithm personnel for viewing and training. Data download and modification: if there are omissions or labeling errors in the data, support data download and modification to adjust the work plan in time. Permission control can ensure data protection and security, prevent unauthorized access, and ensure that sensitive data and key resources are visible and accessible only to necessary users or entities, thereby enhancing data confidentiality. By limiting access rights, the risk of potential security vulnerabilities and illegal activities can be reduced, helping to protect the system from malicious attacks and data tampering. Prevent unauthorized users from modifying, deleting or tampering with data, thereby maintaining data integrity and reliability.

[0096] It can be seen that in this embodiment, constructing the experimental database and the experimental model management table based on Sqlite can achieve unified management and efficient use of data, so as to realize data query function based on the experimental database, support complex SQL query and data analysis operations, thereby extracting valuable information from a large amount of data and experiments, and providing strong support for model analysis optimization.

[0097] In some embodiments, the data management service can manage the experimental database through a data management system, such as Figure 3 As shown in the figure, an architecture diagram of a data management service is provided. The terminals in the data management service can implement data management and experiment management. Data management can perform diversified management on different data sets (such as data set-D01 and data set-D02), and experiment management can perform diversified management on different training experiments (such as experiment-E01 and experiment-E02). Take the example of training a deep learning model based on a training data set containing images through the data management service as follows: Figure 4 As shown, a process diagram of using a data management service is provided. The terminal can clean and annotate the images in the training data set, and then upload the training data set to the experimental database and create a data set D01 in the data management service. The terminal supports users to browse data through web pages. The terminal can create an experiment E01 in the experimental database and select the data set D01 through the terminal. The experiment E01 represents the training of a deep learning model based on the data set D01. When the model training represented by the experiment is completed, the terminal can generate experimental analysis results such as evaluation indicators based on the training log during the training process. The terminal can display the experimental analysis results to the user for the user to view the evaluation indicators. In a specific implementation, the terminal can also respond to the model download instruction triggered by the user to download the deep learning model trained by the training experiment.

[0098] like Figure 5 As shown, an architecture diagram of a data management system is provided, where a terminal can display several pages in an HTML web view, and a controller of the data management system can process page requests and perform operations such as deduplication and storage on data in a Sqlite relational database (such as an experimental database) based on a data table; the controller of the data management system can also process page requests and create experimental tasks based on an experimental (task) table using training data sets and model training parameters in a Sqlite relational database, so as to train a neural network using training experiments.

[0099] In the specific implementation, the data management system can cache the existing data list, data image path list, user information, model experiment list, experiment configuration parameters, etc. that are frequently accessed and do not change much. The expiration time of the cached data is generally set to 48 hours, and the expiration time is determined by the frequency of data changes and access patterns. The data management system can use the LRU (least recently used) algorithm to manage the cache. If a data item has been accessed recently, it is also likely to be accessed again in the future. Therefore, the algorithm will preferentially find and remove the least recently accessed data cache items. Write the cache logic in the Flask application, create a cache dictionary and store it in the system memory; when you need to access data, first check whether the data exists in the cache dictionary through the unique index of the data, generally the experiment name, user name, etc. If there is valid data in the cache, the cached data is returned directly; if it does not exist in the cache or the data has expired, the data is loaded from the database or computing resources, and the data is updated to be stored in the local file system of the server. At the same time, the cache hit rate and performance are monitored and recorded in the log file.

[0100] When users manage data through the data management system, they can select the corresponding dataset folder to be uploaded through the web page. For each file to be uploaded, the data management system will automatically calculate the MD5 hash value of the file on the page. The MD5 hash value is a 128-bit hash value (usually represented by a 32-bit hexadecimal number). For any given input, the MD5 algorithm will produce a unique output. The calculated MD5 value is compared with the data already stored in the data list file. If the MD5 value of the file does not exist in the data list file, it means that this is a new file, and the upload operation is performed. After the upload is completed, the MD5 value of the new file is added to the data list file for future deduplication checks.

[0101] When the data management system performs experimental lifecycle management on the training experiment of the deep learning model, the terminal can display the data creation interface to the user, and the user fills in relevant information according to the prompts, such as the data set name, description, label, data division ratio, etc. (the data management system will automatically divide the data set into training set, validation set and test set according to the ratio, usually 70%:15%:15% or similar ratio). After the terminal successfully uploads the data set to the experimental database in response to the user's upload instruction, the data management system can read the annotation file corresponding to the image through the Python program, select the corresponding defect object box, and draw its corresponding name, area size, grayscale value and other indicators on the original image, and use charts (such as histograms, box plots, etc.) to display the distribution of data (types of defect labels, number of each defect, number of images), provide rich interactive functions, support such as zooming, filtering according to features such as defect type, data export and download, etc., to help users quickly understand data features.

[0102] The terminal can also display the experiment creation page to the user. The user fills in relevant information according to the prompts, such as the purpose of the experiment, and selects the corresponding data set to be trained: set the model learning rate, batch size, training rounds and other training parameters. In the process of training the model using the training set data, the data management system can obtain the training log of the training process every 10 seconds through the Flask application, and calculate the model precision, accuracy, recall and other indicators. In practical applications, accuracy = (TP +TN) / (TP + FP + FN + TN), recall = (TP) / (TP+FN), precision = (TP) / (TP+FP), binary problem score = (TP + TN) / (TP + FP + FN + TN). Among them, TP (True Positives) can refer to true positive examples, that is, the number of samples predicted by the model as positive samples and actually positive samples. TN (True Negatives) can refer to true negative examples, that is, the number of samples predicted by the model as negative samples and actually negative samples. FP (False Positives) can refer to false positives, that is, the number of samples that the model predicts to be positive but are actually negative. FN (False Negatives) can refer to false negatives, that is, the number of samples that the model predicts to be negative but are actually positive. After creating an experiment on the page, the log files in the experimental training are parsed through the Flask application, and the key indicators of the training process are displayed in real time, showing the distribution of the training set and the validation set. The Flask background is created to monitor the experimental process in real time, and record key information such as the start time, running status, and resource usage of the experiment. The log system records every step of the experiment, data changes, and possible error information in detail, providing a basis for subsequent experimental analysis and debugging.

[0103] When the training experiment is completed, Flask sends information to users related to the experiment by calling built-in platform internal messages, etc., to ensure that users can keep abreast of the progress of the experiment. Experimental results collection and organization, design an automated experimental results collection mechanism. When the experiment is completed, Flask starts the event trigger, automatically saves the experimental analysis results (such as model weights, evaluation indicators, etc.) to the path of the experiment folder, and generates an experimental report. By analyzing the training logs during the training process, it can be determined that it is a performance. The terminal can also record the time, location, type and possible causes of the exception in detail to provide a basis for subsequent troubleshooting. It is understandable that for some predictable or common exceptions, the training experiment also designs an automatic recovery mechanism, such as automatically restarting the experiment, reallocating resources, etc., to reduce user intervention and shorten the interruption time of the experiment. Record the entire process and results of exception handling, including processing time, processing methods, processing results, etc., to facilitate users to track problems and subsequent analysis. Exception handling strategies for data anomalies can include: for data anomalies, such as missing data, format errors, etc., data cleaning and verification mechanisms to ensure the quality and integrity of input data. For example, by traversing the data in the data folder through the Flask application, the basic information such as the image size can be read to determine the data integrity. The exception handling strategy for model exceptions can include: for exceptions in the model training process (such as gradient disappearance, gradient explosion, etc.), they can be captured through log keyword search and other methods, and the model parameters can be adjusted on the page to solve the problem. The exception handling strategy for resource exceptions can include: for exceptions caused by insufficient resources or unreasonable resource allocation (such as memory overflow, high GPU occupancy, etc.), the service will increase computing resource investment, add computing cards and restart, reduce the single batch training volume of the model, etc. to solve the problem. For experimental results whose final indicators (such as experimental analysis results) meet the expected results, the terminal can asynchronously package all the data under the experiment in the background, trigger downloads on the web page, and integrate the download results into the use of subsequent stages to ensure the continuity and efficiency of experimental iterations. The experience, lessons and results of the experiment process can be sorted, traced back and archived. The data management system can collect user feedback in a centralized manner on the service page according to a fixed time period (such as monthly or quarterly) to ensure the timeliness of feedback and avoid frequent interruptions to users.

[0104] It is understandable that the data management system can also collect problems and feedback encountered by users during use through the built-in customer service system to ensure that user problems are solved in a timely manner. The feedback is categorized by labels, and statistical analysis is performed on the categorized feedback data to identify the issues that users are most concerned about, the most frequently occurring issues, and the new features that users most expect. Based on the results of data analysis, the administrator prioritizes the feedback issues and determines the issues that need to be solved or optimized first. Furthermore, the data management system also supports keeping up with technological trends, combining the latest technological developments and development trends to ensure that the platform's technical architecture and functional modules always remain advanced, using user feedback as an important basis for iterative upgrades of management tools, and constantly adjusting and optimizing service functions and user experience according to user needs.

[0105] It can be seen that in this embodiment, data redundancy is avoided, the use of storage space is reduced, and query efficiency is improved through data table design. Constraints (such as primary keys, foreign keys, etc.) are added to ensure the integrity and consistency of data and reduce data errors and anomalies. Make data more orderly and easy to manage, which is convenient for data analysis and mining. At the same time, Sqlite, as a lightweight, embedded relational database management system, can be directly embedded in the application without the need for an independent database server, and can run on a variety of operating systems, thereby simplifying the development process and reducing deployment and management costs. With the continuous development of management solutions, Sqlite's data table design can be expanded and adjusted as needed to meet new business needs. Sqlite database files can be directly backed up and restored, which is conducive to ensuring the security and reliability of data; through the full-cycle management of experiments, comprehensive and systematic management can be carried out from the planning, execution, data analysis to the application of results, which can improve the efficiency and accuracy of experiments and standardize the process. The full-cycle management of experiments ensures that each experimental step is carried out in accordance with established specifications by formulating standardized experimental processes, reducing errors caused by human operations and improving the accuracy and repeatability of experiments; full-cycle management can also reasonably allocate the equipment, materials, manpower and other resources required for the experiment to avoid resource waste and improve resource utilization efficiency. Strengthen data management and analysis. During the full-cycle management of experiments, all data are centrally managed and stored, which is convenient for subsequent data analysis and mining. Through professional data analysis tools and methods, experimental data can be processed quickly and accurately, valuable information can be extracted, and strong support can be provided for the interpretation and application of experimental results. Provide result collation tools to help users quickly generate experimental reports, including experimental parameters, model performance comparison charts, etc., to facilitate users to analyze and summarize results.

[0106] It should be understood that, although the steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0107] Based on the same inventive concept, the embodiment of the present application also provides a data management device. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data management device embodiments provided below can refer to the limitations on the data management method above, and will not be repeated here.

[0108] like Figure 6 As shown, the embodiment of the present application provides a data management device 600, including:

[0109] A first obtaining module 602 is used to obtain permission verification information corresponding to an experiment creation instruction for a deep learning model in response to the experiment creation instruction; the experiment creation instruction is used to create a training experiment for the deep learning model;

[0110] The second obtaining module 604 is used to obtain the experiment configuration information and the data set configuration information corresponding to the experiment creation instruction when the permission verification information passes the verification; the experiment configuration information is used to configure the training parameters of the deep learning model in the experiment database; the data set configuration information is used to determine the training data set of the deep learning model from the experiment database;

[0111] A determination module 606 is used to determine a training log of the deep learning model during the process of training the deep learning model using the training data set and the training parameters;

[0112] The display module 608 is used to display the experimental analysis results of the deep learning model, and the experimental analysis results are determined based on the training log.

[0113] In some embodiments, in terms of obtaining the experimental configuration information and the data set configuration information corresponding to the experimental creation instruction, the second obtaining module 604 is specifically used for: when the experimental configuration information and the data set configuration information corresponding to the experimental creation instruction are found in the preset cache dictionary, obtaining the experimental configuration information and the data set configuration information from the cache database corresponding to the preset cache dictionary; or, when the experimental configuration information and the data set configuration information corresponding to the experimental creation instruction cannot be found in the preset cache dictionary, obtaining the experimental configuration information and the data set configuration information from the experimental database.

[0114] In some embodiments, the device also includes a data cache module, which is used to: based on the number of accesses and change frequencies of each data in the experimental database, filter out data whose access times meet the preset number requirements and whose change frequencies meet the preset frequency requirements as cache data; store the cache data in the cache database; and generate a preset cache dictionary based on each cache data in the cache database.

[0115] In some embodiments, the device also includes a data creation module, which is used to: respond to data creation instructions for a deep learning model, obtain an initial data set corresponding to the data creation instructions; obtain a labeling file for each training sample in the initial data set, and determine a sample label for each training sample based on the labeling file; respond to a screening instruction for each training sample, and screen out a target sample from each training sample based on the label information corresponding to the screening instruction and the sample label of each training sample; and generate a data set different from the initial data set based on the target sample.

[0116] In some embodiments, the device also includes a data upload module, which is used to: respond to a data upload instruction for an experimental database, obtain a data set to be uploaded corresponding to the data upload instruction; if the hash value of the data set to be uploaded does not match the hash value of each data set in the experimental database, store the data set to be uploaded in the experimental database.

[0117] In some embodiments, the device also includes an access verification module, which is used to: in response to an access instruction to an experimental database, obtain a pass code corresponding to the access instruction; if the pass code passes the verification, execute the target operation corresponding to the access instruction; the target operation includes a data search operation and a data modification operation.

[0118] In some embodiments, the device also includes a database construction module, which is used to: determine the experimental information based on the experimental name information, experimental description information, creation time information, experimental indicator information and experimental status information of the training experiment; determine the data set information based on the data set name information, data set type information, data set size information, storage path information and data set description information of the training data set; build an experimental database based on the experimental information and data set information, and set the permission verification information of the user of the experimental database.

[0119] Each module in the above data management device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0120] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data related to data management. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above-mentioned data management method are implemented.

[0121] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 8As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, the steps in the above-mentioned data management method are implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen; the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0122] Those skilled in the art will understand that Figure 7 or Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] In some embodiments, a computer device is provided. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the steps in the above method embodiments are implemented.

[0124] In some embodiments, Fig. 9 The figure shows an internal structure diagram of a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0125] In some embodiments, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0126] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0127] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data management method, characterized in that: include: In response to an experiment creation instruction for a deep learning model, obtaining permission verification information corresponding to the experiment creation instruction; The experiment creation instruction is used to create a training experiment for the deep learning model; In the case where the permission verification information passes the verification, the experiment configuration information and the data set configuration information corresponding to the experiment creation instruction are obtained; the experiment configuration information is used to configure the training parameters of the deep learning model in the experiment database; the data set configuration information is used to determine the training data set of the deep learning model from the experiment database; In the process of training the deep learning model using the training data set and the training parameters, determining a training log of the deep learning model; The experimental analysis results of the deep learning model are displayed, where the experimental analysis results are determined based on the training log.

2. The method according to claim 1, characterized in that The obtaining of the experiment configuration information and the data set configuration information corresponding to the experiment creation instruction includes: When the experiment configuration information and the data set configuration information corresponding to the experiment creation instruction are found in the preset cache dictionary, the experiment configuration information and the data set configuration information are obtained from the cache database corresponding to the preset cache dictionary; or When the experiment configuration information and the data set configuration information corresponding to the experiment creation instruction cannot be found in the preset cache dictionary, the experiment configuration information and the data set configuration information are obtained from the experiment database.

3. The method according to claim 2, characterized in that The method further comprises: According to the access times and change frequencies of each data in the experimental database, data whose access times meet the preset times requirement and whose change frequencies meet the preset frequency requirement are selected from each data as cache data; The cache data is stored in the cache database; and the preset cache dictionary is generated according to each cache data in the cache database.

4. The method according to claim 1, characterized in that: The method further comprises: In response to a data creation instruction for the deep learning model, obtaining an initial data set corresponding to the data creation instruction; Obtaining a labeling file for each training sample in the initial data set, and determining a sample label for each training sample according to the labeling file; In response to a screening instruction for each of the training samples, screening out a target sample from each of the training samples according to label information corresponding to the screening instruction and sample labels of each of the training samples; A data set different from the initial data set is generated according to the target sample.

5. The method according to claim 1, characterized in that The method further comprises: In response to a data upload instruction for the experimental database, obtaining a data set to be uploaded corresponding to the data upload instruction; When the hash value of the data set to be uploaded does not match the hash value of each data set in the experimental database, the data set to be uploaded is stored in the experimental database.

6. The method according to claim 1, characterized in that The method further comprises: In response to an access instruction to the experimental database, obtaining a pass code corresponding to the access instruction; When the pass code passes the verification, the target operation corresponding to the access instruction is executed; the target operation includes a data search operation and a data modification operation.

7. The method according to claim 1, characterized in that The method further comprises: Determine the experiment information according to the experiment name information, experiment description information, creation time information, experiment indicator information and experiment status information of the training experiment; Determine the dataset information according to the dataset name information, dataset type information, dataset size information, storage path information and dataset description information of the training dataset; The experimental database is constructed according to the experimental information and the data set information, and permission verification information of a user of the experimental database is set.

8. A data management device, characterized in that: include: A first obtaining module is used to obtain permission verification information corresponding to an experiment creation instruction for a deep learning model in response to the experiment creation instruction; The experiment creation instruction is used to create a training experiment for the deep learning model; A second acquisition module is used to obtain the experiment configuration information and data set configuration information corresponding to the experiment creation instruction when the permission verification information passes the verification; the experiment configuration information is used to configure the training parameters of the deep learning model in the experiment database; the data set configuration information is used to determine the training data set of the deep learning model from the experiment database; A determination module, configured to determine a training log of the deep learning model during a process of training the deep learning model using the training data set and the training parameters; A display module is used to display the experimental analysis results of the deep learning model, where the experimental analysis results are determined based on the training log.

9. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images