Data set management method and device, model evaluation method and device, equipment and medium

Abstract

The invention discloses a data set management method and device, a model evaluation method and device, equipment and a medium, and is applied to the technical field of computers, and the method comprises the following steps: loading each data set file into a memory, and generating a data set data block of each data set file in a cache queue; the data set data block comprises a data dictionary constructed based on a target hierarchy and a structure dictionary constructed based on data items; searching a target data set data block from a cache queue according to the data set number; and according to the query condition, obtaining target data by utilizing the data dictionary and the structure dictionary in the data block of the target data set. The directory hierarchy and the data items are separated, and the structure dictionary and the data dictionary are respectively constructed and stored in the cache queue without depending on the database, so that the problem of incompatibility of data sets does not exist, the query is not influenced by the performance of the database, and due to the fact that the structure dictionary and the data dictionary are stored in the cache queue, the query efficiency is greatly improved. Therefore, the access speed is extremely high, and the data query speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data set management method and a model evaluation method, device, equipment and medium. Background Art

[0002] The existing data sets are generally stored in the database, and the database query function is used to filter the qualified data through specific fields. For example, using a relational database or a non-relational database, each piece of data is stored as a record, and the classification information is recorded by field. In this way, data query will be very dependent on the database. When the amount of data in the data set is large, the database performance will become a bottleneck for data query efficiency. In addition, the database table structure is strongly related to the evaluation data structure, so the database table structure needs to be planned and designed according to the evaluation data structure. Therefore, there will be multiple data sets that are not compatible in the same database.

[0003] Therefore, existing data set management methods have problems such as low data query efficiency, limited queries, and database incompatibility. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a data set management method and a model evaluation method, device, equipment and medium, which solve the problems of low data query efficiency, limited query and database incompatibility in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a data set management method, comprising:

[0006] Load each data set file into the memory, and generate a data set data block of each data set file in the cache queue; the data set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item;

[0007] Searching for a target data set data block from the cache queue according to the data set number;

[0008] According to the query conditions, the target data is acquired using the data dictionary and the structure dictionary in the target data set data block.

[0009] On the one hand, each data set file is loaded into the memory, and a data set data block of each data set file is generated in the cache queue, including:

[0010] Building a structure dictionary based on the directory hierarchy of the data set file; the key of the structure dictionary is the title name of the directory hierarchy, the value of the structure dictionary is the memory block of the directory hierarchy, and the memory block of the directory hierarchy includes the directory name, the unique identifier list and the structure dictionary of the next level directory;

[0011] Building a data dictionary based on the data items of the data set file; the key of the data dictionary is a unique identifier, the value of the data dictionary is a memory block of the data item, and the memory block of the data item includes all data of the data item and the directory path where the data item is located;

[0012] Add the key of the data dictionary to the unique identifier list of the target memory block to obtain a data set data block containing the data dictionary and the structure dictionary; the target memory block is the memory block of the directory level and all upper directories corresponding to each unique identifier;

[0013] The data block of the data set is written into the cache queue.

[0014] On the one hand, according to the query condition, the target data is obtained by using the data dictionary and the structure dictionary in the data block of the target data set, including:

[0015] If the target data is not obtained by using the data dictionary and the structure dictionary in the data block of the target data set according to the query condition, the target data is searched from the cache database; all data of the data item stored in the cache database and the directory path where the data item is located are stored in the cache database;

[0016] If the target data is not found in the cache database, the target data set file corresponding to the data set number is loaded into the memory, and a data set data block of the target data set file is generated in the cache queue.

[0017] On the one hand, after generating the data set data block of the target data set file in the cache queue, the method further includes:

[0018] Writing the data block of the data set to the head of the cache queue, and determining whether the length of the cache queue exceeds a preset length;

[0019] If exceeded, the data dictionary of the excess portion at the tail of the cache queue is deleted.

[0020] On the one hand, according to the query condition, the target data is obtained by using the data dictionary and the structure dictionary in the data block of the target data set, including:

[0021] When the query condition is to randomly select data from the target directory, locating a unique identifier list corresponding to the target directory in the structure dictionary;

[0022] Select a target unique identifier from the list of unique identifiers;

[0023] The corresponding target data is searched from the data dictionary according to the target unique identifier.

[0024] The present invention also provides a model evaluation method, comprising:

[0025] Load each evaluation set file into the memory, and generate an evaluation set data block of each evaluation set file in the cache queue; the evaluation set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item;

[0026] Searching for a target evaluation set data block from the cache queue according to the evaluation set number;

[0027] According to the query conditions, the target evaluation data is obtained by using the data dictionary and the structure dictionary in the target evaluation set data block;

[0028] The target evaluation data is used to evaluate the model to be evaluated to obtain an evaluation result.

[0029] The present invention also provides a data set management device, comprising:

[0030] A data set data block generation module is used to load each data set file into the memory and generate a data set data block of each data set file in the cache queue; the data set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item;

[0031] A data block search module, used for searching a target data set data block from the cache queue according to the data set number;

[0032] The data acquisition module is used to acquire target data according to the query conditions by using the data dictionary and the structure dictionary in the data block of the target data set.

[0033] The present invention also provides a model evaluation device, comprising:

[0034] An evaluation set data block generation module is used to load each evaluation set file into the memory and generate an evaluation set data block for each evaluation set file in the cache queue; the evaluation set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item;

[0035] A target evaluation set data block search module, used to search the target evaluation set data block from the cache queue according to the evaluation set number;

[0036] A target evaluation data acquisition module is used to acquire target evaluation data using a data dictionary and a structure dictionary in the target evaluation set data block according to a query condition;

[0037] The model evaluation module is used to evaluate the model to be evaluated using the target evaluation data to obtain the evaluation results.

[0038] The present invention also provides a device, comprising:

[0039] Memory for storing computer programs;

[0040] A processor is used to implement the above-mentioned method of data set management and / or model evaluation when executing the computer program.

[0041] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the method of data set management and / or model evaluation as described above is implemented.

[0042] The present invention also provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned method for data set management and / or model evaluation.

[0043] It can be seen from the above technical solution that the present invention generates a data set data block of each data set file in a cache queue by loading each data set file into the memory; the data set data block includes a data dictionary constructed based on the target hierarchy and a structure dictionary constructed based on the data item; the target data set data block is searched from the cache queue according to the data set number; and the target data is obtained by using the data dictionary and structure dictionary in the target data set data block according to the query condition. The beneficial effect of the present invention is that the directory hierarchy and the data item are separated, the structure dictionary and the data dictionary are constructed respectively, and the whole is stored as a data set data block in the cache queue. Such a storage method does not rely on the database, so there is no problem of data set incompatibility, and the query will not be affected by the database performance; because the structure dictionary and the data dictionary are stored in the cache queue, the access speed is extremely fast, which speeds up the query speed.

[0044] In addition, the present invention also provides a model evaluation method, a data set management device, a model evaluation device, equipment and medium, which also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0046] Figure 1 A flowchart of a data set management method provided by an embodiment of the present invention;

[0047] Figure 2 A flow chart of a model evaluation method provided by an embodiment of the present invention;

[0048] Figure 3 A flowchart of a method for uploading an evaluation set file provided in an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of the structure of a data set management device provided by an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of the structure of a model evaluation device provided by an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of a model evaluation architecture provided by an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of the structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] First, some nouns involved in this application are analyzed:

[0055] NLP model (Natural Language Processing, natural language processing model);

[0056] LLM model (Large Language Model);

[0057] uuid (Universally Unique Identifier, unique identifier);

[0058] The MD5 value is a 128-bit hash value generated by a specific algorithm. It is used to ensure the integrity and consistency of information transmission and serves as a unique identifier for files.

[0059] Please refer to Figure 1 , Figure 1 A flowchart of a data set management method provided by an embodiment of the present invention. The method may include:

[0060] S101: Load each data set file into the memory, and generate a data set data block of each data set file in a cache queue.

[0061] The execution subject of this embodiment is a management terminal. In this embodiment, by loading the data set file into the memory, the corresponding data set data block is generated in the cache queue of the memory, and the data set data block includes a data dictionary built based on the target level and a structure dictionary built based on the data item. The data set file in this embodiment is a file with multiple directory levels. The data set file in this embodiment can be an evaluation set file for model evaluation; or it can also be other data set files. The data set file of this embodiment can be in json, xml, csv and other formats, generally containing multi-level titles, and containing multiple data under the lowest level title. For example, the evaluation set file in json format contains a first-level directory, a second-level directory and a third-level directory, and the third-level directory contains multiple data items. The data item contains four fields: question, answer, objective (question type), and uuid (unique identifier), where uuid is the ID (identity identification number, i.e., the number of this embodiment) of the data item in the evaluation set. uuid needs to ensure uniqueness in this evaluation set. If the data item passed in by the user does not contain this field, a temporary uuid is automatically generated.

[0062] Furthermore, the above-mentioned process of loading each data set file into the memory and generating a data set data block of each data set file in the cache queue may specifically include the following steps:

[0063] Step 21: Build a structure dictionary based on the directory hierarchy of the data set file; the key of the structure dictionary is the title name of the directory hierarchy, and the value of the structure dictionary is the memory block of the directory hierarchy. The memory block of the directory hierarchy includes the directory name, the unique identifier list and the structure dictionary of the next level directory.

[0064] Specifically, the structure of the structure dictionary is a hierarchical tree structure. The key of the structure dictionary is the title name of the directory level, and the value is name (directory name), uuids (unique identifier list), sub_subject (structure dictionary of the next level directory). The key in sub_subject is the title name of the subdirectory, and the value is the structure of the subdirectory. The structure is composed of a group of different types of data items, which is used to combine multiple related variables for management and operation. The structure here refers to the name, uuids, and sub_subject (structure dictionary of the next level directory) of the subdirectory. The structure of the directory level is nested layer by layer to form a tree structure, which is convenient for management and search by directory. The function of the structure dictionary is to store and manage the directory level of the data set, which is convenient for browsing by directory or filtering by hierarchical organization. Each level of directory contains a list of unique identifiers for all data items at that level, which is convenient for querying data items under the directory.

[0065] Step 22: Build a data dictionary based on the data items in the data set file; the key of the data dictionary is a unique identifier, the value of the data dictionary is a memory block of the data item, and the memory block of the data item includes all the data of the data item and the directory path where the data item is located.

[0066] Specifically, the structure of the data dictionary is: the key is the uuid of each data item, and the value is the memory block of the data item, which contains various data. If the data set is an evaluation set, the memory block is the data content such as question, answer, objective, uuid and the directory path where it is located (such as subject_0, subject_1, subject_2, etc.). The role of the data dictionary is to quickly find and obtain the detailed content of the data item without relying on the directory hierarchy. All information of the data item can be directly accessed through the uuid.

[0067] Step 23: Add the key of the data dictionary to the unique identifier list of the target memory block to obtain a data set data block containing the data dictionary and the structure dictionary; the target memory block is the memory block of the directory level and all parent directories corresponding to each unique identifier.

[0068] Specifically, the key (i.e., uuid) of the data dictionary is added to the memory block of the corresponding level of the structure dictionary and the uuids of all upper-level memory blocks. In other words, the uuid of the data dictionary is filled into the corresponding uuids in the structure dictionary. This realizes the association between the data dictionary and the structure dictionary.

[0069] Step 24: Write the data set data block into the cache queue.

[0070] Each dataset file generates a dataset data block and writes it to the cache queue.

[0071] Furthermore, after constructing the data dictionary based on the data items in the data set file, the following steps may also be included:

[0072] Store all data of the data item and the directory path where the data item is located in the cache database.

[0073] Specifically, after the data dictionary is constructed, this embodiment adds the directory path where the data item is located (such as subject_0, subject_1, subject_2, etc.) and the specific content of the data item to the cache database (such as redis). In this way, the cache database is loaded with the detailed content of all data items in the data set, with uuid as the key of the data item and the data content as the value. The purpose of this operation is to access the cache queue data faster, so as to avoid the data being deleted in the cache queue when querying the data, and then it can be searched from the cache database.

[0074] S102: Searching for a target data set data block from the cache queue according to the data set number.

[0075] Specifically, when a data query instruction is received, the dataset number is parsed from the query instruction, and whether the corresponding dataset data block exists in the cache queue is determined based on the dataset number; if so, the target data can be obtained based on the structure dictionary and data dictionary in the corresponding target dataset database; if not, the dataset file corresponding to the dataset number needs to be reloaded, and the target dataset database has been generated in the cache queue.

[0076] S103: According to the query condition, the target data is acquired using the data dictionary and the structure dictionary in the data block of the target data set.

[0077] This embodiment does not limit the query condition. For example, it may be all data items under the top-level directory; or it may be to obtain a specified number of data items from any directories under the multi-level directory.

[0078] Further, the above method of acquiring the target data by using the data dictionary and the structure dictionary in the data block of the target data set according to the query condition may specifically include the following steps:

[0079] Step 31: When the query condition is to randomly select data from the target directory, locate the unique identifier list corresponding to the target directory in the structure dictionary;

[0080] Step 32: Select a target unique identifier from the unique identifier list;

[0081] Step 33: Search the corresponding target data from the data dictionary according to the target unique identifier.

[0082] Specifically, the target directory is determined according to the query conditions. According to the target directory, the unique identifier list is located in the structure dictionary, and then the target unique identifier that meets the screening conditions is further determined from the unique identifier list. According to the target unique identifier, the specific target data is searched from the data dictionary and returned. If the query condition is a specific data item, the specific content of the data item can be directly obtained from the data dictionary. Similarly, this data set management method is not only applicable to conditional screening, but also to paging display and statistics of data items in multi-layer directories. For example, paging display and quick statistics of data items under multi-level directories are realized through the unique identifier list in the structure dictionary. Each node in the directory structure records the unique identifier list of the data items it contains, which can easily realize hierarchical statistics and paging display to meet the needs of users to view data in different directories.

[0083] Further, the above method of acquiring the target data by using the data dictionary and the structure dictionary in the data block of the target data set according to the query condition may specifically include the following steps:

[0084] Step 41: If the target data is not obtained by using the data dictionary and structure dictionary in the data block of the target data set according to the query condition, the target data is searched from the cache database; all data of the data item stored in the cache database and the directory path where the data item is located are stored in the cache database;

[0085] Step 42: If the target data is not found in the cache database, the target data set file corresponding to the data set number is loaded into the memory, and a data set data block of the target data set file is generated in the cache queue.

[0086] Specifically, this embodiment uses the data set file as the original data source, the data dictionary of the data set data block as the first-level cache, and the cache database as the second-level cache. The system only needs to read the data set file once during operation, reducing disk I / O or file server access operations. Since the data dictionary is stored in the memory, its access speed is extremely fast, so the data item is first in the data dictionary of the cache queue. If the data item already exists, the result can be returned directly, avoiding any disk or network delays. This allows frequently queried data items to be quickly accessed, greatly improving the response speed. When a data item does not exist in the data dictionary, the cache database will be queried, because the cache database is a second-level cache, and the data is stored in the memory, which has a lower read latency. If the data item exists in the cache database, the system will load it into the data dictionary for subsequent use. The cache database can quickly locate data through key-value storage, and the query speed is also very high.

[0087] In step 42, the target data set file corresponding to the data set number is loaded into the memory, and the data set data block of the target data set file is generated in the cache queue. For this part, reference can be made to the specific operations of the above steps S21 to S24. For ease of understanding, this embodiment also provides a reference to the construction of the tree structure understanding dictionary, which can be referred to as follows:

[0088] (1) Initialize the structure: Create an empty structure dictionary and data dictionary to store the loaded content.

[0089] (2) Starting from the top-level directory, recursively traverse each level of the directory and read the data items of the dataset file level by level.

[0090] (3) Recursive termination condition: When there are no more subdirectories under a directory, return to the upper layer and continue to traverse other contents until all directories and data items have been traversed.

[0091] In the process of recursively traversing directories at all levels and reading data items of data set files level by level, the contents of data dictionary and structure dictionary are generated, as follows:

[0092] Process non-leaf nodes, where non-leaf nodes can be understood as top-level directories and middle-level directories: create a memory block corresponding to the node, including name, uuids, and sub_subject. Set name to the title name of the current node, uuids to an empty list, and sub_subject to an empty structure dictionary. Traverse its child nodes and continue recursive traversal as a new level, enter the subdirectory and build a new directory structure memory block, including name, uuids, and sub_subject. After the subdirectory is loaded, add its memory block to the sub_subject of the parent directory. The key is the title name of the child node, and the value is the memory block corresponding to the child node.

[0093] Process the leaf node. The leaf node here can be a bottom-level directory, that is, there is no subdirectory under this node. When traversing to this node, read the contents of the data items one by one: such as extracting fields such as uuid, question, answer, objective, etc., and create a memory block containing this information. In the data dictionary, store the memory block of the data item with uuid as the key. And add the uuid to the memory block of the corresponding level of the structure dictionary and all the upper-level memory blocks uuids.

[0094] Furthermore, after generating the data set data blocks of the target data set file in the cache queue, the following steps may be further included:

[0095] Step 51: writing the data set data block to the head of the cache queue, and determining whether the length of the cache queue exceeds a preset length;

[0096] Step 52: If exceeded, the data dictionary of the excess portion at the tail of the cache queue is deleted.

[0097] Specifically, when loading a data set file in a data query, the generated data set data block is placed at the head of the cache queue for easy query; and the length of the cache queue is determined. If it exceeds the preset length, the data at the tail of the cache queue is deleted. Since the data dictionary occupies a large amount of memory, the data dictionary that exceeds the tail is deleted first. In this way, when a data set is used, the corresponding data set data block is adjusted to the head of the cache queue to ensure that the data block of the data set with high popularity is at the head of the cache queue and the data block of the data set with low popularity is at the tail of the cache queue. If the length of the cache queue reaches the limit when a new data set file is loaded, the memory space is cleared by clearing the data dictionary to reduce cache occupancy.

[0098] Furthermore, after writing the data block of the data set to the head of the cache queue, the following steps may also be included:

[0099] Step 61: Count the query times and query time of the data set data blocks in the cache queue;

[0100] Step 62: adjusting the order of the data set data blocks in the cache queue according to the query number and the query time to obtain a new cache queue;

[0101] Step 63: When the length of the cache queue exceeds the preset length, the data dictionary of the excess portion at the tail of the new cache queue is deleted.

[0102] After writing a new data set data block to the head of the cache queue, this embodiment does not continuously count the query times and query time of each data set data block in the cache queue, but adjusts the position according to the query times and query time. In this way, when the length of the cache queue exceeds the preset length, the data set data blocks with low access popularity can be deleted first, that is, the data dictionary of the data set data blocks with low query times and long query time intervals can be deleted first. In this way, while effectively releasing the cache space, it can also reduce the deletion of data sets with high access popularity from the cache queue, further reducing the loading process of the data set file, greatly improving efficiency and reducing query time.

[0103] The data set management method provided by the embodiment of the present invention is applied, by loading each data set file into the memory, generating a data set data block of each data set file in the cache queue; the data set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item; searching the target data set data block from the cache queue according to the data set number; and obtaining the target data using the data dictionary and structure dictionary in the target data set data block according to the query condition. The beneficial effect of the method is that the directory level and the data item are separated, the structure dictionary and the data dictionary are constructed respectively, and the whole is stored in the cache queue as a data set data block. Such a storage method does not depend on the database, so there is no problem of data set incompatibility, and the query will not be affected by the database performance; since the structure dictionary and the data dictionary are stored in the cache queue, the access speed is extremely fast, which speeds up the query speed. In addition, the user can select a specific number of questions under the major category, or go deep into the sub-directory for refined selection, so as to meet the flexible and on-demand random selection of data items in the multi-level directory, simplify the selection operation under the complex data structure, and meet the personalized needs of the user. In addition, each directory node maintains a list of unique identifiers, so data items can be randomly extracted on demand without accessing the specific content, which improves efficiency. This random selection logic is simple and fast, does not require global search, avoids repeated traversal of data, and can quickly access data item details. Compared with traversing the tree structure layer by layer to find data, this method is more direct and efficient.

[0104] NLP model evaluation has gradually moved from the initial single accuracy evaluation to a multi-dimensional and comprehensive evaluation method to adapt to the increasingly complex application needs. In the future, as the application of models in various fields continues to deepen, the evaluation methods will be further improved and may continue to develop in terms of multi-language support, real-time, and adaptability to low-resource conditions. At the same time, the evaluation set is constantly evolving to adapt to the new capabilities and diverse application scenarios of the LLM model. The current status of the evaluation set reflects the needs of multi-task coverage, multi-language support, and real-world applications. The evaluation is developing towards multi-dimensionality, dynamics, and multi-language. In the future, NLP model evaluation sets may further emphasize cross-domain, multi-task, and real-time dynamic evaluation, covering more low-resource languages ​​and real-world scenarios, in order to comprehensively measure the applicability, reliability, and fairness of the model in practical applications. Most NLP evaluation sets are now organized in a multi-level directory structure to better manage test data of different tasks, languages, fields, difficulty, and versions. Multi-level directories help to clearly display the content hierarchy of the evaluation set, allowing researchers and developers to more easily select and use appropriate subsets. Common data formats of evaluation set files include JSON, CSV, TSV, XML, and Parquet to adapt to different storage requirements and application scenarios. At the same time, with the increase in task complexity and diversity, dynamic evaluation sets and cross-domain datasets are also developing rapidly to support more comprehensive model performance evaluation. The evaluation set is a typical multi-level directory structure. Divide by task: For example, the evaluation set is divided into major categories such as text classification, sentiment analysis, reading comprehension, and machine translation, and each task has sub-tasks. Divide by language or field: Multilingual evaluation sets are classified according to language, such as English, Chinese, and French; cross-domain evaluation sets may be divided according to application scenarios, such as news, medical, and legal. Divide by difficulty or version: Some evaluation sets are divided according to the difficulty of the dataset (such as "easy", "difficult") or version (such as "v1", "v2") to meet the research needs at different stages.

[0105] The existing method is to store the evaluation set in a database, use the database query function to filter the qualified data through specific fields, use a relational database (such as MySQL, PostgreSQL) or a NoSQL database (such as MongoDB), store each data as a record, and record classification information by field. Question (that is, the content in the data item is the question, so the data item in the evaluation set is called the question item) query relies on the database. When the data in the evaluation set is large, the database performance will become a bottleneck for query efficiency. The database table structure is strongly related to the evaluation data structure, and needs to be planned and designed according to the evaluation data structure. It does not support dynamic data structures. If one evaluation set data contains four fields: question, answer, length, and label, and another evaluation set contains four fields: question, answer, timestamp, and emotional, then the two evaluation sets cannot be compatible in one database table. Therefore, in order to solve the above-mentioned problems existing in the management of evaluation sets, the present invention provides a model evaluation method, please refer to Figure 2 , Figure 2 A flow chart of a model evaluation method provided by an embodiment of the present invention. The method may include:

[0106] S201: Load each evaluation set file into the memory, and generate an evaluation set data block of each evaluation set file in the cache queue; the evaluation set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item;

[0107] S202: searching for a target evaluation set data block from a cache queue according to the evaluation set number;

[0108] S203: according to the query condition, the target evaluation data is obtained by using the data dictionary and the structure dictionary in the target evaluation set data block;

[0109] S204: Use the target evaluation data to evaluate the model to be evaluated and obtain the evaluation result.

[0110] The present embodiment is a specific application scenario of the above-mentioned data set management method, which is for the management of evaluation sets in model evaluation. The data set in the above-mentioned data set management method is specifically an evaluation set. Steps S201 to S203 in this embodiment can refer to the above-mentioned data set management method, and will not be repeated here. In this embodiment, the evaluation set refers to a set of test data and standards used to evaluate the performance of a specific model or system. The evaluation set generally includes multiple levels of directories, and each level of directory may contain the next level of directory (subdirectory) or specific question items. Question items contain fields such as question, answer, objective, and uuid. The structure of the evaluation set is a tree hierarchy, and the title of each level represents the directory node, and the bottom level contains specific question data.

[0111] Furthermore, in this embodiment, the evaluation set file uploaded by the user is stored in the file management server in advance, so that the evaluation set file can be loaded from the file management server. Figure 3 , Figure 3 The flowchart of a method for uploading an evaluation set file provided by an embodiment of the present invention may specifically include:

[0112] (1) Upload evaluation set information and files: The user submits the evaluation set file and related information to the system. The system receives the evaluation set file and prepares to enter the data processing flow.

[0113] (2) File format verification: The assessment set management module verifies the format of the uploaded assessment set file to ensure that the file conforms to the format required by the system (such as JSON, CSV, etc.) and checks the integrity of the content. If the verification fails, the process terminates and the user is prompted to modify the file format and upload it again.

[0114] (3) Total statistical data: After the format check is passed, the system will perform statistics on the evaluation set data, such as calculating the total number of data items, for subsequent operations.

[0115] (4) Upload to the file management server: The system uploads the verified evaluation set files to the file management server for storage to ensure that the evaluation set files can be called and managed by subsequent processes. The evaluation set files are uploaded with the MD5 value as the file name to ensure that only one copy of the same evaluation set file needs to be stored in the system to avoid repeated file uploads.

[0116] (5) Write the evaluation set information into the database: The system writes the relevant information of the evaluation set file (including the evaluation set name, uploading user, creation time, total number of data, etc.) into the database and generates an MD5 value for subsequent management and call. Finally, the MD5 value of the evaluation set file is written into the database.

[0117] (6) End: After the evaluation set file is stored and the information is written to the database successfully, the process ends and the evaluation set can be used for subsequent evaluation operations.

[0118] The data set management method provided by the embodiment of the present invention is applied, by loading each evaluation set file into the memory, generating the evaluation set data block of each evaluation set file in the cache queue; the evaluation set data block includes a data dictionary constructed based on the target hierarchy and a structure dictionary constructed based on the data item; searching the target evaluation set data block from the cache queue according to the evaluation set number; obtaining the target evaluation data using the data dictionary and structure dictionary in the target evaluation set data block according to the query condition; and evaluating the model to be evaluated using the target evaluation data to obtain the evaluation result. The beneficial effect of this method is that: the directory hierarchy and the data item are separated, the structure dictionary and the data dictionary are constructed respectively, and the whole is stored as the evaluation set data block in the cache queue. Such storage method does not rely on the database, so there is no problem of data set incompatibility, and the query will not be affected by the database performance; since the structure dictionary and the data dictionary are stored in the cache queue, the access speed is extremely fast, which speeds up the query speed; further improves the efficiency of evaluation data set management, and improves the model training speed and accuracy.

[0119] Based on the above model evaluation method, two data query methods for the evaluation set are provided below:

[0120] 1. Data preview: The above-mentioned evaluation set data block is used to realize the paging display of the problem list of each level of the directory, which can meet the query of multi-level directories and realize the paging display of the multi-level directory problem list of the evaluation set structure. The steps are as follows:

[0121] (1) Determine the query path: Before paging, you need to determine the directory level to which the list of questions (uuids) to be displayed belongs. Use directory_dict (structure dictionary) to identify the hierarchical path. User input can include:

[0122] Top-level directory (such as a certain level of heading): Returns all subdirectories and questions under this heading.

[0123] Intermediate directory (such as a second-level or third-level title): Returns all questions under this subdirectory.

[0124] (2) Aggregate question list: According to the directory hierarchy, obtain the uuids of all questions under the specified directory. The specific steps are as follows:

[0125] If the query is for a top-level directory, the corresponding uuids are found directly in directory_dict. If the query is for a deeper level (such as a third-level title), the sub_subject level in directory_dict is recursively accessed until the uuids under the target directory are found. At this point, a complete uuids can be obtained, including all questions under the specified directory (and all its subdirectories).

[0126] (3) Paging operation: Use the obtained uuids to display in pages. The specific steps are as follows:

[0127] Determine paging parameters: The user provides paging parameters page and page_size.

[0128] Calculate the paging boundaries: Use page and page_size to calculate the start and end positions of the current page.

[0129] Extract the current page data: Extract the uuid of the current page from uuids according to the start and end positions.

[0130] Get detailed information about the question: Use question_dict to extract the corresponding question content by uuid. If the data does not exist in question_dict, load the data from the cache database and record the read data content in question_dict to generate a complete question content list for the page.

[0131] The specific data preview example process is as follows: assuming that the user requests to display the list of questions under a certain secondary title in pages, the structure corresponding to the secondary title is found in directory_dict; the uuids of all question items are extracted from the uuids of the structure; the uuid range to be displayed is calculated using paging parameters (such as page=1 and page_size=10); the detailed information of the questions corresponding to these uuids is obtained in sequence from the question_dict / cache database to generate the content of the current page. In this way, the paging display requirements for multi-level directories can be met through query paths, aggregated question lists, and paging operations.

[0132] 2. Selection of assessment questions: When taking the assessment, the user selects a specified number of questions from multiple titles, or sets the number of questions for each of the multiple sub-directories under the first-level directory, and randomly selects a specified number of questions under the sub-directories. The steps are as follows:

[0133] (1) Analysis of user selection: User selection can include the following two situations: Cross-level selection: The user specifies multiple titles and selects a certain number of questions under each title. Hierarchical setting: The user specifies multiple sub-directories under the first-level directory and selects different numbers of questions from each sub-directory as required. Based on the user input, each directory and its corresponding selection quantity are recorded to form the query parameters.

[0134] (2) Get the directory structure and problem list:

[0135] According to the directory path selected by the user, the corresponding structure is found using directory_dict. For each specified directory path, the uuids under the directory are obtained. These uuids represent all the issues contained in the directory and its subdirectories. If further division into subdirectory levels is required, sub_subject is recursively accessed to obtain the uuids of the subdirectories layer by layer.

[0136] (3) Randomly select a specified number of questions: For each directory's uuids, check whether they meet the user-specified number requirement. If the total number of questions in the current directory is less than the required number, select all questions directly; if the total number of questions is more than the required number, use random sampling to randomly select a specified number of uuids from the directory's uuids. Record the selected uuids in each directory.

[0137] (4) Get question details: After the selection is completed, use the uuid in question_dict to quickly get the detailed information (question, answer, etc.) of each selected question. If the uuid does not exist in question_dict, load the data from the cache database and record the data in question_dict. Integrate the detailed information of all selected questions into the final assessment question list.

[0138] The example process for selecting specific assessment questions is as follows: Assume that the user chooses to randomly select 5 questions from the first-level heading 1. Under the first-level heading 2, 3 questions are randomly selected from the second-level heading 2.1 and the second-level heading 2.2. First-level heading 1: Find the first-level heading 1 in directory_dict, get its uuids, and randomly select 5 questions. First-level heading 2: Find the first-level heading 2, enter its sub_subject, and locate the second-level headings 2.1 and 2.2. For the second-level heading 2.1, randomly select 3 questions from its uuids. For the second-level heading 2.2, randomly select 3 questions from its uuids. Look up all the selected uuids in question_dict or the cache database to obtain the question content.

[0139] The following is an introduction to a data set management device provided in an embodiment of the present invention. The data set management device described below and the data set management method described above can be referenced to each other.

[0140] Please refer to Figure 4 , Figure 4 A schematic diagram of a data set management device provided by an embodiment of the present invention may include:

[0141] The data set data block generation module 100 is used to load each data set file into the memory and generate a data set data block of each data set file in the cache queue; the data set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item;

[0142] A data block search module 200, configured to search for a target data set data block from the cache queue according to the data set number;

[0143] The data acquisition module 300 is used to acquire target data according to the query conditions by using the data dictionary and the structure dictionary in the data block of the target data set.

[0144] Based on the above embodiment, the data set data block generation module 100 may include:

[0145] A structure dictionary generating unit, configured to construct a structure dictionary based on the directory hierarchy of the data set file; the key of the structure dictionary is the title name of the directory hierarchy, the value of the structure dictionary is the memory block of the directory hierarchy, and the memory block of the directory hierarchy includes the directory name, the unique identifier list and the structure dictionary of the next level directory;

[0146] A data dictionary generating unit, used for building a data dictionary based on the data items of the data set file; the key of the data dictionary is a unique identifier, the value of the data dictionary is a memory block of the data item, and the memory block of the data item includes all the data of the data item and the directory path where the data item is located;

[0147] A data set data block generation unit, used for adding the key of the data dictionary to the unique identifier list of the target memory block to obtain a data set data block containing the data dictionary and the structure dictionary; the target memory block is the memory block of the directory level and all upper directories corresponding to each unique identifier;

[0148] A writing unit is used to write the data block of the data set into the cache queue.

[0149] Based on the above embodiment, the data acquisition module 300 may include:

[0150] A search unit, configured to search for the target data from a cache database if the target data is not obtained by using the data dictionary and the structure dictionary in the data block of the target data set according to the query condition; all data of the data item stored in the cache database and the directory path where the data item is located are stored in the cache database;

[0151] A loading unit is used to load the target data set file corresponding to the data set number into the memory if the target data is not found in the cache database, and to generate a data set data block of the target data set file in the cache queue.

[0152] Based on the above embodiment, the data set management device may further include:

[0153] A writing module, used for writing the data block of the data set to the head of the cache queue, and determining whether the length of the cache queue exceeds a preset length;

[0154] The deleting module is used to delete the data dictionary of the excess part at the tail of the cache queue if it exceeds.

[0155] Based on the above embodiment, the data acquisition module 300 may include:

[0156] a unique identifier list determining unit, configured to locate a unique identifier list corresponding to the target directory in the structure dictionary when the query condition is to randomly select data from the target directory;

[0157] a unique identifier determination unit, configured to select a target unique identifier from the unique identifier list;

[0158] The target data determination unit is used to search the corresponding target data from the data dictionary according to the target unique identifier.

[0159] It should be noted that the order of the modules and units in the above data set management device can be changed without affecting the logic.

[0160] The data set management device provided by the embodiment of the present invention is used to load each data set file into the memory through the data set data block generation module 100, and generate the data set data block of each data set file in the cache queue; the data set data block includes a data dictionary constructed based on the target hierarchy and a structure dictionary constructed based on the data item; the data block search module 200 is used to search the target data set data block from the cache queue according to the data set number; the data acquisition module 300 is used to obtain the target data using the data dictionary and structure dictionary in the target data set data block according to the query condition. The beneficial effect of this device is that the directory hierarchy and the data item are separated, the structure dictionary and the data dictionary are constructed respectively, and the whole is stored as a data set data block in the cache queue. This storage method does not rely on the database, so there is no problem of data set incompatibility, and the query will not be affected by the database performance; because the structure dictionary and the data dictionary are stored in the cache queue, the access speed is extremely fast, which speeds up the query speed.

[0161] The model evaluation device provided in an embodiment of the present invention is introduced below. The model evaluation device described below and the model evaluation method described above can be referenced to each other.

[0162] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of a model evaluation device provided by an embodiment of the present invention may include:

[0163] The evaluation set data block generation module 400 is used to load each evaluation set file into the memory and generate the evaluation set data block of each evaluation set file in the cache queue; the evaluation set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data item;

[0164] The target evaluation set data block search module 500 is used to search the target evaluation set data block from the cache queue according to the evaluation set number;

[0165] The target evaluation data acquisition module 600 is used to acquire the target evaluation data by using the data dictionary and the structure dictionary in the target evaluation set data block according to the query condition;

[0166] The model evaluation module 700 is used to evaluate the model to be evaluated using the target evaluation data to obtain an evaluation result.

[0167] It should be noted that the order of the modules and units in the above-mentioned model evaluation device can be changed without affecting the logic.

[0168] The model evaluation device provided by the embodiment of the present invention is applied, and the evaluation set data block generation module 400 is used to load each evaluation set file into the memory, and generate an evaluation set data block for each evaluation set file in the cache queue; the evaluation set data block includes a data dictionary constructed based on the target hierarchy and a structure dictionary constructed based on the data item; a target evaluation set data block search module 500 is used to search the target evaluation set data block from the cache queue according to the evaluation set number; a target evaluation data acquisition module 600 is used to obtain target evaluation data using the data dictionary and structure dictionary in the target evaluation set data block according to the query condition; a model evaluation module 700 is used to evaluate the model to be evaluated using the target evaluation data to obtain the evaluation result. The beneficial effects of the device are: separating the directory hierarchy and the data items, constructing the structure dictionary and the data dictionary respectively, and storing them as a whole as an evaluation set data block in the cache queue. This storage method does not rely on the database, so there is no problem of data set incompatibility, and the query will not be affected by the database performance; because the structure dictionary and the data dictionary are stored in the cache queue, the access speed is extremely fast, which speeds up the query speed, further improves the efficiency of evaluation data set management, and improves the model training speed and accuracy.

[0169] In order to make the present invention easier to understand, please refer to Figure 6 , Figure 6 A schematic diagram of a model evaluation architecture provided for an embodiment of the present invention may specifically include: a file management module, a database module, a cache database module, an evaluation management module, an evaluation module and an interface module.

[0170] The file management module is used to store the user's evaluation set files (i.e. the original files of the evaluation set).

[0171] The database module is used to store relevant information of the evaluation set, such as the evaluation set ID, name, user, file, creation time, update time and other information.

[0172] The evaluation set management module depends on the file management module, cache database module and database module. The main functions of the evaluation set management module are: (1) Management functions such as uploading, deleting, modifying and previewing evaluation set files. It supports uploading data set files and performing format verification, and generates MD5 values. (2) Loading and preparing evaluation sets: Based on user requests, the evaluation set management module reads the corresponding evaluation data from the file management module; (3) Data screening and classification: Screening evaluation data based on user needs, such as selecting data of specific categories, difficulty or labels; (4) Cache and reuse: In order to improve system efficiency, the evaluation set management module is equipped with an evaluation set data block cache queue. The evaluation set data block contains all the evaluation data in the evaluation set, and can respond quickly when users call it repeatedly.

[0173] The evaluation module is the execution part of the model evaluation. It obtains the prepared evaluation data from the evaluation set management module and evaluates the model based on these evaluation data. The evaluation module performs specific evaluation operations, such as accuracy calculation, classification effect evaluation, error analysis, etc. The evaluation module can use the different classifications or filtered evaluation data provided by the evaluation set management module to perform targeted tests. For example, the sentiment analysis task can test the evaluation data of different sentiment categories, and the machine translation task can test the evaluation data of specific language pairs.

[0174] The evaluation business organization process is as follows: the evaluation user initiates an evaluation request through the interface module; the interface module forwards the evaluation request to the evaluation set management module; the evaluation set management module retrieves the evaluation data from the file management module and performs screening, classification, caching and other processing; the prepared evaluation set data is passed to the evaluation module, and the evaluation module evaluates the model according to the data set; the evaluation results are finally returned to the user. The core function of this architecture is to enable the model evaluator to select evaluation data from the evaluation set when evaluating the model. The evaluation module passes the evaluation set ID and query conditions entered by the user to the evaluation set management module. The evaluation set management module selects evaluation data from the corresponding evaluation set according to the conditions and returns it to the evaluation module. After obtaining the evaluation data, the evaluation module uses the evaluation data to evaluate the model. This architecture can realize functions such as file upload, deletion, update, and evaluation data set preview of the evaluation set.

[0175] Figure 7 A schematic diagram of a device provided in an embodiment of the present invention is shown in FIG. Figure 7 As shown, the device includes:

[0176] A memory 60, for storing computer programs;

[0177] The processor 61 is used to implement the steps of the data set management and / or model evaluation method in the above-mentioned embodiment when executing the computer program.

[0178] The device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0179] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0180] The memory 60 may include one or more computer-readable storage media, which may be non-transitory. The memory 60 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 60 is at least used to store the following computer program 601, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps of the data set management and / or model evaluation method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 60 may also include an operating system 602 and data 603, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 602 may include Windows, Unix, Linux, etc. Data 603 may include but is not limited to data for data set management and / or model evaluation methods, etc.

[0181] In some embodiments, the device may further include a display screen 62 , an input / output interface 63 , a communication interface 64 , a power supply 65 , and a communication bus 66 .

[0182] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown.

[0183] It is understandable that if the data set management and / or model evaluation method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the current technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program code.

[0184] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned data set management and / or model evaluation method are implemented.

[0185] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be cross-referenced with other embodiments described in this document.

[0186] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned disclosed data set management and / or model evaluation method.

[0187] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0188] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0189] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0190] The above is a detailed introduction to a data set management method, model evaluation method, device, equipment and computer-readable storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A data set management method, characterized in that: include: Load each data set file into the memory, and generate data set data blocks of each data set file in the cache queue; The data set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data items; Searching for a target data set data block from the cache queue according to the data set number; According to the query conditions, the target data is acquired using the data dictionary and the structure dictionary in the data block of the target data set.

2. The data set management method according to claim 1, characterized in that: Each data set file is loaded into the memory, and a data set data block of each data set file is generated in the cache queue, including: Building a structure dictionary based on the directory hierarchy of the data set file; the key of the structure dictionary is the title name of the directory hierarchy, the value of the structure dictionary is the memory block of the directory hierarchy, and the memory block of the directory hierarchy includes the directory name, the unique identifier list and the structure dictionary of the next level directory; Building a data dictionary based on the data items of the data set file; the key of the data dictionary is a unique identifier, the value of the data dictionary is a memory block of the data item, and the memory block of the data item includes all data of the data item and the directory path where the data item is located; Add the key of the data dictionary to the unique identifier list of the target memory block to obtain a data set data block containing the data dictionary and the structure dictionary; the target memory block is the memory block of the directory level and all upper directories corresponding to each unique identifier; The data block of the data set is written into the cache queue.

3. The data set management method according to claim 1, characterized in that: According to the query conditions, the target data is obtained by using the data dictionary and the structure dictionary in the target data set data block, including: If the target data is not obtained by using the data dictionary and the structure dictionary in the data block of the target data set according to the query condition, the target data is searched from the cache database; all data of the data item stored in the cache database and the directory path where the data item is located are stored in the cache database; If the target data is not found in the cache database, the target data set file corresponding to the data set number is loaded into the memory, and a data set data block of the target data set file is generated in the cache queue.

4. The data set management method according to claim 3, characterized in that: After generating the data set data blocks of the target data set file in the cache queue, the method further includes: Writing the data block of the data set to the head of the cache queue, and determining whether the length of the cache queue exceeds a preset length; If exceeded, the data dictionary of the excess portion at the tail of the cache queue is deleted.

5. The data set management method according to claim 1, characterized in that: According to the query conditions, the target data is obtained by using the data dictionary and the structure dictionary in the target data set data block, including: When the query condition is to randomly select data from the target directory, locating a unique identifier list corresponding to the target directory in the structure dictionary; Select a target unique identifier from the list of unique identifiers; The corresponding target data is searched from the data dictionary according to the target unique identifier.

6. A model evaluation method, characterized in that: include: Load each evaluation set file into the memory, and generate an evaluation set data block of each evaluation set file in the cache queue; The evaluation set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data items; Searching for a target evaluation set data block from the cache queue according to the evaluation set number; According to the query conditions, the target evaluation data is obtained by using the data dictionary and the structure dictionary in the target evaluation set data block; The target evaluation data is used to evaluate the model to be evaluated to obtain an evaluation result.

7. A data set management device, characterized in that: include: A data set data block generation module, used for loading each data set file into the memory, and generating a data set data block of each data set file in the cache queue; The data set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data items; A data block search module, used for searching a target data set data block from the cache queue according to the data set number; The data acquisition module is used to acquire target data according to the query conditions by using the data dictionary and the structure dictionary in the data block of the target data set.

8. A model evaluation device, characterized in that: include: An evaluation set data block generation module is used to load each evaluation set file into the memory and generate an evaluation set data block for each evaluation set file in the cache queue; The evaluation set data block includes a data dictionary constructed based on the target level and a structure dictionary constructed based on the data items; A target evaluation set data block search module, used to search the target evaluation set data block from the cache queue according to the evaluation set number; A target evaluation data acquisition module is used to acquire target evaluation data using a data dictionary and a structure dictionary in the target evaluation set data block according to a query condition; The model evaluation module is used to evaluate the model to be evaluated using the target evaluation data to obtain the evaluation results.

9. A device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the method for data set management and / or model evaluation as described in any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the method for data set management and / or model evaluation according to any one of claims 1 to 6 is implemented.

Images