Method and system for storing and retrieving deep-learning data sets

Abstract

Disclosed in the present invention are a method and system for storing and retrieving deep-learning data sets. The method comprises: according to a preset hash algorithm, calculating a first file code of each data item in an acquired deep-learning data set; performing code capturing on the first file codes, so as to obtain a code value corresponding to each data item, and performing retrieval according to the code value, so as to determine whether there is a corresponding folder; according to the folder, determining whether a data item is duplicated data; and sequentially saving each data item in the deep-learning data set, and after the data item is saved, recording information, such as a file address of the data item, so as to facilitate subsequent retrieval. The space for saving a deep-learning data set is reduced, and the deep-learning data set is retrievable.

Description

A deep learning data set storage and retrieval method and system Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for storing and retrieving deep learning datasets. Background Art

[0002] With the significant improvement in computing power, deep learning has developed rapidly and has become the main growth point and driving force of artificial intelligence. In addition, the advent of the big data era has enabled deep learning to evolve from AI model development-centered to data-centered development, which has also greatly promoted the development efficiency and higher accuracy of AI algorithm models.

[0003] Unlike traditional structured data, deep learning datasets often contain unstructured data such as video, audio, images, and text. These data volumes are so large that traditional databases cannot store them. Currently, datasets are primarily managed by storing them on disk using named files. This not only makes data labels impossible to retrieve and prevents data from flowing, but also results in a large amount of duplicate data between compressed packages, wasting storage space.

[0004] Summary of the Invention

[0005] The present invention discloses a method and system for storing and retrieving deep learning datasets, which solves the technical problems of data duplication and inability to be retrieved in the prior art and reduces the space occupied by data storage.

[0006] To achieve the above objectives, the present invention discloses a method for storing and retrieving deep learning datasets, comprising:

[0007] Traversing each data item in the acquired deep learning dataset according to a preset hash algorithm, and generating a first file code corresponding to each data item;

[0008] Encode and intercept the first file code of each data item respectively, obtain the code value corresponding to each data item respectively, and search according to the code value to determine whether there is a first folder named with the code value;

[0009] If a first folder named with the code value exists, comparing the first file code of the data item corresponding to the code value with the second file code of the data item existing in the first folder to determine whether the second file code is the same as the first file code; if so, deleting the data item corresponding to the first file code; if not, adding the data item corresponding to the first file code to the first folder;

[0010] If the first folder named with the code value does not exist, adding the data item corresponding to the code value to a preset folder, and naming the preset folder according to the code value to obtain the first folder;

[0011] The file address corresponding to each data item is generated according to the first folder, and the file code, data set name and annotation information corresponding to each data item in the deep learning data set are saved in a preset database for retrieval.

[0012] The present invention discloses a method for storing and retrieving a deep learning dataset, comprising traversing each data item in the deep learning dataset using a preset hash algorithm to generate a file code corresponding to each data item one by one, thereby ensuring the uniqueness of the data item. Then, after generating the file code corresponding to the data item, in order to improve the efficiency of data item preservation, several codes are intercepted from the file code as the name of the folder where the data item is located, so that when the folder is subsequently saved, the retrieval result can be obtained by using part of the code for retrieval, thereby improving the efficiency of file preservation. The folder is named using the code value. If the code value of the newly added data item is the same as the name of the existing folder, it is determined that the data item is repeated, and the data item is deleted, thereby avoiding data duplication and reducing space resource occupancy.

[0013] As a preferred example, the deep learning dataset storage and retrieval method further includes:

[0014] Searching the database according to the preset annotation information and the data set name to obtain the file code of the data item to be searched, and generating a search path according to the file code;

[0015] The plurality of data items stored in the first folder are searched according to the search path to obtain the data items to be searched.

[0016] The database described in the present invention stores the annotation information, data set name and file code of the data item to be retrieved. After obtaining the annotation information and data set name of the data item to be retrieved, the file code of the data item to be retrieved can be determined. According to the file code, the address of the folder where it is stored can be determined, and then the folder is searched according to the address to obtain the data item to be retrieved. This solves the technical problem of data flow failure caused by the inability to retrieve annotation information in the prior art, and realizes data retrieval.

[0017] As a preferred example, traversing each data item in the deep learning data set obtained according to the preset hash algorithm and generating a first file code corresponding to each data item includes:

[0018] Looping through the acquired deep learning dataset to obtain a plurality of data items contained in the deep learning dataset; the data items include video, audio, pictures, and text;

[0019] A hash calculation is performed on each of the plurality of data items according to a preset hash algorithm to obtain an MD5 value corresponding to each of the data items.

[0020] The present invention first loops through the deep learning dataset to obtain all data items included in the deep learning dataset to avoid omissions in data storage, and then uses a hash algorithm to generate an MD5 value corresponding to each data item. Based on the high confidentiality of the MD5 value, the accuracy of the data item corresponding to each data item is guaranteed, thereby improving the accuracy of file storage.

[0021] As a preferred example, the first file code of each data item is respectively encoded and intercepted, a code value corresponding to each data item is obtained, and searching is performed according to the code value, including:

[0022] According to a preset interception rule, the first file code corresponding to each data item is encoded and intercepted by a string interception algorithm to obtain the encoding value corresponding to each data item;

[0023] A search is performed from a preset database according to the coding value to determine whether there is a folder named with the coding value in the database.

[0024] After obtaining the file code, the present invention uses a preset string interception algorithm to intercept a part of the code from the file code as a retrieval basis based on the large amount of data in the file code in order to improve file storage efficiency, thereby reducing the amount of data matching and improving retrieval efficiency.

[0025] As a preferred example, adding the data item corresponding to the code value to a preset folder and naming the preset folder according to the code value to obtain the first folder includes:

[0026] Moving the data item into the folder according to a preset file moving method;

[0027] According to the code value corresponding to the data item, the preset folder is renamed using a preset file renaming algorithm to obtain a first folder named with the code value.

[0028] The present invention names a folder according to the coding value and saves data items corresponding to the coding value in the folder, thereby ensuring that during the data storage process, the coding value is used to avoid the storage of duplicate data and avoid the waste of space resources.

[0029] As a preferred example, generating a search path according to the file encoding includes:

[0030] Continue encoding and intercepting the acquired file code to obtain a code value corresponding to the file code, and then obtain a storage path of a folder where the code value is located and a folder name of the folder according to the code value;

[0031] A search path for the data to be retrieved is generated according to the storage path of the folder and the folder name of the folder in combination with the file code.

[0032] The present invention saves the annotation information, file code and data set name of each data item in the database, so that data retrieval can be performed by inputting the annotation information, thereby solving the technical problem in the prior art that data retrieval cannot be performed using the annotation information.

[0033] On the other hand, the present invention discloses a deep learning dataset storage and retrieval system, including a file encoding module, a code interception module, a file saving module and a data saving module;

[0034] The file encoding module is used to traverse each data item in the acquired deep learning data set according to a preset hash algorithm, and generate a first file code corresponding to each data item;

[0035] The code interception module is used to respectively code intercept the first file code of each data item, obtain the code value corresponding to each data item, and perform a search based on the code value to determine whether there is a first folder named with the code value;

[0036] The file saving module is configured to, if a first folder named with the coding value exists, compare the first file coding of the data item corresponding to the coding value with the second file coding of the data item existing in the first folder to determine whether the second file coding is the same as the first file coding; if so, delete the data item corresponding to the first file coding; if not, add the data item corresponding to the first file coding to the first folder; if the first folder named with the coding value does not exist, add the data item corresponding to the coding value to a preset folder, and name the preset folder according to the coding value to obtain the first folder;

[0037] The data saving module is used to generate the file address corresponding to each data item according to the first folder, and save the file code, data set name and annotation information corresponding to each data item in the deep learning data set through a preset database for retrieval.

[0038] The present invention discloses a deep learning data set storage and retrieval system, which includes using a preset hash algorithm to traverse each data item in the deep learning data set to generate a file code corresponding to each data item, thereby ensuring the uniqueness of the data item. Then, after generating the file code corresponding to the data item, in order to improve the efficiency of data item storage, several codes are intercepted from the file code as the name of the folder where the data item is located, so that when the folder is subsequently saved, the search result can be obtained by using part of the code for retrieval, thereby improving the efficiency of file storage, and the folder is named using the code value. If the code value of the newly added data item is the same as the name of the existing folder, it is determined that the data item is repeated, and the data item is deleted, thereby avoiding data duplication and reducing space resource occupancy.

[0039] As a preferred example, the deep learning dataset storage and retrieval system further includes a path module and a retrieval module;

[0040] The path module is used to search the database according to the preset annotation information and the data set name, obtain the file code of the data item to be searched, and generate a search path according to the file code;

[0041] The retrieval module is used to retrieve a plurality of data items stored in the first folder according to the retrieval path to obtain the data items to be retrieved.

[0042] The database described in the present invention stores the annotation information, data set name and file code of the data item to be retrieved. After obtaining the annotation information and data set name of the data item to be retrieved, the file code of the data item to be retrieved can be determined. According to the file code, the address of the folder where it is stored can be determined, and then the folder is searched according to the address to obtain the data item to be retrieved. This solves the technical problem of data flow failure caused by the inability to retrieve annotation information in the prior art, and realizes data retrieval.

[0043] As a preferred example, the file encoding module includes a traversal unit and an encoding unit;

[0044] The traversal unit is used to traverse the acquired deep learning data set in a loop to obtain a plurality of data items contained in the deep learning data set; the data items include video, audio, picture and text;

[0045] The encoding unit is used to perform hash calculation on each of the multiple data items according to a preset hash algorithm to obtain an MD5 value corresponding to each of the data items.

[0046] The present invention first loops through the deep learning dataset to obtain all data items included in the deep learning dataset to avoid omissions in data storage, and then uses a hash algorithm to generate an MD5 value corresponding to each data item. Based on the high confidentiality of the MD5 value, the accuracy of the data item corresponding to each data item is guaranteed, thereby improving the accuracy of file storage.

[0047] As a preferred example, the code interception module includes an interception unit and a retrieval unit;

[0048] The interception unit is used to encode and intercept the first file code corresponding to each data item using a string interception algorithm according to a preset interception rule to obtain the code value corresponding to each data item;

[0049] The retrieval unit is used to search from a preset database according to the coding value, and determine whether there is a folder named with the coding value in the database.

[0050] After obtaining the file code, the present invention uses a preset string interception algorithm to intercept a part of the code from the file code as a retrieval basis based on the large amount of data in the file code in order to improve file storage efficiency, thereby reducing the amount of data matching and improving retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of a method for storing and retrieving deep learning datasets provided by an embodiment of the present invention;

[0052] FIG2 is another flowchart of a method for storing and retrieving a deep learning dataset provided by an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of the structure of a deep learning dataset storage and retrieval system provided by an embodiment of the present invention;

[0054] Figure 4: Another structural diagram of a deep learning dataset storage and retrieval system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0056] Example

[0057] An embodiment of the present invention provides a method for storing and retrieving a deep learning dataset. The specific implementation process of the storage and retrieval method is shown in FIG1 , which mainly includes steps 101 to 105. The steps are:

[0058] Step 101: traverse each data item in the acquired deep learning dataset according to a preset hash algorithm, and generate a first file code corresponding to each data item.

[0059] In this embodiment, this step mainly includes: looping through the acquired deep learning data set to obtain several data items contained in the deep learning data set; the data items include video, audio, pictures and text; performing hash calculation on each of the several data items according to a preset hash algorithm to obtain the MD5 value corresponding to each of the data items.

[0060] Exemplarily, in this embodiment, after obtaining the deep learning dataset to be stored, each data item contained in the deep learning dataset can be obtained by looping through the deep learning dataset. Multiple data items constitute a deep learning dataset, and the data items include video, audio, pictures, text and other data. For example, multiple car pictures constitute a car deep learning dataset. The file encoding corresponding to each of the data items is obtained according to the preset hash algorithm. In this embodiment, the MD5 algorithm can be preferably used for hashing to obtain a unique MD5 value corresponding to each of the data items.

[0061] This embodiment first loops through the deep learning dataset to obtain all data items included in the deep learning dataset to avoid omissions in data storage, and then uses a hash algorithm to generate an MD5 value corresponding to each data item. Based on the high confidentiality of the MD5 value, the accuracy of the data item corresponding to each data item is guaranteed, thereby improving the accuracy of file storage.

[0062] Step 102: Encode and intercept the first file code of each data item respectively, obtain the code value corresponding to each data item respectively, and search according to the code value to determine whether there is a first folder named with the code value.

[0063] In this embodiment, this step mainly includes: according to the preset interception rules, the first file code corresponding to each data item is encoded and intercepted by a string interception algorithm to obtain the encoding value corresponding to each data item; according to the encoding value, a search is performed from a preset database to determine whether there is a first folder named with the encoding value in the database.

[0064] Exemplarily, in this embodiment, several bits of data are intercepted from the file code corresponding to each data item to perform folder search. Preferably, a string interception algorithm can be used to intercept the last two digits of the file code as the code value corresponding to the data item. Based on the above steps, the last two digits of the MD5 value are selected as the code value, and then the last two digits of the code are used to identify and retrieve whether there is a folder named with the code value. Preferably, before performing the folder search, when saving the data item, the mobile file API of the programming language can be called to put the MD5 values ​​with the same last two digits into the same folder and name the folder with the last two digits of the MD5 to form the first folder.

[0065] After obtaining the file code, this embodiment uses a preset string interception algorithm to intercept a portion of the code from the file code as a retrieval basis in order to improve file storage efficiency. This reduces the amount of data to be matched and improves retrieval efficiency.

[0066] Step 103: If there is a first folder named with the coding value, compare the first file code of the data item corresponding to the coding value with the second file code of the data item existing in the first folder to determine whether there is a second file code that is the same as the first file code; if so, delete the data item corresponding to the first file code; if not, add the data item corresponding to the first file code to the first folder.

[0067] Exemplarily, in this embodiment, according to the coding value corresponding to each data item, it is first searched whether there is a folder named with the coding value. In this embodiment, the search is performed based on the last two digits of the MD5 value. If there is a folder named with the last two digits of the MD5 value, the MD5 values ​​corresponding to the several data items stored in the folder are extracted, and then the MD5 value with the stored data item is compared with the MD5 values ​​corresponding to the several data items stored in the folder to determine whether there is the same MD5 value. If so, it means that the data is repeated, and the data to be stored is deleted. If not, the data item can be added to the folder.

[0068] Step 104: If the first folder named with the coding value does not exist, add the data item corresponding to the coding value to a preset folder, and name the preset folder according to the coding value to obtain the first folder.

[0069] In this embodiment, the steps mainly include: moving the data item to the folder according to a preset file moving method; renaming the preset folder through a preset file renaming algorithm according to the coding value corresponding to the data item to obtain a first folder named with the coding value.

[0070] Exemplarily, in this embodiment, if there is no folder named with the coding value, in this embodiment, a search is performed based on the last two digits of the MD5 value. If there is no folder named with the last two digits of the MD5 value, the data to be stored is saved in a preset folder. If the preset folder is renamed according to the last two digits of the MD5 value of the data to be stored.

[0071] This embodiment names a folder according to the coding value and saves the data item corresponding to the coding value in the folder, thereby ensuring that during the data storage process, duplicate data is avoided by using the coding value, thereby avoiding waste of space resources.

[0072] Step 105: Generate a file address corresponding to each data item according to the first folder, and save the file code, data set name and annotation information corresponding to each data item in the deep learning data set through a preset database for retrieval.

[0073] Exemplarily, in this embodiment, after the data item is saved in the folder, the name of the folder where the data item is located, the path of the folder, the MD5 value corresponding to the data item, the annotation information of the data item itself and the name of the data set in which it is located are recorded, and the above-mentioned recorded data are saved in the database, so that the database stores the name of the data set corresponding to the data item, the annotation information, the path of the folder and other information to facilitate subsequent retrieval.

[0074] 2 , the implementation process of a deep learning dataset storage and retrieval method provided in this embodiment further includes steps 201 to 202, which are:

[0075] Step 201: Search the database according to the preset annotation information and data set name, obtain the file code of the data item to be searched, and generate a search path according to the file code.

[0076] In this embodiment, this step mainly includes: continuing to encode and intercept the acquired file code to obtain the code value corresponding to the file code, and then obtaining the storage path of the folder where the code value is located and the folder name of the folder according to the code value; based on the storage path of the folder and the folder name of the folder, combined with the file code, generate the retrieval path of the data to be retrieved.

[0077] Illustratively, in this embodiment, according to the specified annotation information and data set name, the database is searched to obtain the MD5 values ​​of all data items to form a path set. In this embodiment, since the annotation information, data set name and MD5 value information corresponding to the data item are stored in the database, the corresponding MD5 value information can be obtained according to the annotation information and data set name. Then, since the last two digits of the MD5 value are used as the folder name, all data items with the same last two digits of the MD5 value will be placed in the same folder. Therefore, after obtaining the MD5 value, the last two digits of the MD5 value are cut out to further obtain the folder name, and then the path of the data item and the default storage path of the data + folder name + MD5 value set can be obtained.

[0078] This embodiment saves the annotation information, file code and data set name of each data item in the database, so that data retrieval can be performed by inputting the annotation information, thereby solving the technical problem in the prior art that data retrieval cannot be performed using annotation information.

[0079] Step 202: searching for a plurality of data items stored in the first folder according to the search path to obtain the data items to be searched.

[0080] Illustratively, in this embodiment, the data items corresponding to the search path can be accessed and read by traversing the search path in a loop using an API for reading files in a programming language.

[0081] In addition, this embodiment also provides a deep learning dataset storage and retrieval system. Please refer to Figure 3 for the specific structural composition of the storage and retrieval system, which includes a file encoding module 301, an encoding interception module 302, a file saving module 303 and a data saving module 304.

[0082] The file encoding module 301 is used to traverse each data item in the acquired deep learning data set according to a preset hash algorithm, and generate a first file code corresponding to each data item.

[0083] The code interception module 302 is used to code intercept the first file code of each data item, obtain the code value corresponding to each data item, and search according to the code value to determine whether there is a first folder named with the code value.

[0084] The file saving module 303 is used to compare the first file code of the data item corresponding to the code value with the second file code of the data item existing in the first folder if there is a first folder named with the code value, to determine whether there is a second file code that is the same as the first file code; if so, delete the data item corresponding to the first file code; if not, add the data item corresponding to the first file code to the first folder; if there is no first folder named with the code value, add the data item corresponding to the code value to a preset folder, and name the preset folder according to the code value to obtain the first folder.

[0085] The data saving module 304 is used to generate the file address corresponding to each data item according to the first folder, and save the file code, data set name and annotation information corresponding to each data item in the deep learning data set through a preset database for retrieval.

[0086] In this embodiment, the file encoding module 301 includes a traversal unit and an encoding unit.

[0087] The traversal unit is used to traverse the acquired deep learning data set in a loop to obtain a plurality of data items contained in the deep learning data set; the data items include video, audio, pictures and text.

[0088] The encoding unit is used to perform hash calculation on each of the multiple data items according to a preset hash algorithm to obtain an MD5 value corresponding to each of the data items.

[0089] In this embodiment, the code interception module 302 includes an interception unit and a retrieval unit.

[0090] The interception unit is used to perform encoding interception on the first file code corresponding to each data item through a character string interception algorithm according to a preset interception rule, so as to obtain the encoding value corresponding to each data item.

[0091] The retrieval unit is used to search from a preset database according to the coding value, and determine whether there is a folder named with the coding value in the database.

[0092] 4 , the structure of a deep learning dataset storage and retrieval system provided in this embodiment further includes a path module 401 and a retrieval module 402 .

[0093] The path module 401 is used to search the database according to the preset annotation information and the data set name, obtain the file code of the data item to be searched, and generate a search path according to the file code.

[0094] The retrieval module 402 is configured to retrieve a plurality of data items stored in the first folder according to the retrieval path to obtain the data items to be retrieved.

[0095] The present invention discloses a method and system for storing and retrieving deep learning datasets, comprising using a preset hash algorithm to traverse each data item in the deep learning dataset to generate a file code corresponding to each data item, thereby ensuring the uniqueness of the data item. Then, after generating the file code corresponding to the data item, in order to improve the efficiency of data item storage, several codes are intercepted from the file code as the name of the folder where the data item is located, so that when the folder is subsequently saved, the search result can be obtained by using part of the code for retrieval, thereby improving the efficiency of file storage. The folder is named using the code value. If the code of the newly added data item is If the code value is the same as the name of an existing folder, it is determined that the data item is duplicated, and the data item is deleted, thereby avoiding data duplication and reducing the space resource occupancy rate. The database is then used to save the annotation information, data set name and file code of the data item to be retrieved. After obtaining the annotation information and data set name of the data item to be retrieved, the file code of the data item to be retrieved can be determined. The address of the folder where it is saved can be determined according to the file code, and then the folder is searched according to the address to obtain the data item to be retrieved. This solves the technical problem of data flow failure caused by the inability to retrieve annotation information in the prior art, and realizes data retrieval.

[0096] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for storing and retrieving deep learning datasets, characterized in that: include: Traversing each data item in the acquired deep learning data set according to a preset hash algorithm, and generating a first file code corresponding to each data item; Encode and intercept the first file code of each data item respectively, obtain the code value corresponding to each data item respectively, and search according to the code value to determine whether there is a first folder named with the code value; If there is a first folder named with the coding value, compare the first file code of the data item corresponding to the coding value with the second file code of the data item existing in the first folder to determine whether there is a second file code that is the same as the first file code; if there is, delete the data item corresponding to the first file code; if not, add the data item corresponding to the first file code to the first folder; If the first folder named with the code value does not exist, add the data item corresponding to the code value to a preset folder, and name the preset folder according to the code value to obtain the first folder; The file addresses corresponding to each data item are generated according to the first folder, and the file code, data set name and annotation information corresponding to each data item in the deep learning data set are saved in a preset database for retrieval.

2. A deep learning dataset storage and retrieval method as claimed in claim 1, characterized in that: Also includes: Searching the database according to preset annotation information and data set name, obtaining the file code of the data item to be searched, and generating a search path according to the file code; The plurality of data items stored in the first folder are searched according to the search path to obtain the data items to be searched.

3. A deep learning dataset storage and retrieval method as claimed in claim 1, characterized in that: The traversing each data item in the acquired deep learning data set according to a preset hash algorithm and generating a first file code corresponding to each data item respectively includes: Loop through the acquired deep learning data set to obtain a number of data items contained in the deep learning data set; the data items include video, audio, picture and text; A hash calculation is performed on each of the plurality of data items according to a preset hash algorithm to obtain an MD5 value corresponding to each of the data items.

4. A deep learning dataset storage and retrieval method as claimed in claim 1, characterized in that: The first file code of each data item is respectively encoded and intercepted to obtain the code value corresponding to each data item, and the search is performed according to the code value, including: According to a preset interception rule, the first file code corresponding to each data item is intercepted by a string interception algorithm to obtain a code value corresponding to each data item; A search is performed from a preset database according to the coding value to determine whether there is a folder named with the coding value in the database.

5. A deep learning dataset storage and retrieval method as claimed in claim 1, characterized in that: The step of adding the data item corresponding to the coding value to a preset folder and naming the preset folder according to the coding value to obtain the first folder includes: Moving the data item to the folder according to a preset file moving method; According to the code value corresponding to the data item, the preset folder is renamed using a preset file renaming algorithm to obtain a first folder named with the code value.

6. A deep learning dataset storage and retrieval method as claimed in claim 2, characterized in that: The step of generating a search path according to the file encoding comprises: Continue encoding and intercepting the acquired file code to obtain a code value corresponding to the file code, and then obtain a storage path of a folder where the code value is located and a folder name of the folder according to the code value; A search path for the data to be searched is generated according to the storage path of the folder and the folder name of the folder in combination with the file code.

7. A deep learning dataset storage and retrieval system, characterized in that: It includes a file encoding module, a code interception module, a file saving module and a data saving module; The file encoding module is used to traverse the deep learning data set obtained according to the preset hash algorithm and generating a first file code corresponding to each data item; The code interception module is used to code intercept the first file code of each data item, obtain the code value corresponding to each data item, and search according to the code value to determine whether there is a first folder named with the code value; The file saving module is used for comparing the first file code of the data item corresponding to the code value with the second file code of the data item existing in the first folder if there is a first folder named with the code value, to determine whether there is a second file code that is the same as the first file code; if so, deleting the data item corresponding to the first file code; if not, adding the data item corresponding to the first file code to the first folder; if there is no first folder named with the code value, adding the data item corresponding to the code value to a preset folder, and naming the preset folder according to the code value to obtain the first folder; The data saving module is used to generate a file address corresponding to each data item according to the first folder, and save the file code, data set name and annotation information corresponding to each data item in the deep learning data set through a preset database for retrieval.

8. A deep learning dataset storage and retrieval system as claimed in claim 7, characterized in that: It also includes a path module and a retrieval module; The path module is used to search the database according to the preset annotation information and the data set name, obtain the file code of the data item to be searched, and generate a search path according to the file code; The retrieval module is used to retrieve a plurality of data items stored in the first folder according to the retrieval path to obtain the data items to be retrieved.

9. A deep learning dataset storage and retrieval system as claimed in claim 7, characterized in that: The file encoding module includes a traversal unit and an encoding unit; The traversal unit is used to loop through the acquired deep learning data set to obtain a plurality of data items contained in the deep learning data set; the data items include video, audio, picture and text; The encoding unit is used to perform hash calculation on each of the multiple data items according to a preset hash algorithm to obtain an MD5 value corresponding to each of the data items.

10. A deep learning dataset storage and retrieval system as claimed in claim 7, characterized in that: The coding interception module includes an interception unit and a retrieval unit; The interception unit is used to perform encoding interception on the first file code corresponding to each data item by a character string interception algorithm according to a preset interception rule, so as to obtain the encoding value corresponding to each data item; The retrieval unit is used to search from a preset database according to the coding value, and determine whether there is a folder named with the coding value in the database.

Images